Streaming versus batch processing of sensor data in a hazardous weather detection system (original) (raw)

Condition Monitoring for Wireless Sensor Network-Based Automatic Weather Stations

EAI Endorsed Transactions on Internet of Things

Wireless Sensor Network (WSN)-based Automatic Weather Stations (AWSs) perform automatic collection and transmission of weather data. These AWSs face challenges, which lower their performance. Hence, a need for regular monitoring to reduce down time. We propose condition monitoring, comprised of a data receiver, analyser, problem classifier and reporter and visualizer, to mine data relationships, identify possible causes of problems and perform reporting of AWS status. The data receiver uses an M/M/1/k queuing model. We use Successive Pairwise REcord Differences (SPREDs) algorithm to compare arrival rates and packet content so as to establish sensor, node and AWS level performance. We also perform a hybrid of Grubb outlier detection and correlations amongst related variables for data validation. Problems take on one of four states. One connection can receive data at a rate as low as 1ms, without loss while problem identification especially in high density network is improved

On a tandem queue with batch service and its applications in wireless sensor networks

Queueing Systems

We present a tandem network of queues 0,. .. , s − 1. Customers arrive at queue 0 according to a Poisson process with rate λ. There are s independent batch service processes at exponential rates μ 0 ,. .. , μ s−1. Service process i, i = 0,. .. , s−1, at rate μ i is such that all customers of all queues 0,. .. , i simultaneously receive service and move to the next queue. We show that this system has a geometric product-form steady-state distribution. Moreover, we determine the service allocation that minimizes the waiting time in the system and state conditions to approximate such optimal allocations. Our model is motivated by applications in wireless sensor networks, where s observations from different sensors are collected for data fusion. We demonstrate that both optimal centralized and decentralized sensor scheduling can be modeled by our queueing model by choosing the values of μ i appropriately. We quantify the performance gap between the centralized and decentralized schedules for arbitrarily large sensor networks.

Streaming analysis in wireless sensor networks

Wireless Communications and Mobile Computing, 2012

Two new incremental models for online anomaly detection in data streams at nodes in wireless sensor networks are discussed. These models are incremental versions of a model that uses ellipsoids to detect first, second, and higherordered anomalies in arrears. The incremental versions can also be used this way but have additional capabilities offered by processing data incrementally as they arrive in time. Specifically, they can detect anomalies 'on-the-fly' in near real time. They can also be used to track temporal changes in near real-time because of sensor drift, cyclic variation, or seasonal changes. One of the new models has a mechanism that enables graceful degradation of inputs in the distant past (fading memory). Three real datasets from single sensors in deployed environmental monitoring networks are used to illustrate various facets of the new models. Examples compare the incremental version with the previous batch and dynamic models and show that the incremental versions can detect various types of dynamic anomalies in near real time. Copyright

Robust Resource Allocation in Weather Data Processing Systems

2006 International Conference on Parallel Processing Workshops (ICPPW'06)

Reliability of weather data processing systems is of prime importance to ensure the efficient operation of space-based weather monitoring systems. This work defines a heterogeneous weather data processing system that is susceptible to uncertainties in data set arrival times. The resource allocation must be robust with respect to these uncertainties. The tasks to be executed by the data processing system are classified into three broad categories: telemetry, tracking and control (high priority); data processing (medium priority); and data research (low priority). The high priority tasks must be completed before considering medium and low priority tasks. The goal of this research is to find a resource allocation that minimizes makespan of the high priority tasks, and to find a mapping that maximizes a function of the completion time and priority of the medium and low priority tasks. Different heuristic techniques to find near optimal solutions are studied, and their performance is evaluated.

Analysis of Hardware Architecture Markovian Chain Queuing Model for Wireless Sensor Networks

International Journal of Future Generation Communication and Networking, 2017

A wireless Sensor Network (WSN) commonly consumes fairly less energy whereas light and multiple-distributed sensors along with an appropriate wireless network usually come with a lower data rate. An increased system performance along with relatively a higher stability can be achieved by means of applying suitable routing protocol and QoS. QoS represents the effectiveness and robustness of any system concerned. The expansion in relation to productivity of a system may depend on keeping the energy consumption at a controlled level. This paper reports on a study in which quality measurement service was provided by means of analyzing packets in the queuing systems by employing queuing theory. The proposed model is two M/M/1/N queues in tandem with N buffer capacity. The simulation which was performed by MATLAB software indicated that an increase in the service rate of one of the servers while fixing the service rate of another server may result in a waste of hardware resources. Also, the blocking probability for same buffer's size and data arrival rate will not reach less than specific values.

A contribution to modeling sensor communication networks by using finite-source queueing systems

2013 IEEE 8th International Symposium on Applied Computational Intelligence and Informatics (SACI), 2013

In this paper we introduce a finite source retrial queueing model to investigate the performance characteristics of the wireless transmission problem in sensor networks. We divide the sensors into two classes. The first one is the "Emergency" class, which performs the notification of special emergency situations (eg. fire alarms). The second one is the "Normal" class, which measures and transmits environmental data (eg. temperature). For the performance evaluation of the wireless transmission we study and compare two cases: In the first model the RF transmission possibility will be available randomly for the sensor nodes (Non Controlled case) and in the second model the RF transmission requests coming from the emergency class, will access the wireless channel immediately (Controlled case). Our main interest is to give the main steady-state performance measures of the system computed by the help of the MOSEL tool.

Stream processing optimizations for mobile sensing applications

5.6 Simulation of three domains. Domain 1 and 2 use the CPU and have delays of 10 and 20, respectively. Domain 3 uses the network. Domains 1-2 and domain 3 execute in parallel since they use different hardware resources. Domains 1 and 2 use the CPU fairly. 5.7 The energy-delay trade-off for SI and AR when using static sensing. Batching significantly improves energy efficiency. Combining batching with scheduled concurrency provides no additional benefit.

An Evaluation of Data Stream Processing Systems for Data Driven Applications

Procedia Computer Science, 2016

Real-time data stream processing technologies play an important role in enabling time-critical decision making in many applications. This paper aims at evaluating the performance of platforms that are capable of processing streaming data. Candidate technologies include Storm, Samza, and Spark Streaming. To form the recommendation, a prototype pipeline is designed and implemented in each of the platforms using data collected from sensors used in monitoring heavy-haul railway systems. Through the testing and evaluation of each candidate platform, using both quantitative and qualitative metrics, the paper describes the findings, where Storm is found to be the most appropriate candidate.

From Real Time Sensor Streams to Real Time Event Streams

ABSTRACT Sensors are increasingly being deployed for continuous monitoring of physical phenomena, resulting in avalanche of sensor data. Current sensor data streams provide summaries (eg, min., max., avg.) of how phenomena change over time; however, such summaries are of little value to decision makers attempting to attain an insight or an intuitive awareness of the situation.