Stream Data Cleaning for Dynamic Line Rating Application (original) (raw)

Automated Load Curve Data Cleansing in Power Systems

IEEE Transactions on Smart Grid, 2010

Load curve data refers to the electric energy consumption recorded by meters at certain time intervals at delivery points or end user points, and contains vital information for day-today operations, system analysis, system visualization, system reliability performance, energy saving and adequacy in system planning. Unfortunately, it is unavoidable that load curves contain corrupted data and missing data due to various random failure factors in meters and transfer processes. This paper presents the B-Spline smoothing and Kernel smoothing based techniques to automatically cleanse corrupted and missing data. In implementation, a man-machine dialogue procedure is proposed to enhance the performance. The experiment results on the real British Columbia Transmission Corporation (BCTC) load curve data demonstrated the effectiveness of the presented solution.

Learning from power system data stream

2019

Assuming access to synchronized stream of Phasor Measurement Unit (PMU) data over a significant portion of a power system interconnect, say controlled by an Independent System Operator (ISO), what can you extract about past, current and future state of the system? We have focused on answering these practical questions pragmatically-empowered with nothing but standard tools of data analysis, such as PCA, filtering and cross-correlation analysis. Quite surprisingly we have found that even during quiet "no significant events" periods this standard set of statistical tools allows the "phasor-detective" to extract from the data important hidden anomalies, such as problematic control loops at loads and wind farms, and mildly malfunctioning assets, such as transformers and generators. We also discuss and sketch future challenges a mature phasor-detective can possibly tackle by adding machine learning and physics modeling sophistication to the basic approach.

Association Rule Mining to Understand GMDs and their Effects on Power Systems

—Control room operators play an important role in mitigating the effects of disturbances on the power grid. We focus on solar storms and geomagnetic disturbances in this paper. Providing operators with advanced tools that reveal relationships between variables that characterize events in real-time enables faster response. For complex events such as GIC (geomagnetically induced currents), the output of the tool should validate domain knowledge to build trust with the operator. In this paper, we apply association rule mining to discover relationships between physical variables from multiple sources of data relevant to GMDs. We aligned features extracted from ACE (Advanced Composition Explorer) satellite measurements with features extracted from terrestrial magnetometer measurements. We mapped these features during solar storms in 2015 to GIC grid event data processed from synchrophasor streams. Then, we discovered relationships or rules between features relevant for predicting the effects of solar storms on the grid and evaluated our results on the 2015 data. By looking at the predictive value of selected features, we find that features most relevant to GIC vary depending on the prediction latency, reflecting the complex, physical dynamics of GMDs.

A Data-Mining Model for Protection of FACTS-Based Transmission Line

This paper presents a data-mining model for fault-zone identification of a flexible ac transmission systems (FACTS)-based transmission line including a thyristor-controlled series compensator (TCSC) and unified power-flow controller (UPFC), using ensemble decision trees. Given the randomness in the ensemble of decision trees stacked inside the random forests model, it provides effective decision on fault-zone identification. Half-cycle postfault current and voltage samples from the fault inception are used as an input vector against target output "1" for the fault after TCSC/UPFC and " 1" for the fault before TCSC/UPFC for fault-zone identification. The algorithm is tested on simulated fault data with wide variations in operating parameters of the power system network, including noisy environment providing a reliability measure of 99% with faster response time (3/4th cycle from fault inception). The results of the presented approach using the RF model indicate reliable identification of the fault zone in FACTS-based transmission lines. Index Terms-Distance relaying, fault-zone identification, random forests (RFs), support vector machine (SVM), thyristor-controlled series compensator (TCSC), unified power-flow controller (UPFC). S. R. Samantaray (M'08-SM'10) received the B.

Household Electricity Consumption Data Cleansing

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present a new appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as Bspline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also demonstrate that our method is robust in various tests. Our method provides a highly feasible and reliable solution to an emerging industry application.

An Appliance-Driven Approach to Detection of Corrupted Load Curve Data

Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, 2014

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present an appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as Bspline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also show that our method is robust in various tests.

Is My Electricity Bill Accurate? A Model-Driven Approach to Corrupted Load Data Identification

Load curve data in power systems refers to users' electrical energy consumption data periodically collected with meters. It has become one of the most important assets for modern power systems. Many operational decisions are made based on the information discovered in the data. Load curve data, however, usually suffers from corruptions caused by various factors, such as data transmission errors or malfunctioning meters. To solve the problem, tremendous research efforts have been made on load curve data cleansing. Most existing approaches apply outlier detection methods from the supply side (i.e., electricity service providers), which may only have aggregated load data. In this paper, we propose to seek aid from the demand side (i.e., electricity service users). With the help of readily available knowledge on consumers' appliances, we present a new appliance-driven approach to load curve data cleansing. This approach utilizes data generation rules and a Sequential Local Optimization Algorithm (SLOA) to solve the Corrupted Data Identification Problem (CDIP). We evaluate the performance of SLOA with real-world trace data and synthetic data. The results indicate that, comparing to existing load data cleansing methods, such as Bspline smoothing, our approach has an overall better performance and can effectively identify consecutive corrupted data. Experimental results also demonstrate that our method is robust in various tests. Our method provides a highly feasible and reliable solution to an emerging industry application.

Bad Data Detection and Data Filtering in Power System

International Journal of Computer Applications, 2018

With increase in advanced metering infrastructure and sensor systems there is increase in data collection. It is hard to handle a large amount of data and assure the quality of data. Good quality of data is essential in power system before taking decision. So data must be cleaned and filtered before operator takes any decision from the data. Otherwise it will cause hazardous condition if poor quality of data affects decision making without knowledge of operator. Bad Data detection and data cleaning is helpful to get over this risk. With use of MATLAB Bad Data can be easily detected. Bad Data can be also removed and Data filtering as well as Data smoothing is also possible. Data smoothing is necessary for some application ex. Load forecasting in power system. Here it is obtained by using Statistical techniques such as OWA (Optimally Weighted Average) and MA (Moving Average).

Data Analysis for Smart Grid and Communication Technologies

2023

Smart grids and communication technologies are among the significant advancements in the energy sector. These technologies improve energy efficiency by increasing the use of renewable energy sources. However, the integration of these technologies also creates a vast amount of data that requires analysis to optimize energy management and improve system performance. In this context, data analysis plays a crucial role in identifying patterns, predicting energy demand, and managing energy supply. This paper provides an overview of the importance of data analysis in smart grids and communication technologies and discusses various data analysis techniques used in energy management systems. Additionally, it highlights the challenges and opportunities associated with data analysis in the context of smart grids and communication technologies.