Outlier measurement analysis with the robust estimation (original) (raw)
Related papers
Parameter estimation and outlier detection with different estimation methods
The best suitable values of unknown parameters are determined by adjustment of observations done more than the number of unknown parameter in applied sciences. Adjustment methods should determine the outlier, that inevitably exists in the observations, while doing estimation of the best appropriate parameter for the unknown parameters. The adjustment is made on the basis of an objective functions with the best parameter estimation from the mathematical model, written for the observations done more than the number of unknown parameters. The most applied methods used for adjustment are Least Square (LS), Least Absolute Values (LAV) and Total Least Square (TLS) Methods. Although there are many advantages of these methods, the LS method also has some disadvantages, such as more affected from gross observation errors in the observation and spread of gross error in other observations. The solution of LAV method and the results obtained with trial and error method for the parameter estimation are less affected by gross observation error; and this method is used successfully in removing the outlier. Besides, with these methods in recent years, the TLS method is used for the adjustment. In this method, the design matrix used for the solution is also erroneous and the residuals of observation and design matrix are calculated together in the solution. In this study, after three adjustment methods have been explained, the parameter estimation and outlier detection are made with using the application data. The success of adjustment methods for parameter estimation and outlier detection had been determined as well by examining the results of these methods.
The proposition of a new damping function for outliers in the adjustment process
Geomatics, Landmanagement and Landscape, 2015
This study proposes of new damping function in the robust estimation process. The formulae for a behaviour of the damping function were provided and a diagram of adjustment process was presented. The numerical examples are given in order to assess the efficiency of the proposed computational algorithm in accomplishing a typical geodetic task with outliers or gross errors.
On the effects of hard and soft equality constraints in the iterative outlier elimination procedure
PLOS ONE, 2020
Reliability analysis allows for the estimation of a system's probability of detecting and identifying outliers. Failure to identify an outlier can jeopardize the reliability level of a system. Due to its importance, outliers must be appropriately treated to ensure the normal operation of a system. System models are usually developed from certain constraints. Constraints play a central role in model precision and validity. In this work, we present a detailed investigation of the effects of the hard and soft constraints on the reliability of a measurement system model. Hard constraints represent a case in which there exist known functional relations between the unknown model parameters, whereas the soft constraints are employed where such functional relations can be slightly violated depending on their uncertainty. The results highlighted that the success rate of identifying an outlier for the case of hard constraints is larger than soft constraints. This suggested that hard constraints be used in the stage of pre-processing data for the purpose of identifying and removing possible outlying measurements. After identifying and removing possible outliers, one should set up the soft constraints to propagate their uncertainties to the model parameters during the data processing.
A Modified M-estimator for the Detection of Outliers
Pakistan Journal of Statistics and Operation Research, 2005
A new ρ-function is proposed in the family of smoothly redescending M-estimators. The ψ-function associated with this new ρ-function attains much more linearity in its central section before it redescends, compared to other ψ-functions such as those of Andrews sine, Tukey's biweight and Qadir's beta function resulting in its enhanced efficiency. The iteratively reweighted least squares (IRLS) method based on the proposed ρ-function clearly detects outliers and ignoring those outliers refines the subsequent analysis. Three examples selected from the relevant literature, are used for illustrative purposes. A comparative simulation study has been conducted to evaluate its general applications. The proposed weighted least squares (WLS) method indeed achieves the goals for which it is constructed, for it gives quite improved results in all situations and is able to withstand substantial amount of outliers.
SCIENCE NATURE, 2018
A widely used estimation method in estimating regression model parameters is the ordinary least square (OLS) that minimizes the sum of the error squares. In addition to the ease of computing, OLS is a good unbiased estimator as long as the error component assumption () in the given model is met. However, in the application, it is often encountered violations of assumptions. One of the violation types is the violation of distributed error assumption which is caused by the existence of the outlier on observation data. Thus, a solid method is required to overcome the existence of outlier, that is Robust Regression. One of the Robust Regression methods commonly used is robust MM method. Robust MM method is a combination of breakdown point and high efficiency. Results obtained based on simulated data generated using SAS software 9.2, shows that the use of objective weighting function tukey bisquare is able to overcome the existence of extreme outlier. Furthermore, it is determined that ...
A method of robust multivariate outlier replacement
Computers & Chemical Engineering, 2002
Robust multivariate methods for dealing with problems caused by outliers in the data are essential especially when process data are used to validate mechanistic models, develop regression models, and in applications such as controller design and process monitoring. Gross outliers are detected easily by simple methods such as range checking, however, a multivariate outlier is very difficult to discern and techniques that rely on data to generate empirical models may produce erroneous results. In this work, a methodology to perform multivariate outlier replacement in the score space generated by principal component analysis (PCA) is proposed. The objective was to find an accurate estimate of the covariance matrix of the data so that a PCA model might be developed that could then be used for monitoring and fault detection and identification. The methodology uses the concept of winsorization to provide robust estimates of the mean (location) and S.D. (scale) iteratively, yielding a robust set of data. The paper develops the approach, discusses the concept of robust statistics and winsorization, and presents the procedures for robust multivariate outlier filtering. One simulated and two industrial examples are provided to demonstrate the approach.
Identification of Outliers: A Simulation Study
2015
This paper compares two approaches in identifying outliers in multivariate datasets; Mahalanobis distance (MD) and robust distance (RD). MD has been known suffering from masking and swamping effects and RD is an approach that was developed to overcome problems that arise in MD. There are two purposes of this paper, first is to identify outliers using MD and RD and the second is to show that RD performs better than MD in identifying outliers. An observation is classified as an outlier if MD or RD is larger than a cut-off value. Outlier generating model is used to generate a set of data and MD and RD are computed from this set of data. The results showed that RD can identify outliers better than MD. However, in non-outliers data the performance for both approaches are similar. The results for RD also showed that RD can identify multivariate outliers much better when the number of dimension is large.
A Modified Approach for Detection of Outliers
Tukey's boxplot is very popular tool for detection of outliers. It reveals the location, spread and skewness of the data. It works nicely for detection of outliers when the data are symmetric. When the data are skewed it covers boundary away from the whisker on the compressed side while declares erroneous outliers on the extended side of the distribution. Hubert and Vandervieren (2008) made adjustment in Tukey's technique to overcome this problem. However another problem arises that is the adjusted boxplot constructs the interval of critical values which even exceeds from the extremes of the data. In this situation adjusted boxplot is unable to detect outliers. This paper gives solution of this problem and proposed approach detects outliers properly. The validity of the technique has been checked by constructing fences around the true 95% values of different distributions. Simulation technique has been applied by drawing different sample size from chi square, beta and lognormal distributions. Fences constructed by the modified technique are close to the true 95% than adjusted boxplot which proves its superiority on the existing technique.
A Review of various statestical methods for Outlier Detection
2014
— Outlier Detection has many applications. Outlier is instance of the data set which has exceptional characteristics compare to other instance of data and exhibits unusual behavior of the system. There are many methods used for detecting outliers. Every method has its advantages and limitations. In this review paper a relative comparison of few statistical methods is carried out. This shows which method is more efficient in detecting outlier. Keywords-outlier detection methods; mean; standard deviation; median absolute deviation; clever variance and clever mean I.
Methods for outlier detection in prediction
Chemometrics and Intelligent Laboratory Systems, 2002
If a prediction sample is different from the calibration samples, it can be considered as an outlier in prediction. In this work, two techniques, the use of the uncertainty estimation and convex hull method are studied to detect such prediction outliers. Classical techniques (Mahalanobis distance and X-residuals), potential functions and robust techniques are used for comparison. It is concluded that the combination of the convex hull and the uncertainty estimation offers a practical way for detecting outliers in prediction. By adding the potential function method, inliers can also be detected. D