Shrinkage and penalized estimators in weighted least absolute deviations regression models (original) (raw)
Related papers
Weighted LAD-LASSO method for robust parameter estimation and variable selection in regression
Computational Statistics & Data Analysis, 2012
The weighted least absolute deviation (WLAD) regression estimation method and the adaptive least absolute shrinkage and selection operator (LASSO) are combined to achieve robust parameter estimation and variable selection in regression simultaneously. Compared with the LAD-LASSO method, the weighted LAD-LASSO (WLAD-LASSO) method will resist to the heavy-tailed errors and outliers in explanatory variables. Properties of the WLAD-LASSO estimators are investigated. A small simulation study and an example are provided to demonstrate the superiority of the WLAD-LASSO method over the LAD-LASSO method in the presence of outliers in the explanatory variables and the heavy-tailed error distribution.
International Journal of Advanced Statistics and Probability, 2014
Some few decades ago, penalized regression techniques for linear regression have been developed specifically to reduce the flaws inherent in the prediction accuracy of the classical ordinary least squares (OLS) regression technique. In this paper, we used a diabetes data set obtained from previous literature to compare three of these well-known techniques, namely: Least Absolute Shrinkage Selection Operator (LASSO), Elastic Net and Correlation Adjusted Elastic Net (CAEN). After thorough analysis, it was observed that CAEN generated a less complex model.
Preliminary test and Stein-type shrinkage LASSO-based estimators
2018
Suppose the regression vector-parameter is subjected to lie in a subspace hypothesis in a linear regression model. In situations where the use of least absolute and shrinkage selection operator (LASSO) is desired, we propose a restricted LASSO estimator. To improve its performance, LASSO-type shrinkage estimators are also developed and their asymptotic performance is studied. For numerical analysis, we used relative efficiency and mean prediction error to compare the estimators which resulted in the shrinkage estimators to have better performance compared to the LASSO
The International Journal of Biostatistics, 2020
The Highly-Adaptive least absolute shrinkage and selection operator (LASSO) Targeted Minimum Loss Estimator (HAL-TMLE) is an efficient plug-in estimator of a pathwise differentiable parameter in a statistical model that at minimal (and possibly only) assumes that the sectional variation norm of the true nuisance functions (i.e., relevant part of data distribution) are finite. It relies on an initial estimator (HAL-MLE) of the nuisance functions by minimizing the empirical risk over the parameter space under the constraint that the sectional variation norm of the candidate functions are bounded by a constant, where this constant can be selected with cross-validation. In this article we establish that the nonparametric bootstrap for the HAL-TMLE, fixing the value of the sectional variation norm at a value larger or equal than the cross-validation selector, provides a consistent method for estimating the normal limit distribution of the HAL-TMLE. In order to optimize the finite sample ...
Shrinkage reweighted regression
2019
A robust estimator is proposed for the parameters that characterize the linear regression problem. It is based on the notion of shrinkages, often used in Finance and previously studied for outlier detection in multivariate data. A thorough simulation study is conducted to investigate: the efficiency with normal and heavy-tailed errors, the robustness under contamination, the computational times, the affine equivariance and breakdown value of the regression estimator. Two classical data-sets often used in the literature and a real socio-economic data-set about the Living Environment Deprivation of areas in Liverpool (UK), are studied. The results from the simulations and the real data examples show the advantages of the proposed robust estimator in regression.
Robust weighted LAD regression
Computational Statistics & Data Analysis, 2006
The least squares linear regression estimator is well-known to be highly sensitive to unusual observations in the data, and as a result many more robust estimators have been proposed as alternatives. One of the earliest proposals was least-sum of absolute deviations (LAD) regression, where the regression coefficients are estimated through minimization of the sum of the absolute values of the residuals. LAD regression has been largely ignored as a robust alternative to least squares, since it can be strongly affected by a single observation (that is, it has a breakdown point of 1/n, where n is the sample size). In this paper we show that judicious choice of weights can result in a weighted LAD estimator with much higher breakdown point. We discuss the properties of the weighted LAD estimator, and show via simulation that its performance is competitive with that of high breakdown regression estimators, particularly in the presence of outliers located at leverage points. We also apply the estimator to several data sets.
A greedy regression algorithm with coarse weights offers novel advantages
Scientific Reports, 2022
Regularized regression analysis is a mature analytic approach to identify weighted sums of variables predicting outcomes. We present a novel Coarse Approximation Linear Function (CALF) to frugally select important predictors and build simple but powerful predictive models. CALF is a linear regression strategy applied to normalized data that uses nonzero weights + 1 or − 1. Qualitative (linearly invariant) metrics to be optimized can be (for binary response) Welch (Student) t-test p-value or area under curve (AUC) of receiver operating characteristic, or (for real response) Pearson correlation. Predictor weighting is critically important when developing risk prediction models. While counterintuitive, it is a fact that qualitative metrics can favor CALF with ± 1 weights over algorithms producing real number weights. Moreover, while regression methods may be expected to change most or all weight values upon even small changes in input data (e.g., discarding a single subject of hundreds) CALF weights generally do not so change. Similarly, some regression methods applied to collinear or nearly collinear variables yield unpredictable magnitude or the direction (in p-space) of the weights as a vector. In contrast, with CALF if some predictors are linearly dependent or nearly so, CALF simply chooses at most one (the most informative, if any) and ignores the others, thus avoiding the inclusion of two or more collinear variables in the model. Regularized regression modeling strategies for choosing variables and weights are mature and extensively documented 1. Ordinary (linear) least-squares (OLS) minimizes the mean of squared differences (MSE) between the N real entries of a targeted response vector versus a weighted linear combination of p predictors, that is, the matrix product Xβ where X is a N-by-p matrix and β is the weight (column) vector. Seeking optimal weights for OLS dates to the early 1800s and works of Legendre and Gauss. However, OLS can provide uninterpretable and unstable weights when exact or approximate collinearity exists between predictors or when problems are underdetermined (i.e., N << p). Regularized regression may employ Lagrange multipliers to constrain the optimization; examples include Tikhonov regularization (ridge regression, using constrained L2-norm), least absolute shrinkage and selection operator (basic LASSO, using constrained L1-norm), and elastic net (using constrained convex sum of the L1-and L2-norms) 1-4. Each method has its advantages and disadvantages. As one advantage, LASSO regression with L1-norm includes choice of a parameter called s that can indirectly force a solution to have few nonzero weights, enabling cross-referencing of selected predictors and a search for underlying meaning among them 5. Thus, discovering small sets of collectively informative predictors may suggest causal networks. However, regarding instability due to predictors that are collinear or nearly so, various classifiers including LASSO regression are susceptible and must be modified accordingly 6. But overall, the widely acknowledged value and general use of LASSO algorithms 7-11 recommends comparisons with LASSO as the "gold standard". We present "Coarse Approximation Linear Function" (CALF), an algorithm that, applied to some examples of real data, builds models with frugal use of predictors, superior qualitative metric values versus LASSO, as well as superior permutation test performance. CALF inherently precludes collinearity issues and provides better consistency among selected predictors when applied ~ 1000 times to, say, random 90% subsets of subjects, a test called herein "popularities". These properties could yield relatively simple causal interpretation of the interplay of predictors. Furthermore, the improvements in some cases could mean the difference between finding statistical significance or noting a mere trend.
Variable Selection via Biased Estimators in the Linear Regression Model
Open Journal of Statistics
Least Absolute Shrinkage and Selection Operator (LASSO) is used for variable selection as well as for handling the multicollinearity problem simultaneously in the linear regression model. LASSO produces estimates having high variance if the number of predictors is higher than the number of observations and if high multicollinearity exists among the predictor variables. To handle this problem, Elastic Net (ENet) estimator was introduced by combining LASSO and Ridge estimator (RE). The solutions of LASSO and ENet have been obtained using Least Angle Regression (LARS) and LARS-EN algorithms, respectively. In this article, we proposed an alternative algorithm to overcome the issues in LASSO that can be combined LASSO with other exiting biased estimators namely Almost Unbiased Ridge Estimator (AURE), Liu Estimator (LE), Almost Unbiased Liu Estimator (AULE), Principal Component Regression Estimator (PCRE), r-k class estimator and r-d class estimator. Further, we examine the performance of the proposed algorithm using a Monte-Carlo simulation study and real-world examples. The results showed that the LARS-rk and LARS-rd algorithms, which are combined LASSO with r-k class estimator and r-d class estimator, outperformed other algorithms under the moderated and severe multicollinearity.
Shrinkage, pretest, and penalty estimators in generalized linear models
Statistical Methodology, 2015
We consider estimation in generalized linear models when there are many potential predictors and some of them may not have influence on the response of interest. In the context of two competing models where one model includes all predictors and the other restricts variable coefficients to a candidate linear subspace based on subject matter or prior knowledge, we investigate the relative performances of Stein type shrinkage, pretest, and penalty estimators (L 1 GLM, adaptive L 1 GLM, and SCAD) with respect to the unrestricted maximum likelihood estimator (MLE). The asymptotic properties of the pretest and shrinkage estimators including the derivation of asymptotic distributional biases and risks are established. In particular, we give conditions under which the shrinkage estimators are asymptotically more efficient than the unrestricted MLE. A Monte Carlo simulation study shows that the mean squared error (MSE) of an adaptive shrinkage estimator is comparable to the MSE of the penalty estimators in many situations and in particular performs better than the penalty estimators when the dimension of the restricted parameter space is large. The Steinian shrinkage and penalty estimators all improve substantially on the unrestricted MLE. A real data set analysis is also presented to compare the suggested methods.
Penalized Regression Models with Autoregressive Error Terms
Penalized regression methods have recently gained enormous attention in statistics and the field of machine learning due to their ability of reducing the prediction error and identifying important variables at the same time. Numerous studies have been conducted for penalized regression , but most of them are limited to the case when the data are independently observed. In this paper, we study a variable selection problem in penalized regression models with autoregres-sive error terms. We consider three estimators, adaptive LASSO (Least Absolute Shrinkage and Selection Operator), bridge, and SCAD (Smoothly Clipped Absolute Deviation), and propose a computational algorithm that enables us to select a relevant set of variables and also the order of autoregressive error terms simultaneously. In addition, we provide their asymptotic properties such as consistency, selection consistency, and asymptotic normality. The performances of the three estimators are compared one another using simulated and real examples.