Variational Inference for Bayesian Bridge Regression (original) (raw)
Related papers
Variational Full Bayes Lasso: Knots Selection in Regression Splines
2021
We develop a fully automatic Bayesian Lasso via variational inference. This is a scalable procedure for approximating the posterior distribution. Special attention is driven to the knot selection in regression spline. In order to carry through our proposal, a full automatic variational Bayesian Lasso, a Jefferey’s prior is proposed for the hyperparameters and a decision theoretical approach is introduced to decide if a knot is selected or not. Extensive simulation studies were developed to ensure the effectiveness of the proposed algorithms. The performance of the algorithms were also tested in some real data sets, including data from the world pandemic Covid-19. Again, the algorithms showed a very good performance in capturing the data structure.
Semiparametric Regression Using Variational Approximations
Journal of the American Statistical Association, 2019
Semiparametric regression offers a flexible framework for modeling non-linear relationships between a response and covariates. A prime example are generalized additive models where splines (say) are used to approximate non-linear functional components in conjunction with a quadratic penalty to control for overfitting. Estimation and inference are then generally performed based on the penalized likelihood, or under a mixed model framework. The penalized likelihood framework is fast but potentially unstable, and choosing the smoothing parameters
Statistics and Computing, 2009
During recent years, penalized likelihood approaches have attracted a lot of interest both in the area of semiparametric regression and for the regularization of high-dimensional regression models. In this paper, we introduce a Bayesian formulation that allows to combine both aspects into a joint regression model with a focus on hazard regression for survival times. While Bayesian penalized splines form the basis for estimating nonparametric and flexible time-varying effects, regularization of highdimensional covariate vectors is based on scale mixture of normals priors. This class of priors allows to keep a (conditional) Gaussian prior for regression coefficients on the predictor stage of the model but introduces suitable mixture distributions for the Gaussian variance to achieve regularization. This scale mixture property allows to device general and adaptive Markov chain Monte Carlo simulation algorithms for fitting a variety of hazard regression models. In particular, unifying algorithms based on iteratively weighted least squares proposals can be employed both for regularization and penalized semiparametric function estimation. Since sampling based estimates do no longer have the variable selection property well-known for the Lasso in frequentist analyses, we additionally consider spike and slab priors that introduce a further mixing stage that allows to separate between influential and redundant parameters. We demonstrate the different shrinkage properties with three simulation settings and apply the methods to the PBC Liver dataset.
Bayesian varying-coefficient models using adaptive regression splines
Statistical Modelling, 2001
Varying-coefficient models provide a flexible framework for semi-and nonparametric generalized regression analysis. We present a fully Bayesian B-spline basis function approach with adaptive knot selection. For each of the unknown regression functions or varying coefficients, the number and location of knots and the B-spline coefficients are estimated simultaneously using reversible jump Markov chain Monte Carlo sampling. The overall procedure can therefore be viewed as a kind of Bayesian model averaging. Although Gaussian responses are covered by the general framework, the method is particularly useful for fundamentally non-Gaussian responses, where less alternatives are available. We illustrate the approach with a thorough application to two data sets analysed previously in the literature: the kyphosis data set with a binary response and survival data from the Veteran's Administration lung cancer trial.
Smoothing Spline in Semiparametric Additive Regression Model with Bayesian Approach
Journal of Mathematics and Statistics, 2013
Semiparametric additive regression model is a combination of parametric and nonparametric regression models. The parametric components are not linear but following a polynomial pattern, while the nonparametric components are unknown pattern and assumed to be contained in the Sobolev space. The nonparametric components can be approximated by smoothing spline functions. In the development of smoothing spline, the classical statistical approach cannot be applied for solving the inference problem such as constructing confidence intervals for the regression curve. To construct confidence interval of smoothing spline curve in the semiparametric additive regression model, we propose to use Bayesian approach, by assuming improper Gaussian distribution for prior distribution in nonparametric components and multivariate normal distribution for parametric components. In this study, we obtain parameter estimators for parametric component and smoothing spline estimators for the nonparametric component in semiparametric additive regression model. Moreover, we also develop a smoothing parameters selection method simultaneously using Generalized Maximum Likelihood (GML) and confidence intervals for the parameters of the parametric component and the smoothing spline functions of the nonparametric component using Bayesian approach. By computing each posterior mean and posterior variance of parametric component parameters and smoothing spline functions, confidence intervals can be constructed for the parametric component parameters and confidence interval smoothing spline functions for nonparametric components in semiparametric additive regression models. We create R-code to implement estimation model and inference procedure. Our simulation studies reveal estimation and inference method perform reasonably well.
Scalable and Accurate Variational Bayes for High-Dimensional Binary Regression Models
arXiv: Methodology, 2019
State-of-the-art methods for Bayesian inference on regression models with binary responses are either computationally impractical or inaccurate in high dimensions. To cover this gap we propose a novel variational approximation for the posterior distribution of the coefficients in high-dimensional probit regression with Gaussian priors. Our method leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for the local variables. We prove that the resulting variational approximation belongs to a tractable class of unified skew-normal distributions that preserves the skewness of the actual posterior and, unlike for state-of-the-art variational Bayes solutions, converges to the exact posterior as the number of predictors p increases. A scalable coordinate ascent variational algorithm is proposed to obtain the optimal parameters of the approximating dens...
Semiparametric Bayesian inference for regression models
Canadian Journal of Statistics, 1999
This paper presents a method for Bayesian inference for the regression parameters in a linear model with independent and identically distributed errors that does not require the specification of a parametric family of densities for the error distribution. This method first selects a nonparametric kernel density estimate of the error distribution which is unimodal and based on the least-squares residuals. Once the error distribution is selected, the Metropolis algorithm is used to obtain the marginal posterior distribution of the regression parameters. The methodology is illustrated with data sets, and its performance relative to standard Bayesian techniques is evaluated using simulation results.
A variational Bayes approach to a semiparametric regression using Gaussian process priors
Electronic Journal of Statistics, 2017
This paper presents a variational Bayes approach to a semiparametric regression model that consists of parametric and nonparametric components. The assumed univariate nonparametric component is represented with a cosine series based on a spectral analysis of Gaussian process priors. Here, we develop fast variational methods for fitting the semiparametric regression model that reduce the computation time by an order of magnitude over Markov chain Monte Carlo methods. Further, we explore the possible use of the variational lower bound and variational information criteria for model choice of a parametric regression model against a semiparametric alternative. In addition, variational methods are developed for estimating univariate shape-restricted regression functions that are monotonic, monotonic convex or monotonic concave. Since these variational methods are approximate, we explore some of the trade-offs involved in using them in terms of speed, accuracy and automation of the implementation in comparison with Markov chain Monte Carlo methods and discuss their potential and limitations.
Asymptotically Exact Variational Bayes for High-Dimensional Binary Regression Models
2019
State-of-the-art methods for Bayesian inference on regression models with binary responses are either computationally impractical or inaccurate in high dimensions. To cover this gap we propose a novel variational approximation for the posterior distribution of the coefficients in high-dimensional probit regression. Our method leverages a representation with global and local variables but, unlike for classical mean-field assumptions, it avoids a fully factorized approximation, and instead assumes a factorization only for the local variables. We prove that the resulting variational approximation belongs to a tractable class of unified skew-normal distributions that preserves the skewness of the actual posterior and, unlike for state-of-the-art variational Bayes solutions, converges to the exact posterior as the number of predictors p increases. A scalable coordinate ascent variational algorithm is proposed to obtain the optimal parameters of the approximating densities. As we show wit...
Smoothing spline Gaussian regression: more scalable computation via efficient approximation
Journal of the Royal Statistical Society: Series B (Statistical Methodology), 2004
Smoothing splines via the penalized least squares method provide versatile and effective nonparametric models for regression with Gaussian responses. The computation of smoothing splines is generally of the order O(n 3), n being the sample size, which severely limits its practical applicability. In this article, we study more scalable computation of smoothing spline regression via certain low-dimensional approximations that are asymptotically as efficient. A simple algorithm is presented and the Bayes model associated with the approximations is derived, with the latter guiding the porting of Bayesian confidence intervals. The practical choice of the dimension of the approximating space is determined through simulation studies, and empirical comparisons of the approximations with the exact solution are presented. Also evaluated is a simple modification of the generalized cross-validation method for smoothing parameter selection, which to a large extent fixes the occasional undersmoothing problem suffered by generalized cross-validation.