Bayesian measures of model complexity and fit (original) (raw)
Related papers
1998
We consider the problem of comparing complex hierarchical models in which the number of parameters is not clearly de ned. We follow Dempster in examining the posterior distribution of the log-likelihood under each model, from which we derive measures of t and complexity (the e ective number of parameters). These may be combined into a Deviance Information Criterion (DIC), which is shown to have an approximate decision-theoretic justi cation. Analytic and asymptotic identities reveal the measure of complexity to be a generalisation of a wide range of previous suggestions, with particular reference to the neural network literature. The contributions of individual observations to t and complexity can give rise to a diagnostic plot of deviance residuals against leverages. The procedure is illustrated in a number of examples, and throughout it is emphasised that the required quantities are trivial to compute in a Markov chain Monte Carlo analysis, and require no analytic work for new models.
Both frequentist and Bayesian statistics schools have improved statistical tools and model choices for the collected data or measurements. Model selection approaches have advanced due to the difculty of comparing complicated hierarchical models in which linear predictors vary by grouping variables, and the number of model parameters is not distinct. Many regression model selection criteria are considered, including the maximum likelihood (ML) point estimation of the parameter and the logarithm of the likelihood of the dataset. Tis paper demonstrates the information complexity (ICOMP), Bayesian deviance information, or the widely applicable information criterion (WAIC) of the BRMS to hierarchical linear models ftted with repeated measures with a simulation and two real data examples. Te Fisher information matrix for the Bayesian hierarchical model considering fxed and random parameters under maximizing a posterior estimation is derived. Using Gibbs sampling and Hybrid Hamiltonian Monte Carlo approaches, six diferent models were ftted for three distinct application datasets. Te best-ftted candidate models were identifed under each application dataset with the two MCMC approaches. In this case, the Bayesian hierarchical (mixed efect) linear model with random intercepts and random slopes estimated using the Hamiltonian Monte Carlo method best fts the two application datasets. Information complexity (ICOMP) is a better indicator of the best-ftted models than DIC and WAIC. In addition, the information complexity criterion showed that hierarchical models with gradient-based Hamiltonian Monte Carlo estimation are the best ft and have supper convergence relative to the gradient-free Gibbs sampling methods.
Computational Statistics & Data Analysis, 2010
Many diagnostic tools and goodness-of-fit measures, such as the Akaike information criterion (AIC) and the Bayesian deviance information criterion (DIC), are available to evaluate the overall adequacy of linear regression models. In addition, visually assessing adequacy in models has become an essential part of any regression analysis. In this paper, we focus on a spatial consideration of the local DIC measure for model selection and goodness-of-fit evaluation. We use a partitioning of the DIC into the local DIC, leverage, and deviance residuals to assess local model fit and influence for both individual observations and groups of observations in a Bayesian framework. We use visualization of the local DIC and differences in local DIC between models to assist in model selection and to visualize the global and local impacts of adding covariates or model parameters. We demonstrate the utility of the local DIC in assessing model adequacy using HIV prevalence data from pregnant women in the Butare province of Rwanda during 1989-1993 using a range of linear model specifications, from global effects only to spatially varying coefficient models, and a set of covariates related to sexual behavior. Results of applying the diagnostic visualization approach include more refined model selection and greater understanding of the models as applied to the data.
Goodness-of-Fit Diagnostics for Bayesian Hierarchical Models
Biometrics, 2012
This article proposes methodology for assessing goodness of fit in Bayesian hierarchical models. The methodology is based on comparing values of the pivotal discrepancy measures, computed using parameter values drawn from the posterior distribution, versus known reference distributions. Because the resulting diagnostics can be calculated from standard output of Markov chain Monte Carlo algorithms, their computational costs are minimal. Several simulation studies are provided, each of which suggests that diagnostics based on pivotal discrepancy measures have higher statistical power than comparable posterior-predictive diagnostic checks in detecting model departures. The proposed methodology is illustrated in a clinical application; an application to discrete data is described in supplementary material.
Bayesian case-deletion model complexity and information criterion
Statistics and Its Interface, 2014
We establish a connection between Bayesian case influence measures for assessing the influence of individual observations and Bayesian predictive methods for evaluating the predictive performance of a model and comparing different models fit to the same dataset. Based on such a connection, we formally propose a new set of Bayesian case-deletion model complexity (BCMC) measures for quantifying the effective number of parameters in a given statistical model and its properties in linear models are explored. Adding certain functions of BCMC to a conditional deviance function leads to a Bayesian case-deletion information criterion (BCIC) for comparing models. We systematically investigate some properties of BCIC and its connections with other information criteria, such as the Deviance Information Criterion (DIC). We illustrate the proposed methodology for the linear mixed model with simulations and a real data example.
A Survey of Model Evaluation Approaches With a Tutorial on Hierarchical Bayesian Methods
Cognitive Science: A Multidisciplinary Journal, 2008
This article reviews current methods for evaluating models in the cognitive sciences, including theoretically based approaches, such as Bayes factors and minimum description length measures; simulation approaches, including model mimicry evaluations; and practical approaches, such as validation and generalization measures. This article argues that, although often useful in specific settings, most of these approaches are limited in their ability to give a general assessment of models. This article argues that hierarchical methods, generally, and hierarchical Bayesian methods, specifically, can provide a more thorough evaluation of models in the cognitive sciences. This article presents two worked examples of hierarchical Bayesian analyses to demonstrate how the approach addresses key questions of descriptive adequacy, parameter interference, prediction, and generalization in principled and coherent ways.
ST ] 2 5 A pr 2 01 9 Reference Bayesian analysis for hierarchical models
2019
This paper proposes an alternative approach for constructing invariant Jeffreys prior distributions tailored for hierarchical or multilevel models. In particular, our proposal is based on a flexible decomposition of the Fisher information for hierarchical models which overcomes the marginalization step of the likelihood of model parameters. The Fisher information matrix for the hierarchical model is derived from the Hessian of the Kullback-Liebler (KL) divergence for the model in a neighborhood of the parameter value of interest. Properties of the KL divergence are used to prove the proposed decomposition. Our proposal takes advantage of the hierarchy and leads to an alternative way of computing Jeffreys priors for the hyperparameters and an upper bound for the prior information. While the Jeffreys prior gives the minimum information about parameters, the proposed bound gives an upper limit for the information put in any prior distribution. A prior with information above that limit ...
Average (E)BIC-like Criteria for Bayesian Model Selection
2017
Markov chain Monte Carlo (MCMC) has been an indispensable tool for Bayesian analysis of complex statistical models even for high-dimensional problems. However, there still lacks a consistent criterion for selecting models based on the outputs of MCMC. The existing deviance information criterion (DIC) is known to be inconsistent and non-invariant for reparameterization. This paper proposes an Average BIC-like (ABIC) model selection criterion and an Average EBIC-like (AEBIC) model selection criterion for low and high-dimensional problems, respectively; establishes their consistency under mild conditions; and illustrates their applications using generalized linear models. The proposed criteria overcome shortcomings of DIC. The numerical results indicate that the proposed criteria can significantly outperform DIC as well as the MLE-based criteria, such as AIC, BIC and EBIC, in terms of model selection accuracy.