Bayesian Model Selection in Finite Mixtures by Marginal Density Decompositions (original) (raw)

Gibbs Sampling Based Bayesian Analysis of Mixtures with Unknown Number of Components

For mixture models with unknown number of components, Bayesian approaches, as considered by and , are reconciled here through a simple Gibbs sampling approach. Specifically, we consider exactly the same direct set up as used by , but put Dirichlet process prior on the mixture components; the latter has also been used by albeit in a different set up. The reconciliation we propose here yields a simple Gibbs sampling scheme for learning about all the unknowns, including the unknown number of components. Thus, we completely avoid complicated reversible jump Markov chain Monte Carlo (RJMCMC) methods, yet tackle variable dimensionality simply and efficiently. Moreover, we demonstrate, using both simulated and real data sets, and pseudo-Bayes factors, that our proposed model outperforms that of , while enjoying, at the same time, computational superiority over the methods proposed by and . We also discuss issues related to clustering and argue that in principle, our approach is capable of learning about the number of clusters in the sample as well as in the population, while the approach of is suitable for learning about the number of clusters in the sample only.

A practical sampling approach for a Bayesian mixture model with unknown number of components

Statistical Papers, 2007

Recently, mixture distribution becomes more and more popular in many scientific fields. Statistical computation and analysis of mixture models, however, are extremely complex due to the large number of parameters involved. Both EM algorithms for likelihood inference and MCMC procedures for Bayesian analysis have various difficulties in dealing with mixtures with unknown number of components. In this paper, we propose a direct sampling approach to the computation of Bayesian finite mixture models with varying number of components. This approach requires only the knowledge of the density function up to a multiplicative constant. It is easy to implement, numerically efficient and very practical in real applications. A simulation study shows that it performs quite satisfactorily on relatively high dimensional distributions. A well-known genetic data set is used to demonstrate the simplicity of this method and its power for the computation of high dimensional Bayesian mixture models.

A default Bayesian test for the number of components in a mixture

Journal of Statistical Planning and Inference, 2003

In the last few years, there has been an increasing interest for default Bayes methods for hypothesis testing and model selection. The availability of such methods is potentially very useful in mixture models, where the elicitation process on the (unknown number of) parameters is usually rather di cult. Two recent yet already popular approaches, namely intrinsic Bayes factor (J. Amer. Statist. Assoc. 91 (1996) 109) and fractional Bayes factor (J. Roy. Statist. Soc. Ser. B 57 (1995) 99), have been proven quite successful in generating sensible prior distributions, to compute actual Bayes factors. From a theoretical viewpoint, the application of these methods to a mixture model selection problem involves two di culties. The ÿrst is the choice of a "good" default prior for the mixture. The second problem is related to the fact that, for improper default priors, the prior predictive distribution of the data need not exist; In this paper, we argue that the problem of choosing among mixture models can be reduced to the problem of comparing models with simpler structures. It is shown that these simpler models can be compared via standard default Bayesian methods.

A Bayesian predictive approach to determining the number of components in a mixture distribution

Statistics and Computing, 1995

This paper describes a Bayesian approach to mixture modelling and a method based on predictive distribution to determine the number of components in the mixtures. The implementation is done through the use of the Gibbs sampler. The method is described through the mixtures of normal and gamma distributions. Analysis is presented in one simulated and one real data example. The Bayesian results are then compared with the likelihood approach for the two examples.

Corrigendum: On Bayesian analysis of mixtures with an unknown number of components

Journal of the Royal Statistical Society: Series B (Statistical Methodology), 1998

New methodology for fully Bayesian mixture analysis is developed, making use of reversible jump Markov chain Monte Carlo methods, that are capable of jumping between the parameter subspaces corresponding to di erent numbers of components in the mixture. A sample from the full joint distribution of all unknown variables is thereby generated, and this can be used as a basis for a thorough presentation of many aspects of the posterior distribution. The methodology is applied here to the analysis of univariate normal mixtures, using a hierarchical prior model that o ers an approach to dealing with weak prior information while avoiding the mathematical pitfalls of using improper priors in the mixture context.

Bayesian density estimation and model selection using nonparametric hierarchical mixtures

Computational Statistics & Data Analysis, 2010

We consider mixtures of parametric densities on the positive reals with a normalized generalized gamma process (Brix, 1999) as mixing measure. This class of mixtures encompasses the Dirichlet process mixture (DPM) model, but it is supposedly more flexible in the detection of clusters in the data. With an almost sure approximation of the posterior distribution of the mixing process we can run a Markov chain Monte Carlo algorithm to estimate linear and nonlinear functionals of the predictive distributions. The best-fitting mixing measure is found by minimizing a Bayes factor for parametric against non-parametric alternatives. We illustrate the method with simulated and hystorical data, finding a tradeoff between the best-fitting model and the correct identification of the number of components in the mixture.

Approximate Bayesian computation for finite mixture models

Journal of Statistical Computation and Simulation, 2020

Finite mixture models are used in statistics and other disciplines, but inference for mixture models is challenging. The multimodality of the likelihood function and the so called label switching problem contribute to the challenge. We propose extensions of the Approximate Bayesian Computation Population Monte-Carlo (ABC-PMC) algorithm as an alternative framework for inference on finite mixture models. There are several decisions to make when implementing an ABC-PMC algorithm for finite mixture models, including the selection of the kernel used for moving the particles through the iterations, how to address the label switching problem and the choice of informative summary statistics. Examples are presented to demonstrate the performance of the proposed ABC-PMC algorithm for mixture modeling.

Performance of Bayesian Model Selection Criteria for Gaussian Mixture Models

2009

Bayesian methods are widely used for selecting the number of components in a mixture models, in part because frequentist methods have difficulty in addressing this problem in general. Here we compare some of the Bayesianly motivated or justifiable methods for choosing the number of components in a one-dimensional Gaussian mixture model: posterior probabilities for a well-known proper prior, BIC, ICL, DIC and AIC. We also introduce a new explicit unit-information prior for mixture models, analogous to the prior to which BIC corresponds in regular statistical models. We base the comparison on a simulation study, designed to reflect published estimates of mixture model parameters from the scientific literature across a range of disciplines. We found that BIC clearly outperformed the five other methods, with the maximum a posteriori estimate from the established proper prior second.

Identifying Mixtures of Mixtures Using Bayesian Estimation

Journal of Computational and Graphical Statistics, 2016

The use of a finite mixture of normal mixtures model in model-based clustering allows to capture non-Gaussian data clusters. However, identifying the clusters from the normal components is challenging and in general either achieved by imposing constraints on the model or by using post-processing procedures.

Overfitting Bayesian Mixture Models with an Unknown Number of Components

This paper proposes solutions to three issues pertaining to the estimation of finite mixture models with an unknown number of components: the non-identifiability induced by overfit-ting the number of components, the mixing limitations of standard Markov Chain Monte Carlo (MCMC) sampling techniques, and the related label switching problem. An overfitting approach is used to estimate the number of components in a finite mixture model via a Zmix algorithm. Zmix provides a bridge between multidimensional samplers and test based estimation methods, whereby priors are chosen to encourage extra groups to have weights approaching zero. MCMC sampling is made possible by the implementation of prior parallel tempering, an extension of parallel tempering. Zmix can accurately estimate the number of components, posterior parameter estimates and allocation probabilities given a sufficiently large sample size. The results will reflect uncertainty in the final model and will report the range of possible candidate models and their respective estimated probabilities from a single run. Label switching is resolved with a computationally lightweight method, Zswitch, developed for overfitted mixtures by exploiting the intuitiveness of allocation-based relabel-ling algorithms and the precision of label-invariant loss functions. Four simulation studies are included to illustrate Zmix and Zswitch, as well as three case studies from the literature. All methods are available as part of the R package Zmix, which can currently be applied to univariate Gaussian mixture models.