Relabelling in Bayesian mixture models by pivotal units (original) (raw)
Related papers
Probabilistic relabelling strategies for the label switching problem in Bayesian mixture models
Statistics and Computing, 2010
The label switching problem is caused by the likelihood of a Bayesian mixture model being invariant to permutations of the labels. The permutation can change multiple times between Markov Chain Monte Carlo (MCMC) iterations making it difficult to infer component-specific parameters of the model. Various so-called 'relabelling' strategies exist with the goal to 'undo' the label switches that have occurred to enable estimation of functions that depend on component-specific parameters. Most existing approaches rely upon specifying a loss function, and relabelling by minimising its posterior expected loss. In this paper we develop probabilistic approaches to relabelling that allow estimation and incorporation of the uncertainty in the relabelling process. Variants of the probabilistic relabelling algorithm are introduced and compared to existing loss function based methods. We demonstrate that the idea of probabilistic relabelling can be expressed in a rigorous framework based on the EM algorithm.
PloS one, 2015
The label switching problem occurs as a result of the nonidentifiability of posterior distribution over various permutations of component labels when using Bayesian approach to estimate parameters in mixture models. In the cases where the number of components is fixed and known, we propose a relabelling algorithm, an allocation variable-based (denoted by AVP) probabilistic relabelling approach, to deal with label switching problem. We establish a model for the posterior distribution of allocation variables with label switching phenomenon. The AVP algorithm stochastically relabel the posterior samples according to the posterior probabilities of the established model. Some existing deterministic and other probabilistic algorithms are compared with AVP algorithm in simulation studies, and the success of the proposed approach is demonstrated in simulation studies and a real dataset.
Mixture models with an unknown number of components via a new posterior split–merge MCMC algorithm
Applied Mathematics and Computation, 2014
In this paper we introduce a Bayesian analysis for mixture models with an unknown number of components via a new posterior split-merge MCMC algorithm. Our strategy for splitting is based on data in which allocation probabilities are calculated based on posterior distribution from the previously allocated observations. This procedure is easy to be implemented and determines a quick split proposal. The acceptance probability for split-merge movements are calculated according to metropolised Carlin and Chib's procedure. The performance of the proposed algorithm is verified using artificial datasets as well as two real datasets. The first real data set is the benchmark galaxy data, while the second is the publicly available data set on Escherichia coli bacterium.
Bayesian analysis of finite mixture models of distributions from exponential families
Computational Statistics, 2006
This paper deals with the Bayesian analysis of finite mixture models with a fixed number of component distributions from natural exponential families with quadratic variance function (NEF-QVF). A unified Bayesian framework addressing the two main difficulties in this context is presented, i.e., the prior distribution choice and the parameter unidentifiability problem. In order to deal with the first issue, conjugate prior distributions are used. An algorithm to calculate the parameters in the prior distribution to obtain the least informative one into the class of conjugate distributions is developed. Regarding the second issue, a general algorithm to solve the label-switching problem is presented. These techniques are easily applied in practice as it is shown with an illustrative example.
Approximate Bayesian computation for finite mixture models
Journal of Statistical Computation and Simulation, 2020
Finite mixture models are used in statistics and other disciplines, but inference for mixture models is challenging. The multimodality of the likelihood function and the so called label switching problem contribute to the challenge. We propose extensions of the Approximate Bayesian Computation Population Monte-Carlo (ABC-PMC) algorithm as an alternative framework for inference on finite mixture models. There are several decisions to make when implementing an ABC-PMC algorithm for finite mixture models, including the selection of the kernel used for moving the particles through the iterations, how to address the label switching problem and the choice of informative summary statistics. Examples are presented to demonstrate the performance of the proposed ABC-PMC algorithm for mixture modeling.
A practical sampling approach for a Bayesian mixture model with unknown number of components
Statistical Papers, 2007
Recently, mixture distribution becomes more and more popular in many scientific fields. Statistical computation and analysis of mixture models, however, are extremely complex due to the large number of parameters involved. Both EM algorithms for likelihood inference and MCMC procedures for Bayesian analysis have various difficulties in dealing with mixtures with unknown number of components. In this paper, we propose a direct sampling approach to the computation of Bayesian finite mixture models with varying number of components. This approach requires only the knowledge of the density function up to a multiplicative constant. It is easy to implement, numerically efficient and very practical in real applications. A simulation study shows that it performs quite satisfactorily on relatively high dimensional distributions. A well-known genetic data set is used to demonstrate the simplicity of this method and its power for the computation of high dimensional Bayesian mixture models.
Bayesian analysis of finite mixtures of multinomial and negative-multinomial distributions
Computational Statistics & Data Analysis, 2007
The Bayesian implementation of finite mixtures of distributions has been an area of considerable interest within the literature. Computational advances on approximation techniques such as Markov chain Monte Carlo (MCMC) methods have been a keystone to Bayesian analysis of mixture models. This paper deals with the Bayesian analysis of finite mixtures of two particular types of multidimensional distributions: the multinomial and the negative-multinomial ones. A unified framework addressing the main topics in a Bayesian analysis is developed for the case with a known number of component distributions. In particular, theoretical results and algorithms to solve the label-switching problem are provided. An illustrative example is presented to show that the proposed techniques are easily applied in practice.
Pattern Analysis & Applications, 2009
In this paper, we present a fully Bayesian approach for generalized Dirichlet mixtures estimation and selection. The estimation of the parameters is based on the Monte Carlo simulation technique of Gibbs sampling mixed with a Metropolis-Hastings step. Also, we obtain a posterior distribution which is conjugate to a generalized Dirichlet likelihood. For the selection of the number of clusters, we used the integrated likelihood. The performance of our Bayesian algorithm is tested and compared with the maximum likelihood approach by the classification of several synthetic and real data sets. The generalized Dirichlet mixture is also applied to the problems of IR eye modeling and introduced as a probabilistic kernel for Support Vector Machines.
Gibbs Sampling Based Bayesian Analysis of Mixtures with Unknown Number of Components
For mixture models with unknown number of components, Bayesian approaches, as considered by and , are reconciled here through a simple Gibbs sampling approach. Specifically, we consider exactly the same direct set up as used by , but put Dirichlet process prior on the mixture components; the latter has also been used by albeit in a different set up. The reconciliation we propose here yields a simple Gibbs sampling scheme for learning about all the unknowns, including the unknown number of components. Thus, we completely avoid complicated reversible jump Markov chain Monte Carlo (RJMCMC) methods, yet tackle variable dimensionality simply and efficiently. Moreover, we demonstrate, using both simulated and real data sets, and pseudo-Bayes factors, that our proposed model outperforms that of , while enjoying, at the same time, computational superiority over the methods proposed by and . We also discuss issues related to clustering and argue that in principle, our approach is capable of learning about the number of clusters in the sample as well as in the population, while the approach of is suitable for learning about the number of clusters in the sample only.
Adaptive MCMC with online relabeling
Bernoulli, 2015
When targeting a distribution that is artificially invariant under some permutations, Markov chain Monte Carlo (MCMC) algorithms face the label-switching problem, rendering marginal inference particularly cumbersome. Such a situation arises, for example, in the Bayesian analysis of finite mixture models. Adaptive MCMC algorithms such as adaptive Metropolis (AM), which self-calibrates its proposal distribution using an online estimate of the covariance matrix of the target, are no exception. To address the label-switching issue, relabeling algorithms associate a permutation to each MCMC sample, trying to obtain reasonable marginals. In the case of adaptive Metropolis (Bernoulli 7 (2001) 223-242), an online relabeling strategy is required. This paper is devoted to the AMOR algorithm, a provably consistent variant of AM that can cope with the label-switching problem. The idea is to nest relabeling steps within the MCMC algorithm based on the estimation of a single covariance matrix that is used both for adapting the covariance of the proposal distribution in the Metropolis algorithm step and for online relabeling. We compare the behavior of AMOR to similar relabeling methods. In the case of compactly supported target distributions, we prove a strong law of large numbers for AMOR and its ergodicity. These are the first results on the consistency of an online relabeling algorithm to our knowledge. The proof underlines latent relations between relabeling and vector quantization.