Adaptive Mixtures of Factor Analyzers (original) (raw)
Related papers
Modelling high-dimensional data by mixtures of factor analyzers
Computational Statistics & Data Analysis, 2003
We focus on mixtures of factor analyzers from the perspective of a method for model-based density estimation from high-dimensional data, and hence for the clustering of such data. This approach enables a normal mixture model to be ÿtted to a sample of n data points of dimension p, where p is large relative to n. The number of free parameters is controlled through the dimension of the latent factor space. By working in this reduced space, it allows a model for each component-covariance matrix with complexity lying between that of the isotropic and full covariance structure models. We shall illustrate the use of mixtures of factor analyzers in a practical example that considers the clustering of cell lines on the basis of gene expressions from microarray experiments.
Incremental Mixtures of Factor Analysers
Proceedings of the Pattern Recognition, 17
A mixture of factor analyzers is a semiparametric density estimator that performs clustering and dimensionality reduction in each cluster (component) simultaneously. It performs nonlinear dimensionality reduction by modeling the density as a mixture of local linear models. The approach can be used for classification by modeling each class-conditional density using a mixture model and the complete data is then a mixture of mixtures. We propose an incremental mixture of factor analysis algorithm where the number of components (local models) in the mixture and the number of factors in each component (local dimensionality) are determined adaptively. Our results on different pattern classification tasks prove the utility of our approach and indicate that our algorithms find a good trade-off between model complexity and accuracy.
Constrained EM Algorithms for Gaussian Mixtures of Factor Analyzers
2012
Mixtures of factor analyzers are becoming more and more popular in the area of model based clustering of high-dimensional data. In data modeling, according to the likelihood approach, it is well known that the loglikelihood function may present spurious maxima and singularities and this is due to specific patterns of the estimated covariance structure. To reduce such drawbacks, in this paper we introduce and implement a procedure for the parameter estimation of mixtures of factor analyzers, which maximizes the likelihood function in a constrained parameter space, having no singularities and a reduced number of spurious local maxima. We then analyze and measure its performance, compared to the usual non-constrained approach, via some simulations and applications to real data sets.
Data-driven EM constraints for Gaussian mixtures of factor analyzers
Mixtures of factor analyzers are becoming more and more popular in the area of model based clustering of high-dimensional data. In this paper we implement a data-driven methodology to maximize the likelihood function in a constrained parameter space, to overcome the well known issue of singularities and to reduce spurious maxima in the EM algorithm. Simulation results and applications to real data show that the problematic convergence of the EM, even more critical when dealing with factor analyzers, can be greatly improved.
Maximum likelihood estimation in constrained parameter spaces for mixtures of factor analyzers
Mixtures of factor analyzers are becoming more and more popular in the area of model based clustering of multivariate data. According to the likelihood approach in data modeling, it is well known that the unconstrained likelihood function may present spurious maxima and singularities. To reduce such drawbacks, in this paper we introduce a procedure for parameter estimation of mixtures of factor analyzers, which maximizes the likelihood function under the mild requirement that the eigenvalues of the covariance matrices lie into some interval [a, b].Moreover, we give a recipe on how to select appropriate bounds for the constrained EM algorithm, directly from the handled data. We then analyze and measure its performance, compared with the usual non-constrained approach, and also with other constrained models in the literature. Results show that the data-driven constraints improve the estimation and the subsequent classification, at the same time.
Robust estimation for mixtures of Gaussian factor analyzers, based on trimming and constraints
Mixtures of Gaussian factors are powerful tools for modeling an unobserved heterogeneous population, offering - at the same time - dimension reduction and model-based clustering. Unfortunately, the high prevalence of spurious solutions and the disturbing effects of outlying observations, along maximum likelihood estimation, open serious issues. In this paper we consider restrictions for the component covariances, to avoid spurious solutions, and trimming, to provide robustness against violations of normality assumptions of the underlying latent factors. A detailed AECM algorithm for this new approach is presented. Simulation results and an application to the AIS dataset show the aim and effectiveness of the proposed methodology.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
Mixtures of factor analyzers enable model-based density estimation to be undertaken for high-dimensional data, where the number of observations is not very large relative to their dimension ¡ . In practice, there is often the need to reduce further the number of parameters in the specification of the component-covariance matrices. To this end, we propose the use of common component-factor loadings, which considerably reduces further the number of parameters. Moreover, it allows the data to be displayed in low-dimensional plots.
Clustering and classification via cluster-weighted factor analyzers
Advances in Data Analysis and Classification, 2013
In model-based clustering and classification, the cluster-weighted model is a convenient approach when the random vector of interest is constituted by a response variable Y and by a vector X of p covariates. However, its applicability may be limited when p is high. To overcome this problem, this paper assumes a latent factor structure for X in each mixture component, under Gaussian assumptions. This leads to the cluster-weighted factor analyzers (CWFA) model. By imposing constraints on the variance of Y and the covariance matrix of X, a novel family of sixteen CWFA models is introduced for model-based clustering and classification. The alternating expectation-conditional maximization algorithm, for maximum likelihood estimation of the parameters of all models in the family, is described; to initialize the algorithm, a 5-step hierarchical procedure is proposed, which uses the nested structures of the models within the family and thus guarantees the natural ranking among the sixteen likelihoods. Artificial and real data show that these models have very good clustering and classification performance and that the algorithm is able to recover the parameters very well.
Mixtures of factor analysers. Bayesian estimation and inference by stochastic simulation
Machine Learning, 2003
Factor Analysis (FA) is a well established probabilistic approach to unsupervised learning for complex systems involving correlated variables in high-dimensional spaces. FA aims principally to reduce the dimensionality of the data by projecting high-dimensional vectors on to lower-dimensional spaces. However, because of its inherent linearity, the generic FA model is essentially unable to capture data complexity when the input space is nonhomogeneous. A finite Mixture of Factor Analysers (MFA) is a globally nonlinear and therefore more flexible extension of the basic FA model that overcomes the above limitation by combining the local factor analysers of each cluster of the heterogeneous input space. The structure of the MFA model offers the potential to model the density of high-dimensional observations adequately while also allowing both clustering and local dimensionality reduction. Many aspects of the MFA model have recently come under close scrutiny, from both the likelihood-based and the Bayesian perspectives. In this paper, we adopt a Bayesian approach, and more specifically a treatment that bases estimation and inference on the stochastic simulation of the posterior distributions of interest. We first treat the case where the number of mixture components and the number of common factors are known and fixed, and we derive an efficient Markov Chain Monte Carlo (MCMC) algorithm based on Data Augmentation to perform inference and estimation. We also consider the more general setting where there is uncertainty about the dimensionalities of the latent spaces (number of mixture components and number of common factors unknown), and we estimate the complexity of the model by using the sample paths of an ergodic Markov chain obtained through the simulation of a continuous-time stochastic birth-and-death point process. The main strengths of our algorithms are that they are both efficient (our algorithms are all based on familiar and standard distributions that are easy to sample from, and many characteristics of interest are by-products of the same process) and easy to interpret. Moreover, they are straightforward to implement and offer the possibility of assessing the goodness of the results obtained. Experimental results on both artificial and real data reveal that our approach performs well, and can therefore be envisaged as an alternative to the other approaches used for this model.