On the maximum-entropy approach to undersized samples (original) (raw)
Related papers
2019
The application of standard sufficient dimension reduction methods for reducing the dimension space of predictors without losing regression information requires inverting the covariance matrix of the predictors. This has posed a number of challenges especially when analyzing high-dimensional data sets in which the number of predictors p is much larger than number of samples n, (n≪ p). A new covariance estimator, called the Maximum Entropy Covariance (MEC) that addresses loss of covariance information when similar covariance matrices are linearly combined using Maximum Entropy (ME) principle is proposed in this work. By benefitting naturally from slicing or discretizing range of the response variable, y into H non-overlapping categories, h_1,... ,h_H, MEC first combines covariance matrices arising from samples in each y slice h∈ H and then select the one that maximizes entropy under the principle of maximum uncertainty. The MEC estimator is then formed from convex mixture of such ent...
Maximum Entropy Multivariate Density Estimation: An exact goodness-of-fit approach
Computing Research Repository, 2004
We consider the problem of estimating the population probability distribution given a finite set of multivariate samples, using the maximum entropy approach. In strict keeping with Jaynes' original definition, our precise formulation of the problem considers contributions only from the smoothness of the estimated distribution (as measured by its entropy) and the loss functional associated with its goodness-of-fit to the sample data, and in particular does not make use of any additional constraints that cannot be justified from the sample data alone. By mapping the general multivariate problem to a tractable univariate one, we are able to write down exact expressions for the goodness-of-fit of an arbitrary multivariate distribution to any given set of samples using both the traditional likelihood-based approach and a rigorous information-theoretic approach, thus solving a long-standing problem. As a corollary we also give an exact solution to the 'forward problem' of determining the expected distributions of samples taken from a population with known probability distribution. *
Estimating the covariance matrix: a new approach
Journal of Multivariate Analysis, 2003
In this paper, we consider the problem of estimating the covariance matrix and the generalized variance when the observations follow a nonsingular multivariate normal distribution with unknown mean. A new method is presented to obtain a truncated estimator that utilizes the information available in the sample mean matrix and dominates the James-Stein minimax estimator. Several scale equivariant minimax estimators are also given. This method is then applied to obtain new truncated and improved estimators of the generalized variance; it also provides a new proof to the results of Shorrock and Zidek (1976) and Sinha (1976).
Dimension reduction in multivariate analysis using maximum entropy criterion
Journal of Statistics and Management Systems, 2006
In the present communication dimension reduction criteria in Multivariate data with no external variables are studied by using Entropy Optimization Principles. Maximum entropy criterion is provided and its relation with other criteria for selection of principal variables in multivariate analysis is established. A comparative study of performance of principal variables with the corresponding number of principal components is made by considering empirical data set.
Comparative statics of the generalized maximum entropy estimator of the general linear model
European Journal of Operational Research, 2008
The generalized maximum entropy method of information recovery requires that an analyst provides prior information in the form of finite bounds on the permissible values of the regression coefficients and error values for its implementation. Using a new development in the method of comparative statics, the sensitivity of the resulting coefficient and error estimates to the prior information is investigated. A negative semidefinite matrix reminiscent of the Slutsky-matrix of neoclassical microeconomic theory is shown to characterize the said sensitivity, and an upper bound for the rank of the matrix is derived.
The main theme of this paper is a modification of the likelihood ratio test (LRT) for testing high dimensional covariance matrix. Recently, the correct asymptotic distribution of the LRT for a large-dimensional case (the case p/n approaches to a constant γ ∈ (0, 1]) is specified by researchers. The correct procedure is named as corrected LRT. Despite of its correction, the corrected LRT is a function of sample eigenvalues that are suffered from redundant variability from high dimensionality and, subsequently, still does not have full power in differentiating hypotheses on the covariance matrix. In this paper, motivated by the successes of a linearly shrunken covariance matrix estimator (simply shrinkage estimator) in various applications, we propose a regularized LRT that uses, in defining the LRT, the shrinkage estimator instead of the sample covariance matrix. We compute the asymptotic distribution of the regularized LRT, when the true covariance matrix is the identity matrix and a spiked covariance matrix. The obtained asymptotic results have applications in testing various hypotheses on the covariance matrix. Here, we apply them to testing the identity of the true covariance matrix, which is a long standing problem in the literature, and show that the regularized LRT outperforms the corrected LRT, which is its non-regularized counterpart. In addition, we compare the power of the regularized LRT to those of recent non-likelihood based procedures.
Estimation of the entropy of a multivariate normal distribution
Journal of Multivariate Analysis, 2005
Motivated by problems in molecular biosciences wherein the evaluation of entropy of a molecular system is important for understanding its thermodynamic properties, we consider the efficient estimation of entropy of a multivariate normal distribution having unknown mean vector and covariance matrix. Based on a random sample, we discuss the problem of estimating the entropy under the quadratic loss function. The best affine equivariant estimator is obtained and, interestingly, it also turns out to be an unbiased estimator and a generalized Bayes estimator. It is established that the best affine equivariant estimator is admissible in the class of estimators that depend on the determinant of the sample covariance matrix alone. The risk improvements of the best affine equivariant estimator over the maximum likelihood estimator (an estimator commonly used in molecular sciences) are obtained numerically and are found to be substantial in higher dimensions, which is commonly the case for atomic coordinates in macromolecules such as proteins. We further establish that even the best affine equivariant estimator is inadmissible and obtain Stein-type and Brewster-Zidek-type estimators dominating it. The Brewster-Zidek-type estimator is shown to be generalized Bayes.
On the Expected Likelihood Approach for Assessment of Regularization Covariance Matrix
IEEE Signal Processing Letters, 2015
Regularization, which consists in shrinkage of the sample covariance matrix to a target matrix, is a commonly used and effective technique in low sample support covariance matrix estimation. Usually, a target matrix is chosen and optimization of the shrinkage factor is carried out, based on some relevant metric. In this letter, we rather address the choice of the target matrix. More precisely, we aim at evaluating, from observation of the data matrix, whether a given target matrix is a good regularizer. Towards this end, the expected likelihood (EL) approach is investigated. At a Þrst step, we re-interpret the regularized covariance matrix estimate as the minimum mean-square error estimate in a Bayesian model where the target matrix serves as a prior. The likelihood function of the data is then derived, and the EL principle is subsequently applied. Over-sampled and under-sampled scenarios are considered.