An optimization criterion for generalized discriminant analysis on undersampled problems (original) (raw)
Related papers
A discriminant analysis for undersampled data
One of the inherent problems in pattern recognition is the undersampled data problem, also known as the curse of dimensionality reduction. In this paper a new algorithm called pairwise discriminant analysis (PDA) is proposed for pattern recognition. PDA, like linear discriminant analysis (LDA), performs dimensionality reduction and clustering, without suffering from undersampled data to the same extent as LDA.
The Linear Discriminant Analysis (LDA) technique is an important and well-developed area of classification, and to date many linear (and also nonlinear) discrimination methods have been put forward. A complication in applying LDA to real data occurs when the number of features exceeds that of observations. In this case, the covariance estimates do not have full rank, and thus cannot be inverted. There are a number of ways to deal with this problem. In this paper, we propose improving LDA in this area, and we present a new approach which uses a generalization of the Moore-Penrose pseudoinverse to remove this weakness. Our new approach, in addition to managing the problem of inverting the covariance matrix, significantly improves the quality of classification, also on data sets where we can invert the covariance matrix. Experimental results on various data sets demonstrate that our improvements to LDA are efficient and our approach outperforms LDA.
Linear Discriminant Analysis for Subclustered Data
2008
Linear discriminant analysis (LDA) is a widely-used feature extraction method in classification. However, the original LDA has limitations due to the assumption of a unimodal structure for each cluster, which is not satisfied in many applications such as facial image data when variations, e.g. angle and illumination, can significantly influence the images. In this paper, we propose a novel method called hierarchical LDA (h-LDA), which takes into account hierarchical subcluster structures of the data in the LDA formulation and algorithm. We develop a theoretical basis of hierarchical LDA by identifying its relation to two-way multivariate analysis of variance (MANOVA) based on the data model and variance decomposition. Furthermore, an efficient algorithm for a regularized version of h-LDA (h-RLDA) is presented using the QR decomposition and the generalized SVD. To validate the effectiveness of the proposed method, we compare face recognition performance among h-RLDA, LDA, PCA, and TensorFaces. Our experiments show that h-RLDA produces better prediction accuracy than other methods. When only a small subset of features are used (reduced dimensionality), the superiority of h-RLDA over other methods becomes more significant. It is also shown that h-RLDA is a computationally much more efficient alternative to TensorFaces.
LINEAR DISCRIMINANT ANALYSIS WITH A GENERALIZATION OF THE MOORE–PENROSE PSEUDOINVERSE
The Linear Discriminant Analysis (LDA) technique is an important and well-developed area of classification, and to date many linear (and also nonlinear) discrimination methods have been put forward. A complication in applying LDA to real data occurs when the number of features exceeds that of observations. In this case, the covariance estimates do not have full rank, and thus cannot be inverted. There are a number of ways to deal with this problem. In this paper, we propose improving LDA in this area, and we present a new approach which uses a generalization of the Moore-Penrose pseudoinverse to remove this weakness. Our new approach, in addition to managing the problem of inverting the covariance matrix, significantly improves the quality of classification, also on data sets where we can invert the covariance matrix. Experimental results on various data sets demonstrate that our improvements to LDA are efficient and our approach outperforms LDA.
Generalized Linear Discriminant Analysis: A Unified Framework and Efficient Model Selection
IEEE Transactions on Neural Networks, 2008
High-dimensional data are common in many domains, and dimensionality reduction is the key to cope with the curse-of-dimensionality. Linear discriminant analysis (LDA) is a well-known method for supervised dimensionality reduction. When dealing with high-dimensional and low sample size data, classical LDA suffers from the singularity problem. Over the years, many algorithms have been developed to overcome this problem, and they have been applied successfully in various applications. However, there is a lack of a systematic study of the commonalities and differences of these algorithms, as well as their intrinsic relationships. In this paper, a unified framework for generalized LDA is proposed, which elucidates the properties of various algorithms and their relationships. Based on the proposed framework, we show that the matrix computations involved in LDA-based algorithms can be simplified so that the cross-validation procedure for model selection can be performed efficiently. We conduct extensive experiments using a collection of high-dimensional data sets, including text documents, face images, gene expression data, and gene expression pattern images, to evaluate the proposed theories and algorithms.
SIAM Journal on Matrix Analysis and Applications, 2011
In this paper the uncorrelated linear discriminant analysis (ULDA) for undersampled problems is studied. The main contributions of the present work include the following: (i) all solutions of the optimization problem used for establishing the ULDA are parameterized explicitly; (ii) the optimal solutions among all solutions of the corresponding optimization problem are characterized in terms of both the ratio of between-class distance to within-class distance and the maximum likelihood classification, and it is proved that these optimal solutions are exactly the solutions of the corresponding optimization problem with minimum Frobenius norm, also minimum nuclear norm; these properties provide a good mathematical justification for preferring the minimum-norm transformation over other possible solutions as the optimal transformation in ULDA; (iii) explicit necessary and sufficient conditions are provided to ensure that these minimal solutions lead to a larger ratio of between-class distance to within-class distance, thereby achieving larger discrimination in the reduced subspace than that in the original data space, and our numerical experiments show that these necessary and sufficient conditions hold true generally. Furthermore, a new and fast ULDA algorithm is developed, which is eigendecomposition-free and SVD-free, and its effectiveness is demonstrated by some real-world data sets.
Linear discriminant analysis for data with subcluster structure
2008 19th International Conference on Pattern Recognition, 2008
Linear discriminant analysis (LDA) is a widely-used feature extraction method in classification. However, the original LDA has limitations due to the assumption of a unimodal structure for each cluster, which is satisfied in many applications such as facial image data when variations such as angle and illumination can significantly influence the images of the same person. In this paper, we propose a novel method, hierarchical LDA(h-LDA), which takes into account hierarchical subcluster structures in the data sets. Our experiments show that regularized h-LDA produces better accuracy than LDA, PCA, and tensorFaces.
2006
The linear discriminant analysis based on the generalized singular value decomposition (LDA/GSVD) has been introduced to circumvent the nonsingularity restriction inherent in the classical LDA. The LDA/GSVD provides a framework in which a dimension reducing transformation can be effectively obtained for undersampled problems. In this paper, relationships between support vector machines (SVMs) and the generalized linear discriminant analysis applied to the support vectors are studied. Based on the GSVD, the weight vector of the hard-margin SVM is proved to be equivalent to the dimension reducing transformation vector generated by LDA/GSVD applied to the support vectors of the binary class. We also show that the dimension reducing transformation vector and the weight vector of soft-margin SVMs are related when a subset of support vectors are considered. These results can be generalized when kernelized SVMs and the kernelized LDA/GSVD called KDA/GSVD are considered. Through these relationships, it is shown that support vector classification is related to data reduction as well as dimension reduction by LDA/GSVD.
Subclass discriminant analysis
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006
Over the years, many Discriminant Analysis (DA) algorithms have been proposed for the study of high-dimensional data in a large variety of problems. Each of these algorithms is tuned to a specific type of data distribution (that which best models the problem at hand). Unfortunately, in most problems the form of each class pdf is a priori unknown, and the selection of the DA algorithm that best fits our data is done over trial-and-error. Ideally, one would like to have a single formulation which can be used for most distribution types. This can be achieved by approximating the underlying distribution of each class with a mixture of Gaussians. In this approach, the major problem to be addressed is that of determining the optimal number of Gaussians per class, i.e., the number of subclasses. In this paper, two criteria able to find the most convenient division of each class into a set of subclasses are derived. Extensive experimental results are shown using five databases. Comparisons are given against Linear Discriminant Analysis (LDA), Direct LDA (DLDA), Heteroscedastic LDA (HLDA), Nonparametric DA (NDA), and Kernel-Based LDA (K-LDA). We show that our method is always the best or comparable to the best.
Reduction of High Dimensional Data Using Discriminant Analysis Methods
International Research Journal of Advanced Engineering and Science, 2019
In recent years, analysis of high dimensional data for several applications such as content based retrieval, speech signals, fMRI scans, electrocardiogram signal analysis, multimedia retrieval, market based applications etc. has become a major problem. To overcome this challenge, dimensionality reduction techniques, which enable high dimensional data to be represented in a low dimensional space have been developed and deployed for varieties of application to fast track the study of the information structure. In this paper, a comparative study of LDA and a KDA among the dimensionality reduction techniques were considered using data samples collected from survey and it was implemented using object oriented programming language (C#). The results reveal that less data components were discovered by LDA across the different dataset tested in comparison with KDA.