LDA/QR: an efficient and effective dimension reduction algorithm and its theoretical foundation (original) (raw)

A New Method Combining LDA and PLS for Dimension Reduction

PLoS ONE, 2014

Linear discriminant analysis (LDA) is a classical statistical approach for dimensionality reduction and classification. In many cases, the projection direction of the classical and extended LDA methods is not considered optimal for special applications. Herein we combine the Partial Least Squares (PLS) method with LDA algorithm, and then propose two improved methods, named LDA-PLS and ex-LDA-PLS, respectively. The LDA-PLS amends the projection direction of LDA by using the information of PLS, while ex-LDA-PLS is an extension of LDA-PLS by combining the result of LDA-PLS and LDA, making the result closer to the optimal direction by an adjusting parameter. Comparative studies are provided between the proposed methods and other traditional dimension reduction methods such as Principal component analysis (PCA), LDA and PLS-LDA on two data sets. Experimental results show that the proposed method can achieve better classification performance.

Reduction of High Dimensional Data Using Discriminant Analysis Methods

International Research Journal of Advanced Engineering and Science, 2019

In recent years, analysis of high dimensional data for several applications such as content based retrieval, speech signals, fMRI scans, electrocardiogram signal analysis, multimedia retrieval, market based applications etc. has become a major problem. To overcome this challenge, dimensionality reduction techniques, which enable high dimensional data to be represented in a low dimensional space have been developed and deployed for varieties of application to fast track the study of the information structure. In this paper, a comparative study of LDA and a KDA among the dimensionality reduction techniques were considered using data samples collected from survey and it was implemented using object oriented programming language (C#). The results reveal that less data components were discovered by LDA across the different dataset tested in comparison with KDA.

IDR/QR: an incremental dimension reduction algorithm via QR decomposition

2005

Dimension reduction is critical for many database and data mining applications, such as efficient storage and retrieval of high-dimensional data. In the literature, a well-known dimension reduction scheme is Linear Discriminant Analysis (LDA). The common aspect of previously proposed LDA based algorithms is the use of Singular Value Decomposition (SVD). Due to the difficulty of designing an incremental solution for the eigenvalue problem on the product of scatter matrices in LDA, there is little work on designing incremental LDA algorithms. In this paper, we propose an LDA based incremental dimension reduction algorithm, called IDR/QR, which applies QR Decomposition rather than SVD. Unlike other LDA based algorithms, this algorithm does not require the whole data matrix in main memory. This is desirable for large data sets. More importantly, with the insertion of new data items, the IDR/QR algorithm can constrain the computational cost by applying efficient QR-updating techniques. Finally, we evaluate the effectiveness of the IDR/QR algorithm in terms of classification accuracy on the reduced dimensional space. Our experiments on several real-world data sets reveal that the accuracy achieved by the IDR/QR algorithm is very close to the best possible accuracy achieved by other LDA based algorithms. However, the IDR/QR algorithm has much less computational cost, especially when new data items are dynamically inserted.

Dimension reduction transformations in discriminant analysis

2001

Dimension reduction transformations in discriminant analysis are introduced. Their properties, as well as sufficient conditions for their characterization, are studied. Special attention is given to the continuous case, of particular importance in applications. An effective data based dimension reduction algorithm is proposed and its behavior illustrated in a classification problem where the class conditional probability distributions are multivariate normal with different covariance matrices

Linear Discriminant Dimensionality Reduction

Lecture Notes in Computer Science, 2011

Fisher criterion has achieved great success in dimensionality reduction. Two representative methods based on Fisher criterion are Fisher Score and Linear Discriminant Analysis (LDA). The former is developed for feature selection while the latter is designed for subspace learning. In the past decade, these two approaches are often studied independently. In this paper, based on the observation that Fisher score and LDA are complementary, we propose to integrate Fisher score and LDA in a unified framework, namely Linear Discriminant Dimensionality Reduction (LDDR). We aim at finding a subset of features, based on which the learnt linear transformation via LDA maximizes the Fisher criterion. LDDR inherits the advantages of Fisher score and LDA and is able to do feature selection and subspace learning simultaneously. Both Fisher score and LDA can be seen as the special cases of the proposed method. The resultant optimization problem is a mixed integer programming, which is difficult to solve. It is relaxed into a L2,1-norm constrained least square problem and solved by accelerated proximal gradient descent algorithm. Experiments on benchmark face recognition data sets illustrate that the proposed method outperforms the state of the art methods arguably.

Linear discriminant analysis: A detailed tutorial

Linear Discriminant Analysis (LDA) is a very common technique for dimensionality reduction problems as a pre-processing step for machine learning and pattern classification applications. At the same time, it is usually used as a black box, but (sometimes) not well understood. The aim of this paper is to build a solid intuition for what is LDA, and how LDA works, thus enabling readers of all levels be able to get a better understanding of the LDA and to know how to apply this technique in different applications. The paper first gave the basic definitions and steps of how LDA technique works supported with visual explanations of these steps. Moreover, the two methods of computing the LDA space, i.e. class-dependent and class-independent methods, were explained in details. Then, in a step-by-step approach, two numerical examples are demonstrated to show how the LDA space can be calculated in case of the class-dependent and class-independent methods. Furthermore, two of the most common LDA problems (i.e. Small Sample Size (SSS) and non-linearity problems) were highlighted and illustrated, and state-of-the-art solutions to these problems were investigated and explained. Finally, a number of experiments was conducted with different datasets to (1) investigate the effect of the eigenvectors that used in the LDA space on the robustness of the extracted feature for the classification accuracy, and (2) to show when the SSS problem occurs and how it can be addressed.

Multiclass classifiers based on dimension reduction with generalized LDA

Pattern Recognition, 2007

Linear discriminant analysis (LDA) has been widely used for dimension reduction of data sets with multiple classes. The LDA has been recently extended to various generalized LDA methods that are applicable regardless of the relative sizes between the data dimension and the number of data items. In this paper, we propose several multiclass classifiers based on generalized LDA algorithms, taking advantage of the dimension reducing transformation matrix without requiring additional training or any parameter optimization. A marginal linear discriminant classifier, a Bayesian linear discriminant classifier, and a one-dimensional Bayesian linear discriminant classifier are introduced for multiclass classification. Our experimental results illustrate that these classifiers produce higher tenfold cross validation accuracy than kNN and centroid based classification in the reduced dimensional space providing efficient general multiclass classifiers.

Comparative Analysis of Dimensionality Reduction Methods

Undergraduate Research Project , 2019

The face is the primary focus of attention and plays a major role in identification and establishing the uniqueness of a particular person from the rest of the human society. In most of the face recognition systems around, extraction or selection of facial features and the implementation of recognition are through efficient algorithms, these algorithms are modified for different purposes. The specific objective was to design a model that uses face recognition to compare dimensionality reduction methods. This research work provided a comparative analysis on three methods of dimensionality reduction. The methodology was based on using face recognition to analysis the accuracy and time of recognition of faces (with variations) using principal component analysis (PCA), linear discriminant analysis (LDA) and convolution neural network (CNN). The model used a dataset of 400 facial images (black and white) which combines Olivetti facial dataset and a local facial dataset (extracted from pictures of some students in RCF FUTA). Eigenfaces and fisherfaces were used to train the SVM classifier. The result from LDA and CNN showed better performance than PCA.

Fast Linear Discriminant Analysis using QR Decomposition and Regularization

2007

Linear Discriminant Analysis (LDA) is among,the most optimal dimension,reduction methods,for classification, which provides a high degree of class separability for numerous applications from science and engineering. However, problems arise with this classical method when one or both of the scatter matrices is singular. Singular scatter matrices are not unusual in many applications, especially for highdimensional data. For high-dimensional undersampled

Application of Linear and Nonlinear Dimensionality Reduction Methods

Dimensionality reduction methods have proved to be important tools in exploratory analysis as well as confirmatory analysis for data mining in various fields of science and technology. Where ever applications involve reducing to fewer dimensions, feature selection, pattern recognition, clustering, dimensionality reduction methods have been used to overcome the curse of dimensionality. In particular, Principal Component Analysis (PCA) is widely used and accepted linear dimensionality reduction method which has achieved successful ...