Supervised dictionary learning (original) (raw)

Supervised Dictionary Learning and Sparse Representation-A Review

Dictionary learning and sparse representation (DLSR) is a recent and successful mathematical model for data representation that achieves state-ofthe-art performance in various fields such as pattern recognition, machine learning, computer vision, and medical imaging. The original formulation for DLSR is based on the minimization of the reconstruction error between the original signal and its sparse representation in the space of the learned dictionary. Although this formulation is optimal for solving problems such as denoising, inpainting, and coding, it may not lead to optimal solution in classification tasks, where the ultimate goal is to make the learned dictio- * Corresponding author Email addresses: mehrdad.gangeh@utoronto.ca (Mehrdad J. Gangeh), afarahat@pami.uwaterloo.ca (Ahmed K. Farahat), aghodsib@uwaterloo.ca (Ali Ghodsi), mkamel@pami.uwaterloo.ca (Mohamed S. Kamel)

Sparse coding and dictionary learning for image understanding

Procedings of the British Machine Vision Conference 2010, 2010

Sparse coding-that is, modeling data vectors as sparse linear combinations of dictionary elements-is widely used in machine learning, neuroscience, signal processing, and statistics. This talk addresses the problem of learning the dictionary, adapting it to specific data and image understanding tasks. In particular, I will present a fast on-line approach to unsupervised dictionary learning and more generally sparse matrix factorization, and demonstrate its applications in image restoration tasks such as denoising, demosaicking, and inpainting. I will also present a general formulation of supervised dictionary learning adapted to tasks such as classification and regression. We have developed an efficient algorithm for solving the corresponding optimization problem, and I will demonstrate its application to handwritten digit classification, image deblurring and digital zooming, inverse half toning, and the detection of fake artworks.

Dictionary Learning for Sparse Representation: A Novel Approach

IEEE Signal Processing Letters, 2000

A dictionary learning problem is a matrix factorization in which the goal is to factorize a training data matrix, , as the product of a dictionary, , and a sparse coefficient matrix, , as follows, . Current dictionary learning algorithms minimize the representation error subject to a constraint on (usually having unit column-norms) and sparseness of . The resulting problem is not convex with respect to the pair . In this letter, we derive a first order series expansion formula for the factorization, . The resulting objective function is jointly convex with respect to and . We simply solve the resulting problem using alternating minimization and apply some of the previously suggested algorithms onto our new problem. Simulation results on recovery of a known dictionary and dictionary learning for natural image patches show that our new problem considerably improves performance with a little additional computational load.

A sparse reconstruction based algorithm for image and video classification

2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012

The success of sparse reconstruction based classification algorithms largely depends on the choice of overcomplete bases (dictionary). Existing methods either use the training samples as the dictionary elements or learn a dictionary by optimizing a cost function with an additional discriminating component. While the former method requires a good number of training samples per class and is not suitable for video signals, the later adds instability and more computational load. This paper presents a sparse reconstruction based classification algorithm that mitigates the above difficulties. We argue that learning class-specific dictionaries, one per class, is a natural approach to discrimination. We describe each training signal by an error vector consisting of the reconstruction errors the signal produces w.r.t each dictionary. This representation is robust to noise, occlusion and is also highly discriminative. The efficacy of the proposed method is demonstrated in terms of high accuracy for image-based Species and Face recognition and video-based Action recognition.

Dictionary Learning Algorithms for Sparse Representation

Neural Computation, 2003

Algorithms for data-driven learning of domain-specific overcomplete dictionaries are developed to obtain maximum likelihood and maximum a posteriori dictionary estimates based on the use of Bayesian models with concave/Schur-concave (CSC) negative log priors. Such priors are appropriate for obtaining sparse representations of environmental signals within an appropriately chosen (environmentally matched) dictionary. The elements of the dictionary can be interpreted as concepts, features, or words capable of succinct expression of events encountered in the environment (the source of the measured signals). This is a generalization of vector quantization in that one is interested in a description involving a few dictionary entries (the proverbial "25 words or less"), but not necessarily as succinct as one entry. To learn an environmentally adapted dictionary capable of concise expression of signals generated by the environment, we develop algorithms that iterate between a representative set of sparse representations found by variants of FOCUSS and an update of the dictionary using these sparse representations.

Learned dictionaries for sparse image representation: properties and results

Wavelets and Sparsity XIV, 2011

Sparse representation of images using learned dictionaries have been shown to work well for applications like image denoising, impainting, image compression, etc. In this paper dictionary properties are reviewed from a theoretical approach, and experimental results for learned dictionaries are presented. The main dictionary properties are the upper and lower frame (dictionary) bounds, and (mutual) coherence properties based on the angle between dictionary atoms. Both l 0 sparsity and l 1 sparsity are considered by using a matching pursuit method, order recursive matching Pursuit (ORMP), and a basis pursuit method, i.e. LARS or Lasso. For dictionary learning the following methods are considered: Iterative least squares (ILS-DLA or MOD), recursive least squares (RLS-DLA), K-SVD and online dictionary learning (ODL). Finally, it is shown how these properties relate to an image compression example.

An Efficient Dictionary Learning Algorithm for Sparse Representation

2010 Chinese Conference on Pattern Recognition (CCPR), 2010

Sparse and redundant representation of data assumes an ability to describe signals as linear combinations of a few atoms from a dictionary. If the model of the signal is unknown, the dictionary can be learned from a set of training signals. Like the K-SVD, many of the practical dictionary learning algorithms are composed of two main parts: sparse-coding and dictionary-update. This paper first proposes a Stagewise least angle regression (St-LARS) method for performing the sparse-coding operation. The St-LARS applies a hard-thresholding strategy into the original least angle regression (LARS) algorithm, which enables it to select many atoms at each iteration and thus results in fast solutions while still provides good results. Then, a dictionary update method named approximated singular value decomposition (ASVD) is used on the dictionary update stage. It is a quick approximation of the exact SVD computation and can reduce the complexity of it. Experiments on both synthetic data and 3-D image denoising demonstrate the advantages of the proposed algorithm over other dictionary learning methods not only in terms of better trained dictionary but also in terms of computation time.

Sparse representations for pattern classification using learned dictionaries

2009

Sparse representations have been often used for inverse problems in signal and image processing. Furthermore, frameworks for signal classification using sparse and overcomplete representations have been developed. Data-dependent representations using learned dictionaries have been significant in applications such as feature extraction and denoising.

Sparse Representation for Signal Classification

In this paper, application of sparse representation (factorization) of signals over an overcomplete basis (dictionary) for signal classification is discussed. Searching for the sparse representation of a signal over an overcomplete dictionary is achieved by optimizing an objective function that includes two terms: one that measures the signal reconstruction error and another that measures the sparsity. This objective function works well in applications where signals need to be reconstructed, like coding and denoising. On the other hand, discriminative methods, such as linear discriminative analysis (LDA), are better suited for classification tasks. However, discriminative methods are usually sensitive to corruption in signals due to lacking crucial properties for signal reconstruction. In this paper, we present a theoretical framework for signal classification with sparse representation. The approach combines the discrimination power of the discriminative methods with the reconstruction property and the sparsity of the sparse representation that enables one to deal with signal corruptions: noise, missing data and outliers. The proposed approach is therefore capable of robust classification with a sparse representation of signals. The theoretical results are demonstrated with signal classification tasks, showing that the proposed approach outperforms the standard discriminative methods and the standard sparse representation in the case of corrupted signals.

An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation

In recent years there has been a growing interest in the study of sparse representation of signals. Using an overcomplete dictionary that contains prototype signal-atoms, signals are described by sparse linear combinations of these atoms. Applications that use sparse representation are many and include compression, regularization in inverse problems, feature extraction, and more. Recent activity in this field has concentrated mainly on the study of pursuit algorithms that decompose signals with respect to a given dictionary. Designing dictionaries to better fit the above model can be done by either selecting one from a prespecified set of linear transforms or adapting the dictionary to a set of training signals. Both of these techniques have been considered, but this topic is largely still open. In this paper we propose a novel algorithm for adapting dictionaries in order to achieve sparse signal representations. Given a set of training signals, we seek the dictionary that leads to the best representation for each member in this set, under strict sparsity constraints. We present a new method-the K-SVD algorithm-generalizing the K-means clustering process. K-SVD is an iterative method that alternates between sparse coding of the examples based on the current dictionary and a process of updating the dictionary atoms to better fit the data. The update of the dictionary columns is combined with an update of the sparse representations, thereby accelerating convergence. The K-SVD algorithm is flexible and can work with any pursuit method (e.g., basis pursuit, FOCUSS, or matching pursuit). We analyze this algorithm and demonstrate its results both on synthetic tests and in applications on real image data.