Non-parallel support vector classifiers with different loss functions (original) (raw)

Multiclass Proximal Support Vector Machines

Journal of Computational and Graphical Statistics, 2006

We propose an extension of proximal support vector machines (PSVM) to the multiclass case. Unlike the one-versus-rest approach that constructs the decision rule based on multiple binary classification tasks, the multiclass PSVM (MPSVM) considers all classes simultaneously and provides a unifying framework when there are either equal or unequal misclassification costs. The MPSVM is built in a regularization framework of reproducing kernel Hilbert space (RKHS) and implements the Bayes rule asymptotically. With regard to computation, the MPSVM simply solves a system of linear equations and demands much less computational effort than the SVM, which can be slow due to optimizing a large-scaled quadratic programming under linear constraints. Some efficient algorithm is suggested and one stable computation strategy is also provided for ill-posed cases. The effectiveness of the MPSVM was demonstrated by both simulation studies and applications to cancer classification using microarray data.

A support vector machine formulation to pca analysis and its kernel version

IEEE Transactions on Neural Networks, 2003

In this letter, we present a simple and straightforward primal-dual support vector machine formulation to the problem of principal component analysis (PCA) in dual variables. By considering a mapping to a high-dimensional feature space and application of the kernel trick (Mercer theorem) kernel PCA is obtained as introduced by Schölkopf et al. While least squares support vector machine classifiers have a natural link with kernel Fisher discriminant analysis (minimizing the within class scatter around targets +1 and 1), for PCA analysis one can take the interpretation of a one-class modeling problem with zero target value around which one maximizes the variance. The score variables are interpreted as error variables within the problem formulation. In this way primal-dual constrained optimization problem interpretations to linear and kernel PCA analysis are obtained in a similar style as for least square-support vector machine (LS-SVM) classifiers. Index Terms-Kernel methods, kernel principal component analysis (PCA), least squares-support vector machine (LS-SVM), PCA analysis, SVMs.

Study on proximal support vector machine as a classifier

2012

Proximal Support Vector machine based on Least Mean Square Algorithm classi-fiers (LMS-SVM) are tools for classification of binary data. Proximal Support Vector based on Least Mean Square Algorithm classifiers is completely based on the theory of Proximal Support Vector Machine classifiers (PSVM). PSVM classifies binary pat- terns by assigning them to the closest of two parallel planes that are pushed apart as far as possible. The training time for the classifier is found to be faster compared to their previous versions of Support Vector Machines. But due to the presence of slack variable or error vector the classification accuracy of the Proximal Support Vector Machine is less. So we have come with an idea to update the adjustable weight vectors at the training phase such that all the data points fall out-side the region of separation and falls on the correct side of the hyperplane and to enlarge the width of the separable region.To implement this idea, Least Mean Square (LMS) algo...

Generalization of linear and non-linear support vector machine in multiple fields: a review

Computer Science and Information Technologies, 2023

Support vector machines (SVMs) are a set of related supervised learning methods used for classification and regression. They belong to a family of generalized linear classifiers. In other terms, SVM is a classification and regression prediction tool that uses machine learning theory to maximize predictive accuracy. In this article, the discussion about linear and non-linear SVM classifiers with their functions and parameters is investigated. Due to the equality type of constraints in the formulation, the solution follows from solving a set of linear equations. Besides this, if the under-consideration problem is in the form of a non-linear case, then the problem must convert into linear separable form with the help of kernel trick and solve it according to the methods. Some important algorithms related to sentimental work are also presented in this paper. Generalization of the formulation of linear and non-linear SVMs is also open in this article. In the final section of this paper, the different modified sections of SVM are discussed which are modified by different research for different purposes.

A Note on Support Vector Machine Degeneracy

Lecture Notes in Computer Science, 1999

When training Support Vector Machines (SVMs) over non-separable data sets, one sets the threshold b using any dual cost coefficient that is strictly between the bounds of 0 and C. We show that there exist SVM training problems with dual optimal solutions with all coefficients at bounds, but that all such problems are degenerate in the sense that the "optimal separating hyperplane" is given by w = 0, and the resulting (degenerate) SVM will classify all future points identically (to the class that supplies more training data). We also derive necessary and sufficient conditions on the input data for this to occur. Finally, we show that an SVM training problem can always be made degenerate by the addition of a single data point belonging to a certain unbounded polyhedron, which we characterize in terms of its extreme points and rays.

Formulations of support vector machines: a note from an optimization point of view

2001

In this article, we discuss issues about formulations of support vector machines (SVM) from an optimization point of view. First, SVMs map training data into a higher-(maybe infinite-) dimensional space. Currently primal and dual formulations of SVM are derived in the finite dimensional space and readily extend to the infinite-dimensional space. We rigorously discuss the primal-dual relation in the infinite-dimensional spaces.

Non-parallel semi-supervised classification based on kernel spectral clustering

In this paper, a non-parallel semi-supervised algorithm based on kernel spectral clustering is formulated. The prior knowledge about the labels is incorporated into the kernel spectral clustering formulation via adding regularization terms. In contrast with the existing multi-plane classifiers such as Multisurface Proximal Support Vector Machine (GEPSVM) and Twin Support Vector Machines (TWSVM) and its least squares version (LSTSVM) we will not use a kernel-generated surface. Instead we apply the kernel trick in the dual. Therefore as opposed to conventional non-parallel classifiers one does not need to formulate two different primal problems for the linear and nonlinear case separately. The proposed method will generate two non-parallel hyperplanes which then are used for out-of-sample extension. Experimental results demonstrate the efficiency of the proposed method over existing methods.

Comprehensive review on twin support vector machines

Annals of Operations Research, 2022

Twin support vector machine (TWSVM) and twin support vector regression (TSVR) are newly emerging efficient machine learning techniques which offer promising solutions for classification and regression challenges respectively. TWSVM is based upon the idea to identify two nonparallel hyperplanes which classify the data points to their respective classes. It requires to solve two small sized quadratic programming problems (QPPs) in lieu of solving single large size QPP in support vector machine (SVM) while TSVR is formulated on the lines of TWSVM and requires to solve two SVM kind problems. Although there has been good research progress on these techniques; there is limited literature on the comparison of different variants of TSVR. Thus, this review presents a rigorous analysis of recent research in TWSVM and TSVR simultaneously mentioning their limitations and advantages. To begin with, we first introduce the basic theory of support vector machine, TWSVM and then focus on the various improvements and applications of TWSVM, and then we introduce TSVR and its various enhancements. Finally, we suggest future research and development prospects.

Special issue on support vector machines

Neurocomputing, 2003

Support vector machines (SVMs) are currently a very active research area within machine learning. Motivated by statistical learning theory, SVMs have been successfully applied to numerous tasks, among others in data mining, computer vision, and bioinformatics. SVMs are examples of a broader category of learning approaches which utilize the concept of kernel substitution, which makes the task of learning more tractable by exploiting an implicit mapping into a high-dimensional space. SVMs have many appealing properties for machine learning. For example, the classic SVM learning task involves convex quadratic programming, a problem that does not su er from the 'local minima' problem and whose solution may easily be found by using one of the many specially e cient algorithms developed for it in the optimization theory. Furthermore, recently developed model selection strategies can be applied, so that few, if any, learning parameters need to be set by the operator. Above all, they have been found to work very well in practice.