A geometric approach to support vector regression (original) (raw)
Related papers
Duality, geometry, and support vector regression
2002
We develop an intuitive geometric framework for support vector regression (SVR). By examining when-tubes exist, we show that SVR can be regarded as a classification problem in the dual space. Hard and soft-tubes are constructed by separating the convex or reduced convex hulls respectively of the training data with the response variable shifted up and down by. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted datasets. Maximizing the margin corresponds to shrinking the effective-tube. In the proposed approach the effects of the choices of all parameters become clear geometrically.
A Geometric Approach to Support Vector Machine (SVM) Classification
IEEE Transactions on Neural Networks, 2006
The geometric framework for the support vector machine (SVM) classification problem provides an intuitive ground for the understanding and the application of geometric optimization algorithms, leading to practical solutions of real world classification problems. In this work, the notion of "reduced convex hull" is employed and supported by a set of new theoretical results. These results allow existing geometric algorithms to be directly and practically applied to solve not only separable, but also nonseparable classification problems both accurately and efficiently. As a practical application of the new theoretical results, a known geometric algorithm has been employed and transformed accordingly to solve nonseparable problems successfully.
Duality and Geometry in SVM Classifiers
Proceedings of the Seventeenth International Conference on Machine Learning, 2000
We develop an intuitive geometric interpretation of the standard support vector machine (SVM) for classification of both linearly separable and inseparable data and provide a rigorous derivation of the concepts behind the geometry. For the separable case finding the maximum margin between the two sets is equivalent to finding the closest points in the smallest convex sets that contain each class (the convex hulls). We now extend this argument to the inseparable case by using a reduced convex hull reduced away from outliers. We prove that solving the reduced convex hull formulation is exactly equivalent to solving the standard inseparable SVM for appropriate choices of parameters. Some additional advantages of the new formulation are that the effect of the choice of parameters becomes geometrically clear and that the formulation may be solved by fast nearest point algorithms. By changing norms these arguments hold for both the standard 2-norm and 1-norm SVM.
On margin and support vector separability in Support Vector Machines for Regression
1999
In this report we show some simple properties of SVM for regression. In particular we show that for close to zero, minimizing the norm of w is equivalent to maximizing the distance between the optimal approximating hyperplane solution of SVMR and the closest points in the data set. So, in this case, there exists a complete analogy between SVM for regression and classi cation, and the -tube plays the same role as the margin between classes. Moreover we show that for every the set of support vectors found by SVMR is linearly separable in the feature space and the optimal approximating hyperplane is a separator for this set. As a consequence, we show that for every regression problem there exists a classi cation problem which is linearly separable in the feature space. This is due to the fact that the solution of SVMR separates the set of support vectors in two classes: the support vectors living above and the one living below the optimal approximating hyperplane solution of SVMR. The position of the support vectors with respect to the hyperplane is given by the sign of ( i ? i ). Finally, we present a simple algorithm for obtaining a sparser representation of the optimal approximating hyperplane by using SVM for classi cation.
Properties of support vector machines for regression
1999
In this report we show that the -tube size in Support Vector Machine (SVM) for regression is 2 = p 1 + jjwjj 2 . By using this result we show that, in the case all the data points are inside the -tube, minimizing jjwjj 2 in SVM for regression is equivalent to maximizing the distance between the approximating hyperplane and the farest points in the training set. Moreover, in the most general setting in which the data points live also outside the -tube, we show that, for a xed value of , minimizing jjwjj 2 is equivalent to maximizing the sparsity of the representation of the optimal approximating hyperplane, that is equivalent to minimizing the number of coe cients di erent from zero in the expression of the optimal w. Then, the solution found by SVM for regression is a tradeo between sparsity of the representation and closeness to the data. We also include a complete derivation of SVM for regression in the case of linear approximation.
A Geometric Nearest Point Algorithm for the Efficient Solution of the SVM Classification Task
IEEE Transactions on Neural Networks, 2000
Geometric methods are very intuitive and provide a theoretically solid approach to many optimization problems. One such optimization task is the Support Vector Machine (SVM) classification, which has been the focus of intense theoretical as well as application oriented research in Machine Learning. In this work, the incorporation of recent results in Reduced Convex Hulls (RCH) to a Nearest Point Algorithm (NPA) leads to an elegant and efficient solution to the SVM classification task, with encouraging practical results to real world classification problems, i.e., linear or non-linear, separable or non-separable.
The regularization parameter of support vector machines is intended to improve their generalization performance. Since the feasible region of binary class support vector machines with finite dimensional feature space is a polytope, we note that classifiers at vertices of this unbounded polytope correspond to certain ranges of the regularization parameter. This reduces the search for a suitable regularization parameter to a search of (finite number of) vertices of this polytope. We propose an algorithm that identifies neighbouring vertices of a given vertex and thereby identifies the classifiers corresponding to the set of vertices of this polytope. A classifier can then be chosen from them based on a suitable test error criterion. We illustrate our results with an example which demonstrates that this path can be complicated. A portion of the path is sandwiched between two finite intervals of path, each generated by separate sets of vertices and edges.
Multivariate convex support vector regression with semidefinite programming
Knowledge-Based Systems, 2012
As one of important nonparametric regression method, support vector regression can achieve nonlinear capability by kernel trick. This paper discusses multivariate support vector regression when its regression function is restricted to be convex. This paper approximates this convex shape restriction with a series of linear matrix inequality constraints and transforms its training to a semidefinite programming problem, which is computationally tractable. Extensions to multivariate concave case, ' 2-norm Regularization, ' 1 and ' 2-norm loss functions, are also studied in this paper. Experimental results on both toy data sets and a real data set clearly show that, by exploiting this prior shape knowledge, this method can achieve better performance than the classical support vector regression.
A class of new Support Vector Regression models
Applied Soft Computing, 2020
We propose a novel convex loss function termed as 'ϵ-penalty loss function', to be used in Support Vector Regression (SVR) model. The proposed ϵ-penalty loss function is shown to be optimal for a more general noise distribution. The popular ϵ-insensitive loss function and the Laplace loss function are particular cases of the proposed loss function. Making the use of the proposed loss function, we have proposed two new Support Vector Regression models in this paper. The first model which we have termed with 'ϵ-Penalty Support Vector Regression' (ϵ-PSVR) model minimizes the proposed loss function with L 2-norm regularization. The second model minimizes the proposed loss function with L 1-Norm regularization and has been termed as 'L 1-Norm Penalty Support Vector Regression' (L 1-Norm PSVR) model. The proposed loss function can offer different rates of penalization inside and outside of the ϵ-tube. This strategy enables the proposed SVR models to use the full information of the training set which make them to generalize well. Further, the numerical results obtained from the experiments carried out on various artificial, benchmark datasets and financial time series datasets show that the proposed SVR models own better generalization ability than existing SVR models.