Support vector regression with ANOVA decomposition kernels (original) (raw)

A Computationally Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool

Advances in Soft Computing, 2007

Many machine learning methods just consider the quality of prediction results as their final purpose. To make the prediction process transparent (reversible), spline kernel based methods were proposed by Gunn. However, the original solution method, termed SUpport vector Parsimonious ANOVA (SUPANOVA) was computationally very complex and demanding. In this paper, we propose a new heuristic to compute the optimal sparse vector in SUPANOVA that replaces the original solver for the convex quadratic problem of very high dimensionality. The resulting system is much faster without the loss of precision, as demonstrated in this paper on two benchmarks: the iris data set and the Boston housing market data benchmark.

SKT: A Computationaly Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool

2006

Page 1. Presented at the 11th Online World Conference on Soft Computing in Industrial Applications September 18 ��� October 6, 2006 SKT: A Computationaly Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool Boleslaw Szymanski, Lijuan Zhu, Long Han and Mark Embrechts Rensselaer Polytechnic Institute, Troy, NY 12180, USA and Alexander Ross and Karsten Sternickel Cardiomag Imaging, Inc. Schenectady, NY 12304, USA Page 2. 1. Introduction: SUPANOVA Kernels 2.

A note on the decomposition methods for support vector regression

2001

Abstract The dual formulation of support vector regression involves with two closely related sets of variables. When the decomposition method is used, many existing approaches use pairs of indices from these two sets as the working set. Basically they select a base set first and then expand it so that all indices are pairs. This makes the implementation different from that for support vector classification. In addition, a larger optimization sub-problem has to be solved in each iteration.

The support vector decomposition machine

Proceedings of the 23rd international conference on Machine learning - ICML '06, 2006

In machine learning problems with tens of thousands of features and only dozens or hundreds of independent training examples, dimensionality reduction is essential for good learning performance. In previous work, many researchers have treated the learning problem in two separate phases: first use an algorithm such as singular value decomposition to reduce the dimensionality of the data set, and then use a classification algorithm such as naïve Bayes or support vector machines to learn a classifier. We demonstrate that it is possible to combine the two goals of dimensionality reduction and classification into a single learning objective, and present a novel and efficient algorithm which optimizes this objective directly. We present experimental results in fMRI analysis which show that we can achieve better learning performance and lower-dimensional representations than two-phase approaches can.

Multi-scale Support Vector Regression

2010 International Joint Conference on Neural Networks (IJCNN 2010), 2010

A multi-kernel Support Vector Machine model, called Hierarchical Support Vector Regression (HSVR), is proposed here. This is a self-organizing (by growing) multiscale version of a Support Vector Regression (SVR) model. It is constituted of hierarchical layers, each containing a standard SVR with Gaussian kernel, at decreasing scales. HSVR have been applied to a noisy synthetic dataset. The results illustrate their power in denoising the original data, obtaining an effective multiscale reconstruction of better quality than that obtained by standard SVR. Furthermore with this approach the well known problem of tuning the SVR parameters is strongly simplified.

Support vector regression methods for functional data

Progress in Pattern Recognition, …, 2008

Many regression tasks in practice dispose in low gear instance of digitized functions as predictor variables. This has motivated the development of regression methods for functional data. In particular, Naradaya-Watson Kernel (NWK) and Radial Basis Function (RBF) estimators have been recently extended to functional nonparametric regression models. However, these methods do not allow for dimensionality reduction. For this purpose, we introduce Support Vector Regression (SVR) methods for functional data. These are formulated in the framework of approximation in reproducing kernel Hilbert spaces. On this general basis, some of its properties are investigated, emphasizing the construction of nonnegative definite kernels on functional spaces. Furthermore, the performance of SVR for functional variables is shown on a real world benchmark spectrometric data set, as well as comparisons with NWK and RBF methods. Good predictions were obtained by these three approaches, but SVR achieved in addition about 20% reduction of dimensionality.

Feature selection for support vector regression via Kernel penalization

2010

his paper presents a novel feature selection approach (KP-SVR) that determines a non-linear regression function with minimal error and simultaneously minimizes the number of features by penalizing their use in the dual formulation of SVR. The approach optimizes the width of an anisotropic RBF Kernel using an iterative algorithm based on the gradient descent method, eliminating features that have low relevance for the regression model. Our approach presents an explicit stopping criterion, indicating clearly when eliminating further features begins to affect negatively the model's performance. Experiments with two real-world benchmark problems demonstrate that our approach accomplishes the best performance compared to well-known feature selection methods using consistently a small number of features.

A geometric approach to support vector regression

Neurocomputing, 2003

We develop an intuitive geometric framework for support vector regression (SVR). By examining when-tubes exist, we show that SVR can be regarded as a classiÿcation problem in the dual space. Hard and soft-tubes are constructed by separating the convex or reduced convex hulls, respectively, of the training data with the response variable shifted up and down by. A novel SVR model is proposed based on choosing the max-margin plane between the two shifted data sets. Maximizing the margin corresponds to shrinking the e ective-tube. In the proposed approach, the e ects of the choices of all parameters become clear geometrically. The kernelized model corresponds to separating the convex or reduced convex hulls in feature space. Generalization bounds for classiÿcation can be extended to characterize the generalization performance of the proposed approach. We propose a simple iterative nearest-point algorithm that can be directly applied to the reduced convex hull case in order to construct soft-tubes. Computational comparisons with other SVR formulations are also included.