A Computationally Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool (original) (raw)
Related papers
SKT: A Computationaly Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool
2006
Page 1. Presented at the 11th Online World Conference on Soft Computing in Industrial Applications September 18 ��� October 6, 2006 SKT: A Computationaly Efficient SUPANOVA: Spline Kernel Based Machine Learning Tool Boleslaw Szymanski, Lijuan Zhu, Long Han and Mark Embrechts Rensselaer Polytechnic Institute, Troy, NY 12180, USA and Alexander Ross and Karsten Sternickel Cardiomag Imaging, Inc. Schenectady, NY 12304, USA Page 2. 1. Introduction: SUPANOVA Kernels 2.
Using efficient supanova kernel for heart disease diagnosis
proc. ANNIE, 2006
Many machine learning methods focus on the quality of prediction results as their final purpose. Spline kernel based methods attempt to provide also transparency to the prediction identifying features that are important in the decision process. In this paper, we present a new heuristic for computing efficiently sparse kernel in SUPANOVA. We applied it to a benchmark Boston housing market dataset and to socially important problem of improving the detection of heart diseases in the population using a novel, non-invasive measurement of the heart activities based on magnetic field produced by the human heart. On this data, 83.7% predictions were correct, exceeding the results obtained using the standard Support Vector Machine and equivalent kernels. Equally good results were achieved by the spline kernel on a benchmark Boston housing market dataset.
Ridge Regression Learning Algorithm in Dual Variables
In this paper we study a dual version of the Ridge Regression procedure. It allows us to perform non-linear regression by constructing a linear regression function in a high dimensional feature space. The feature space representation can result in a large increase in the number of parameters used by the algorithm. In order to combat this \curse of dimensionality", the algorithm allows the use of kernel functions, as used in Support Vector methods. We also discuss a powerful family of kernel functions which is constructed using the ANOVA decomposition method from the kernel corresponding to splines with an innite number of nodes. This paper introduces a regression estimation algorithm which is a combination of these two elements: the dual version of Ridge Regression is applied to the ANOVA enhancement of the in nitenode splines. Experimental results are then presented (based on the Boston Housing data set) which indicate the performance of this algorithm relative to other algorithms.
Penalised spline support vector classifiers: computational issues
Computational Statistics, 2008
We study computational issues for support vector classification with penalised spline kernels. We show that, compared with traditional kernels, computational times can be drastically reduced in large problems making such problems feasible for sample sizes as large as ∼10 6 . The optimisation technology known as interior point methods plays a central role. Penalised spline kernels are also shown to allow simple incorporation of low-dimensional structure such as additivity. This can aid both interpretability and performance.
Journal of Global Optimization, 2014
Multivariate adaptive regression splines (MARS) has become a popular data mining (DM) tool due to its flexible model building strategy for high dimensional data. Compared to well-known others, it performs better in many areas such as finance, informatics, technology and science. Many studies have been conducted on improving its performance. For this purpose, an alternative backward stepwise algorithm is proposed through Conic-MARS (CMARS) method which uses a penalized residual sum of squares for MARS as a Tikhonov regularization problem. Additionally, by modifying the forward step of MARS via mapping approach, a time efficient procedure has been introduced by S-FMARS. Inspiring from the advantages of MARS, CMARS and S-FMARS, two hybrid methods are proposed in this study, aiming to produce time efficient DM tools without degrading their performances especially for large datasets. The resulting methods, called SMARS and SCMARS, are tested in terms of several performance criteria such as accuracy, complexity, stability and robustness via simulated and real life datasets. As a DM application, the hybrid methods are also applied to an important field of finance for predicting interest rates offered by a Turkish bank to its customers. The results show that the proposed hybrid methods, being the most time efficient 104 J Glob Optim (2014) 60:103-120 with competing performances, can be considered as powerful choices particularly for large datasets.
Nonlinear Support Vector Machines Through Iterative Majorization and I-Splines
Studies in Classification, Data Analysis, and Knowledge Organization, 2007
To minimize the primal support vector machine (SVM) problem, we propose to use iterative majorization. To do so, we propose to use iterative majorization. To allow for nonlinearity of the predictors, we use (non)monotone spline transformations. An advantage over the usual kernel approach in the dual problem is that the variables can be easily interpreted. We illustrate this with an example from the literature.
Support vector regression with ANOVA decomposition kernels
1999
Abstract Support Vector Machines using ANOVA Decomposition Kernels (SVAD)[Vapng] are a way of imposing a structure on multi-dimensional kernels which are generated as the tensor product of one-dimensional kernels. This gives more accurate control over the capacity of the learning machine (VC-dimension). SVAD uses ideas from ANOVA decomposition methods and extends them to generate kernels which directly implement these ideas.
Two New Kernel Least Squares Based Methods for Regression
2006
Kernel Ridge Regression (KRR) and the Kernel Aggregating Algorithm for Regression (KAAR) are existing regression methods based on Least Squares. KRR is a well established regression technique, while KAAR is the result of relatively recent work. KAAR is similar to KRR but with some extra regularisation that makes it predict better when the data is heavily corrupted by noise. In the general case, however, this extra regularisation is excessive and therefore KRR performs better. In this paper, two new methods for regression, Iterative KAAR (IKAAR) and Controlled KAAR (CKAAR) are introduced. IKAAR and CKAAR make it possible to control the amount of extra regularisation or to remove it completely, which makes them generalisations of both KRR and KAAR. Some properties of these new methods are proved and their predictive performance on both synthetic and real-world datasets (including the well known Boston Housing dataset) is compared to that of KRR and that of KAAR. Empirical results that have been checked for statistical significance suggest that in general both IKAAR and CKAAR make predictions that are equivalent or better than those of KRR and KAAR. 1 Introduction 2 Background 2.1 Ridge Regression 2.2 The Aggregating Algorithm for Regression 2.3 Kernel Methods 3 Methods 3.1 Motivation and Introduction 3.2 Iterative KAAR 3.3 Controlled KAAR 4 Experimental Results 4.1 Method 4.
The combination of spline and kernel estimator for nonparametric regression and its properties
Applied Mathematical Sciences, 2015
Consider additive nonparametric regression model with two predictor variables components. In the first predictor component, the regression curve is approached using Spline regression, and in the second predictor component, the regression curve is approached using Kernel regression. Random error of regression model is assumed to have independent normal distribution with zero mean and the same variance. This article provides an estimator of Spline regression curve, estimator of Kernel regression curve, and an estimator of a combination of Spline and Kernel regressions. The produced estimators are biased estimators, but all estimators are classified as linear estimators in observation. Estimator of a combination of Spline and Kernel regression depended on knot points and bandwith parameter. The best estimator of a combination of Spline and Kernel regression is found by minimizing Generalized Cross Validation (GCV) function.
Feature selection for support vector regression via Kernel penalization
2010
his paper presents a novel feature selection approach (KP-SVR) that determines a non-linear regression function with minimal error and simultaneously minimizes the number of features by penalizing their use in the dual formulation of SVR. The approach optimizes the width of an anisotropic RBF Kernel using an iterative algorithm based on the gradient descent method, eliminating features that have low relevance for the regression model. Our approach presents an explicit stopping criterion, indicating clearly when eliminating further features begins to affect negatively the model's performance. Experiments with two real-world benchmark problems demonstrate that our approach accomplishes the best performance compared to well-known feature selection methods using consistently a small number of features.