Function and Surface Approximation Based on Enhanced Kernel Regression for Small Sample Sets (original) (raw)
Related papers
Creation of specific-to-problem kernel functions for function approximatio
2009
Although there is a large diversity in the literature related to kernel methods, there are only a few works which do not use kernels based on Radial Basis Functions (RBF) for regression problems. The reason for that is that they present very good generalization capabilities and smooth interpolation. This paper studies an initial framework to create specific-to-problem kernels for application to regression models. The kernels are created without prior knowledge about the data to be approximated by means of a Genetic Programming algorithm. The quality of a kernel is evaluated independently of a particular model, using a modified version of a non parametric noise estimator. For a particular problem, performances of generated kernels are tested against common ones using weighted k-nn in the kernel space. Results show that the presented method produces specific-to-problem kernels that outperform the common ones for this particular case. Parallel programming is utilized to deal with large computational costs.
Two New Kernel Least Squares Based Methods for Regression
2006
Kernel Ridge Regression (KRR) and the Kernel Aggregating Algorithm for Regression (KAAR) are existing regression methods based on Least Squares. KRR is a well established regression technique, while KAAR is the result of relatively recent work. KAAR is similar to KRR but with some extra regularisation that makes it predict better when the data is heavily corrupted by noise. In the general case, however, this extra regularisation is excessive and therefore KRR performs better. In this paper, two new methods for regression, Iterative KAAR (IKAAR) and Controlled KAAR (CKAAR) are introduced. IKAAR and CKAAR make it possible to control the amount of extra regularisation or to remove it completely, which makes them generalisations of both KRR and KAAR. Some properties of these new methods are proved and their predictive performance on both synthetic and real-world datasets (including the well known Boston Housing dataset) is compared to that of KRR and that of KAAR. Empirical results that have been checked for statistical significance suggest that in general both IKAAR and CKAAR make predictions that are equivalent or better than those of KRR and KAAR. 1 Introduction 2 Background 2.1 Ridge Regression 2.2 The Aggregating Algorithm for Regression 2.3 Kernel Methods 3 Methods 3.1 Motivation and Introduction 3.2 Iterative KAAR 3.3 Controlled KAAR 4 Experimental Results 4.1 Method 4.
Higher Order Functions for Kernel Regression
Lecture Notes in Computer Science, 2014
Kernel regression is a well-established nonparametric method, in which the target value of a query point is estimated using a weighted average of the surrounding training examples. The weights are typically obtained by applying a distance-based kernel function, which presupposes the existence of a distance measure.This paper investigates the use of Genetic Programming for the evolution of task-specific distance measures as an alternative to Euclidean distance. Results on seven real-world datasets show that the generalisation performance of the proposed system is superior to that of Euclidean-based kernel regression and standard GP.
2010
Least squares support vector machines (LSSVM) with Gaussian kernel represent the most used of the kernel methods existing in the literature for regression and time series prediction. These models have a good behaviour for these types of problems due to their generalization capabilities and their smooth interpolation, but they are very dependent on the feature selection performed and their computational cost notably increases with the number of training samples. Time series prediction can be tackled as a regression problem by constructing a set of input/output data from the series; this traditional approach and the use of typical recursive or direct strategies present serious drawbacks in long-term prediction. This paper presents an alternative based on the settings of specific-to-problem kernels to be applied to time series prediction focusing on large scale prediction. A simple methodology for kernel creation based on the periodicities in time series data is proposed. An alternative to LSSVM models with lower computational cost, the Kernel Weighted K-Nearest Neighbours (KWKNN) is described for function approximation. A parallel version of KWKNN is also presented to deal with large data sets.
Toward an optimal ensemble of kernel-based approximations with engineering applications
Structural and Multidisciplinary Optimization, 2008
This paper presents a general approach toward the optimal selection and ensemble (weighted average) of kernel-based approximations to address the issue of model selection. That is, depending on the problem under consideration and loss function, a particular modeling scheme may outperform the others, and, in general, it is not known a priori which one should be selected. The surrogates for the ensemble are chosen based on their performance, favoring non-dominated models, while the weights are adaptive and inversely proportional to estimates of the local prediction variance of the individual surrogates. Using both well-known analytical test functions and, in the surrogate-based modeling of a field scale alkalisurfactant-polymer enhanced oil recovery process, the ensemble of surrogates, in general, outperformed the best individual surrogate and provided among the best predictions throughout the domains of interest.
Adaptively weighted kernel regression
Journal of Nonparametric Statistics, 2013
The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden.
An optimization perspective on kernel partial least squares regression
2003
This work provides a novel derivation based on optimization for the partial least squares (PLS) algorithm for linear regression and the kernel partial least squares (K-PLS) algorithm for nonlinear regression. This derivation makes the PLS algorithm, popularly and successfully used for chemometrics applications, more accessible to machine learning researchers. The work introduces Direct K-PLS, a novel way to kernelize PLS based on direct factorization of the kernel matrix. Computational results and discussion illustrate the relative merits of K-PLS and Direct K-PLS versus closely related kernel methods such as support vector machines and kernel ridge regression.
A functional approximation comparison between neural networks and polynomial regression
WSEAS Transactions on Mathematics archive, 2008
Multi-layered perceptron (MLP) neural networks are well known as universal approximators. They are often used as estimation tools in place of the classical statistical methods. The focus of this study is to compare the approximation ability of MLP with a traditional statistical regression model, namely the polynomial regression. Comparison among the single hidden layer MLP, double hidden layer MLP and polynomial regression is carried out on the basis of similar number of weights or parameters. The performance of these three categories is measured using fraction of variance unexplained (FVU). The closer the FVU value is to zero, the better the estimation result and this is associated with a higher degree of accuracy. From the empirical results obtained in this study, we conclude that overall polynomial regression performs slightly better than MLP for a similar number of parameter except for the complicated interaction function. Meanwhile, double hidden layer MLP outperforms single hidden layer MLP. The MLP is more appropriate than the polynomial regression in approximating the complicated interaction function.
Assessing the Performance of Ordinary Least Square and Kernel Regression
Continental J. Applied Sciences, 2020
The assessment of Ordinary Least Squares (OLS) and kernel regression on their predictive performance was studied. We used simulated data to assess the performance of estimators using small and large sample. However, the mean square error (MSE) and root mean square error (RMSE) was used to find out the most efficient among the estimated models. The results show that, when n=100 the ordinary least square is more efficient than the kernel regression due to having the least MSE and RMSE in both distributions. Whereas for n=500 the ordinary least square and the kernel regression have the same performance for normal distributed data while for lognormal, the result also shows that the kernel regression perform better than the ordinary least square. Finally, when, n=1000 the kernel regression is more efficient than the ordinary least square for having the least MSE and RMSE in both distributions. The overall results show that the kernel regression estimate is more efficient than the ordinary least square (OLS) estimate.