Quadratically Constrained Quadratic Programming for Subspace Selection in Kernel Regression Estimation (original) (raw)

Abstract

In this contribution we consider the problem of regression estimation. We elaborate on a framework based on functional analysis giving rise to structured models in the context of reproducing kernel Hilbert spaces. In this setting the task of input selection is converted into the task of selecting functional components depending on one (or more) inputs. In turn the process of learning with embedded selection of such components can be formalized as a convex-concave problem. This results in a practical algorithm that can be implemented as a quadratically constrained quadratic programming (QCQP) optimization problem. We further investigate the mechanism of selection for the class of linear functions, establishing a relationship with LASSO.

Preview

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Pelckmans, K., Goethals, I., De Brabanter, J., Suykens, J., De Moor, B.: Componentwise Least Squares Support Vector Machines. In: Wang, L. (ed.) Support Vector Machines: Theory and Applications, pp. 77–98. Springer, Heidelberg (2005)
    Google Scholar
  2. Tibshirani, R.: Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 58(1), 267–288 (1996)
    MATH MathSciNet Google Scholar
  3. Chen, S.: Basis Pursuit. PhD thesis, Department of Statistics, Stanford University (November 1995)
    Google Scholar
  4. Lanckriet, G., Cristianini, N., Bartlett, P., El Ghaoui, L., Jordan, M.: Learning the Kernel Matrix with Semidefinite Programming. The Journal of Machine Learning Research 5, 27–72 (2004)
    Google Scholar
  5. Tsang, I.: Efficient hyperkernel learning using second-order cone programming. IEEE transactions on neural networks 17(1), 48–58 (2006)
    Article MathSciNet Google Scholar
  6. Ben-Tal, A., Nemirovski, A.: Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications. Society for Industrial Mathematics (2001)
    Google Scholar
  7. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)
    MATH Google Scholar
  8. Cucker, F., Zhou, D.: Learning Theory: An Approximation Theory Viewpoint (Cambridge Monographs on Applied & Computational Mathematics). Cambridge University Press, New York (2007)
    MATH Google Scholar
  9. Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 337–404 (1950)
    Article MATH MathSciNet Google Scholar
  10. Berlinet, A., Thomas-Agnan, C.: Reproducing Kernel Hilbert Spaces in Probability and Statistics. Kluwer Academic Publishers, Dordrecht (2004)
    Google Scholar
  11. Wahba, G.: Spline Models for Observational Data. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 59. SIAM, Philadelphia (1990)
    MATH Google Scholar
  12. Gu, C.: Smoothing spline ANOVA models. Series in Statistics (2002)
    Google Scholar
  13. Chen, Z.: Fitting Multivariate Regression Functions by Interaction Spline Models. J. of the Royal Statistical Society. Series B (Methodological) 55(2), 473–491 (1993)
    MATH Google Scholar
  14. Light, W., Cheney, E.: Approximation theory in tensor product spaces. Lecture Notes in Math., vol. 1169 (1985)
    Google Scholar
  15. Takemura, A.: Tensor Analysis of ANOVA Decomposition. Journal of the American Statistical Association 78(384), 894–900 (1983)
    Article MATH MathSciNet Google Scholar
  16. Huang, J.: Functional ANOVA models for generalized regression. J. Multivariate Analysis 67, 49–71 (1998)
    Article MATH Google Scholar
  17. Lin, Y.: Tensor Product Space ANOVA Models. The Annals of Statistics 28(3), 734–755 (2000)
    Article MATH MathSciNet Google Scholar
  18. Grant, M., Boyd, S., Ye, Y.: CVX: Matlab Software for Disciplined Convex Programming (2006)
    Google Scholar
  19. Donoho, D., Johnstone, J.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81(3), 425–455 (2003)
    Article MathSciNet Google Scholar
  20. Miller, A.: Subset Selection in Regression. CRC Press, Boca Raton (2002)
    MATH Google Scholar

Download references

Author information

Authors and Affiliations

  1. K.U. Leuven, ESAT-SCD, Kasteelpark Arenberg 10, B-3001, Leuven, Belgium
    Marco Signoretto, Kristiaan Pelckmans & Johan A. K. Suykens

Authors

  1. Marco Signoretto
  2. Kristiaan Pelckmans
  3. Johan A. K. Suykens

Editor information

Véra Kůrková Roman Neruda Jan Koutník

Rights and permissions

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Signoretto, M., Pelckmans, K., Suykens, J.A.K. (2008). Quadratically Constrained Quadratic Programming for Subspace Selection in Kernel Regression Estimation. In: Kůrková, V., Neruda, R., Koutník, J. (eds) Artificial Neural Networks - ICANN 2008. ICANN 2008. Lecture Notes in Computer Science, vol 5163. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87536-9\_19

Download citation

Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Publish with us