Using the correlation criterion to position and shape RBF units for incremental modelling (original) (raw)

Abstract

A novel technique is proposed for the incremental construction of sparse radial basis function (RBF) networks. The correlation between an RBF regressor and the training data is used as the criterion to position and shape the RBF node, and it is shown that this is equivalent to incrementally minimise the modelling mean square error. A guided random search optimisation method, called the repeated weighted boosting search, is adopted to append RBF nodes one by one in an incremental regression modelling procedure. The experimental results obtained using the proposed method demonstrate that it provides a viable alternative to the existing state-of-the-art modelling techniques for constructing parsimonious RBF models that generalise well.

Access this article

Log in via an institution

Subscribe and save

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. S. Chen, S. A. Billings, W. Luo. Orthogonal Least Squares Methods and Their Application to Non-linear System Identification. International Journal of Control, vol. 50, no. 5, pp. 1873–1896, 1989.
    MathSciNet Google Scholar
  2. S. Chen, C. F. N. Cowan, P. M. Grant. Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks. IEEE Transactions on Neural Networks, vol. 2, no. 2, pp. 302–309, 1991.
    Article Google Scholar
  3. M. J. L. Orr. Regularization in the Selection of Radial Basis Function Centres. Neural Computation, vol. 7, no. 3, pp. 606–623, 1995.
    Google Scholar
  4. S. Chen, E. S. Chng, K. Alkadhimi. Regularised Orthogonal Least Squares Algorithm for Constructing Radial Basis Function Networks. International Journal of Control, vol. 64, no. 5, pp. 829–837, 1996.
    MathSciNet Google Scholar
  5. S. Chen, Y. Wu, B. L. Luk. Combined Genetic Algorithm Optimisation and Regularised Orthogonal Least Squares Learning for Radial Basis Function Networks. IEEE Transactions on Neural Networks, vol. 10, no. 5, pp. 1239–1243, 1999.
    Article Google Scholar
  6. S. Chen, X. Hong, C. J. Harris. Sparse Kernel Regression Modelling Using Combined Locally Regularized Orthogonal Least Squares and D-Optimality Experimental Design. IEEE Transactions on Automatic Control, vol. 48, no. 6, pp. 1029–1036, 2003.
    Article MathSciNet Google Scholar
  7. S. Chen, X. Hong, C. J. Harris, P. M. Sharkey. Sparse Modelling Using Orthogonal Forward Regression with PRESS Statistic and Regularization. IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 34, no. 2, pp. 898–911, 2004.
    Article Google Scholar
  8. V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, New York, 1995.
    MATH Google Scholar
  9. S. Gunn. Support Vector Machines for Classification and Regression. Technical Report, Information: Signals, Images, Systems (ISIS) Research Group, Department of Electronics and Computer Science, University of Southampton, UK, May 1998.
    Google Scholar
  10. S. S. Chen, D. L. Donoho, M. A. Saunders. Atomic Decomposition by Basis Pursuit. SIAM Review, vol. 43, no. 1, pp. 129–159, 2001.
    Article MathSciNet Google Scholar
  11. M. E. Tipping. Sparse Bayesian Learning and the Relevance Vector Machine. Journal of Machine Learning Research, vol. 1, pp. 211–244, 2001.
    Article MathSciNet Google Scholar
  12. B. Schölkopf, A. J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, 2002.
    Google Scholar
  13. P. Vincent, Y. Bengio. Kernel Matching Pursuit. Machine Learning, vol. 48, no. 1, pp. 165–187, 2002.
    Article Google Scholar
  14. G. R. G. Lanckriet, N. Cristianini, P. Bartlett, L. E. Ghaoui, M. I. Jordan. Learning the Kernel Matrix with Semidefinite Programming. Journal of Machine Learning Research, vol. 5, pp. 27–72, 2004.
    Google Scholar
  15. T. Doddy. Priori Knowledge for Time Series Modeling. Ph.D. dissertation, Department of Electronics and Computer Science, University of Southampton, U.K., 2000.
    Google Scholar
  16. M. Stone. Cross Validation Choice and Assessment of Statistical Predictions. Journal of Royal Statistics Society, Series B, vol. 36, pp. 117–147, 1974.
    Google Scholar
  17. R. H. Myers. Classical and Modern Regression with Applications. 2nd ed., PWS-KENT, Boston, 1990.
    Google Scholar
  18. S. Lee, R. M. Kil. A Gaussian Potential Function Network with Hierarchically Self-Organizing Learning. Neural Networks, vol. 4, no. 2, pp. 207–224, 1991.
    Article Google Scholar
  19. S. Chen, X. Hong, C. J. Harris, X. X. Wang. Identification of Nonlinear Systems Using Generalized Kernel Models. IEEE Transactions on Control Systems Technology, vol. 13, no. 3, pp. 401–411, 2005.
    Article Google Scholar
  20. G. B. Huang, P. Saratchandran, N. Sundararajan. A Generalized Growing and Pruning RBF (GGAP-RBF) Neural Network for Function Approximation. IEEE Transactions on Neural Networks, vol. 16, no. 1, pp. 57–67, 2005.
    Article Google Scholar
  21. J. Moody, C. J. Darken. Fast Learning in Networks of Locally-Tuned Processing Units. Neural Computation, vol. 1, pp. 281–294, 1989.
    Google Scholar
  22. S. Chen, S. A. Billings, P. M. Grant. Recursive Hybrid Algorithm for Non-linear System Identification Using Radial Basis Function Networks. International Journal of Control, vol. 55, pp. 1051–1070, 1992.
    MathSciNet Google Scholar
  23. S. Chen. Nonlinear Time Series Modelling and Prediction Using Gaussian RBF Networks with Enhanced Clustering and RLS Learning. Electronics Letters, vol. 31, no. 2, pp. 117–118, 1995.
    Article Google Scholar
  24. P. E. An, M. Brown, S. Chen, C. J. Harris. Comparative Aspects of Neural Network Algorithms for On-line Modelling of Dynamic Processes. Proceedings of the Institution of Mechanical Engineers—Part I: Journal of Systems and Control Engineering, vol. 207, pp. 223–241, 1993.
    Google Scholar
  25. B. A. Whitehead, T. D. Choate. Evolving Space-filling Curves to Distribute Radial Basis Functions over an Input Space. IEEE Transactions on Neural Networks, vol. 5, no. 1, pp. 15–23, 1994.
    Article Google Scholar
  26. S. Chen, X. X. Wang, C. J. Harris. Experiments with Repeating Weighted Boosting Search for Optimization in Signal Processing Applications. IEEE Transactions on Systems, Man, and Cybernetics, Part B, vol. 35, no. 4, pp. 682–693, 2005.
    Article Google Scholar
  27. S. E. Fahlman, C. Lebiere. The Cascade-Correlation Learning Architecture. Neural Information Processing Systems 2, D.S. Touretzky, Ed., Morgan-Kaufmann, San Fransisco, CA, pp. 524–532, 1990.
    Google Scholar
  28. H. Akaike. A New Look at the Statistical Model Identification. IEEE Transactions on Automatic Control, vol. 19, pp. 716–723, 1974.
    Article MathSciNet Google Scholar
  29. I. J. Leontaritis, S. A. Billings. Model Selection and Validation Methods for Non-linear Systems. International Journal of Control, vol. 45, no. 1, pp. 311–341, 1987.
    Google Scholar
  30. X. Hong, C. J. Harris. Nonlinear Model Structure Design and Construction Using Orthogonal Least Squares and D-Optimality Design. IEEE Transactions on Neural Networks, vol. 13, no. 5, pp. 1245–1250, 2002.
    Article Google Scholar
  31. D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison Wesley, Reading, MA, 1989.
    MATH Google Scholar
  32. K. F. Man, K. S. Tang, S. Kwong. Genetic Algorithms: Concepts and Design. Springer-Verlag, London, 1998.
    Google Scholar
  33. L. Ingber. Simulated Annealing: Practice Versus Theory. Mathematical and Computer Modeling, vol. 18, no. 11, pp. 29–57, 1993.
    Article MathSciNet Google Scholar
  34. S. Chen, B. L. Luk. Adaptive Simulated Annealing for Optimization in Signal Processing Applications. Signal Processing, vol. 79, no. 1, pp. 117–128, 1999.
    Article MATH Google Scholar
  35. S. A. Billings, S. Chen, R. J. Backhouse. The Identification of Linear and Non-linear Models of a Turbocharged Automotive Diesel Engine. Mechanical Systems and Signal Processing, vol. 3, no. 2, pp. 123–142, 1989.
    Article Google Scholar
  36. G. E. P. Box, G. M. Jenkins. Time Series Analysis, Forecasting and Control. Holden Day, San Francisco, 1976.
    MATH Google Scholar

Download references

Author information

Authors and Affiliations

  1. Neural Computing Research Group, Aston University, Birmingham, B4 7ET, UK
    Xun-Xian Wang
  2. School of Electronics and Computer Science University of Southampton, Southampton, SO17 1BJ, UK
    Sheng Chen & Chris J. Harris

Authors

  1. Xun-Xian Wang
  2. Sheng Chen
  3. Chris J. Harris

Corresponding author

Correspondence toSheng Chen.

Additional information

Xun-Xian Wang received his Ph.D. degree in the control theory and application field from Tsinghua University, Beijing, China, in July 1999. From August 1999 to August 2001, he was a postdoctoral researcher in the State Key Laboratory of Intelligent Technology and Systems, Beijing, China. From September 2001 to December 2004, he was a research fellow at the University of Portsmouth, Portsmouth, U.K. Since January 2005, he has been a research fellow at Neural Computing Research Group, Aston University, Birmingham, U.K.

His main research interests include machine learning and neural networks, control theory and systems as well as robotics.

Sheng Chen received his Ph.D. degree in control engineering from the City University, London, U.K., in 1986. He joined the School of Electronics and Computer Science, University of Southampton, Southampton, U.K., in September 1999. He previously held research and academic appointments at the University of Sheffield, the University of Edinburgh, and the University of Portsmouth, all in U.K.

He has published over 260 research papers. His research works include wireless communications, machine learning and neural networks, finite-precision digital controller design, and evolutionary computation methods.

Professor Chen holds a higher doctorate degree, D.Sc, from the University of Southampton. In the database of the world’s most highly cited researchers, compiled by Institute for Scientific Information (ISI) of the USA, Dr. Chen is on the list of the highly cited researchers in the engineering category.

Chris J. Harris received his Ph.D. degree from the University of Southampton, Southampton, U.K. He previously held appointments at the University of Hull, the UMIST, the University of Oxford, and the University of Cranfield, all in U.K., as well as being employed by the U.K. Ministry of Defence. He returned to the University of Southampton as the Lucas Professor of Aerospace Systems Engineering in 1987 to establish the Advanced Systems Research Group and, more recently, Image, Speech and Intelligent Systems Group.

He has authored and co-authored 12 research books and over 400 research papers, and he is the associate editor of numerous international journals. His research interests include the general area of intelligent and adaptive systems theory and its application to intelligent autonomous systems such as autonomous vehicles, management infrastructures such as command & control, intelligent control, and estimation of dynamic processes, multi-sensor data fusion, and systems integration.

Professor Harris holds a higher doctorate degree, D.Sc, from the University of Southampton. Dr. Harris was elected to the Royal Academy of Engineering in 1996, was awarded the IEE Senior Achievement medal in 1998 for his work in autonomous systems, and the highest international award in IEE, the IEE Faraday medal, in 2001 for his work in intelligent control and neurofuzzy systems.

Rights and permissions

About this article

Cite this article

Wang, XX., Chen, S. & Harris, C.J. Using the correlation criterion to position and shape RBF units for incremental modelling.Int J Automat Comput 3, 392–403 (2006). https://doi.org/10.1007/s11633-006-0392-2

Download citation

Keywords