Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation (original) (raw)

Abstract

A new method of pruning away hidden neurons in neural networks is presented in this paper. The hidden neuron is removed by analyzing the orthogonal projection correlations among the outputs of other hidden neurons. The method guarantees the least loss of weight information in terms of orthogonal projection. The remaining weights and thresholds are updated based on the weight crosswise propagation. A practical technique for penalizing the superfluous hidden neurons is explored. Retraining is needed after pruning. Extensive experiments are conducted, and the results demonstrate that the method gives better initial points for retraining and retraining costs less epochs.

Access this article

Log in via an institution

Subscribe and save

• Starting from 10 chapters or articles per month
• Access and download chapters and articles from more than 300k books and 2,500 journals
• Cancel anytime View plans

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

1. Muller B, Reinhardt J (1990) Neural networks—an introduction. Springer, Berlin Heidelberg New York
   Google Scholar
2. Hirose Y, Koichi Y, Hijiya S (1991) Back-propagation algorithm which varies the number of hidden units. Neural Netw 4:61–66
   Article Google Scholar
3. Eigel-Danielson V, Augusteijn MF (1993) Neural network pruning and its effect on generalization, some experimental results. Neural Parallel Sci Comput 1(1):59–70
   MATH Google Scholar
4. Engelbrecht AP, Fetcher L, Cloete I (1999) Variance analysis of sensitivity information for pruning multilayer feedforward neural networks. In: Proceedings of the international joint conference on neural networks, Washington, pp. 379–385
5. Engelbrecht AP (2001) A new pruning heuristic based on variance analysis of sensitivity information. IEEE Trans Neural Netw 12(6):1386–1399
   Article Google Scholar
6. Liang X, Bode J, Wang X (1995) Network expansion and network compression—further discussion on structure optimization methods. In: Proceedings of the IEEE international conference on neural networks, Perth, pp. 680–685
7. Setiono R (1997) A penalty-function approach for pruning feedforward neural networks. Neural Comput 9(1):185–204
   Article MATH Google Scholar
8. Cun YL, Denker JS, Solla SA (1989) Optimal brain damage. In: Proceedings of the IEEE conference on neural information processing systems, Denver, pp. 598–605
9. Hassibi B, Stork DG (1992) Second order derivatives for network pruning: optimal brain surgeon. In: Proceedings of the neural information processing systems, vol. 5, pp. 293–299
10. Hagiwara M (1994) A simple and effective method for removal of hidden units and weights. Neurocomputing 6:207–218
    Article Google Scholar
11. Cantu-Paz E (2003) Pruning neural networks with distribution estimation algorithms. In: Cantu-Paz E (ed) Lecture notes in computer science, vol. 2723. Springer, Berlin Heidelberg New York, pp. 790–800
12. Reed R (1993) Pruning algorithms—a survey. IEEE Trans Neural Netw 4(5):740–747
    Article Google Scholar
13. Laar V, Heskes J (1999) Pruning using parameter and neuronal metrics. Neural Comput 11:977–993
    Article Google Scholar
14. Fetcher L, Katkovnik V, Steffens FE (1998) Optimizing the number of hidden nodes of a feedforward artificial neural network. In: Proceedings of the IEEE world congress on computational intelligence, Anchorage, pp. 1608–1612
15. Liang X (1993) Methods of digging tunnels into the error hypersurfaces. Neural Parallel Sci Comput 1(4):381–394
    MATH MathSciNet Google Scholar
16. Liang X (1995) Network expansion and network compression—further discussion on structure optimization methods. In: Proceedings of the IEEE international conference on neural networks, Perth, pp. 680–685
17. Liang X, Xia S (1995) Methods of training and constructing multilayer perceptrons with arbitrary pattern sets. Int J Neural Syst 6(3):233–247
    Article Google Scholar
18. Liang X (2004) A study of removing hidden neurons in cascade-correlation neural networks. In: Proceedings of the international joint conference on neural networks, Budapest, pp. 1015–1020
19. Ben-Israel A, Greville TE (1974) Generalized inverses—theory and application. Wiley-Interscience, New York
    Google Scholar
20. Devito CL (1990) Functional analysis and linear operator theory. Addison-Wesley, Redwood City
    MATH Google Scholar
21. Barnett S (1971) Matrices in control theory. Van Nostrand Reinhold, London
    MATH Google Scholar
22. Bauer FL (1971) Elimination with weighted row, combinations for solving linear equations and least square problems. In: Wilkinson JH, Reinsch C (eds) Linear algebra. Springer, Berlin Heidelberg New York, pp. 119–133
    Google Scholar
23. Liang X (2004) Complexity of error hypersurfaces in multilayer perceptrons with binary pattern sets. Int J Neural Syst 14(3):189–200
    Article Google Scholar
24. Liang X, Xia S, Du J (1995) How to solve N-bit encoder problem with just one hidden unit and polynomially increasing weights and thresholds. Neurocomputing 7(1):85–87
    Article Google Scholar
25. Kruglyak L (1990) How to solve the N bit encoder problem with just two hidden units. Neural Comput 2(4):399–401
    Google Scholar
26. Stork DG, Allen JD (1993) How to solve the N-bit encoder problem with just one hidden unit. Neurocomputing 5:141–143
    Article Google Scholar
27. Narendra S, Parthasarathy K (1991) Gradient methods for the optimization of dynamical systems containing neural networks. IEEE Trans Neural Netw 2(2):252–262
    Article Google Scholar
28. Pao YH, Phillips SM, Sobajic DJ (1992) Neural computing and the intelligent control systems. Int J Control 56:263–289
    MATH MathSciNet Google Scholar

Download references

Acknowledgments

The author would thank the anonymous reviewers for their valuable comments and suggestions which helped improve the paper greatly. The project is sponsored by the NSF of China under grant number 70571003.

Author information

Authors and Affiliations

1. Institute of Computer Science and Technology, Peking University, Beijing, 100871, China
   Xun Liang
2. Department of Economics and Operations Research, Stanford University, Stanford, CA, 95035, USA
   Xun Liang

Corresponding author

Correspondence toXun Liang.

Rights and permissions

About this article

Cite this article

Liang, X. Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation.Neural Comput & Applic 16, 57–68 (2007). https://doi.org/10.1007/s00521-006-0057-7

Download citation

• Received: 07 January 2005
• Accepted: 19 April 2006
• Published: 08 June 2006
• Issue date: January 2007
• DOI: https://doi.org/10.1007/s00521-006-0057-7

Keywords