Learning the parts of objects by non-negative matrix factorization (original) (raw)
- Letter
- Published: 21 October 1999
Nature volume 401, pages 788–791 (1999)Cite this article
- 76k Accesses
- 62 Altmetric
- Metrics details
Abstract
Is perception of the whole based on perception of its parts? There is psychological1 and physiological2,3 evidence for parts-based representations in the brain, and certain computational theories of object recognition rely on such representations4,5. But little is known about how brains or computers might learn the parts of objects. Here we demonstrate an algorithm for non-negative matrix factorization that is able to learn parts of faces and semantic features of text. This is in contrast to other methods, such as principal components analysis and vector quantization, that learn holistic, not parts-based, representations. Non-negative matrix factorization is distinguished from the other methods by its use of non-negativity constraints. These constraints lead to a parts-based representation because they allow only additive, not subtractive, combinations. When non-negative matrix factorization is implemented as a neural network, parts-based representations emerge by virtue of two properties: the firing rates of neurons are never negative and synaptic strengths do not change sign.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Additional access options:
Similar content being viewed by others
References
- Palmer,S. E. Hierarchical structure in perceptual representation. Cogn. Psychol. 9, 441–474 ( 1977).
Article Google Scholar - Wachsmuth,E., Oram,M. W. & Perrett, D. I. Recognition of objects and their component parts: responses of single units in the temporal cortex of the macaque. Cereb. Cortex 4, 509–522 (1994).
Article CAS PubMed Google Scholar - Logothetis,N. K. & Sheinberg,D. L. Visual object recognition. Annu. Rev. Neurosci. 19, 577 –621 (1996).
Article CAS PubMed Google Scholar - Biederman,I. Recognition-by-components: a theory of human image understanding. Psychol. Rev. 94, 115–147 (1987).
Article PubMed Google Scholar - Ullman,S. High-Level Vision: Object Recognition and Visual Cognition (MIT Press, Cambridge, MA, 1996).
Book MATH Google Scholar - Turk,M. & Pentland,A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).
Article CAS PubMed Google Scholar - Field,D. J. What is the goal of sensory coding? Neural Comput. 6, 559–601 (1994).
Article Google Scholar - Foldiak,P. & Young,M. Sparse coding in the primate cortex. The Handbook of Brain Theory and Neural Networks 895 –898 (MIT Press, Cambridge, MA, 1995 ).
Google Scholar - Olshausen,B. A. & Field,D. J. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 ( 1996).
Article ADS CAS PubMed Google Scholar - Lee,D. D. & Seung,H. S. Unsupervised learning by convex and conic coding. Adv. Neural Info. Proc. Syst. 9, 515–521 (1997).
Google Scholar - Paatero,P. Least squares formulation of robust non-negative factor analysis. Chemometr. Intell. Lab. 37, 23–35 (1997).
Article CAS Google Scholar - Nakayama,K. & Shimojo,S. Experiencing and perceiving visual surfaces. Science 257, 1357– 1363 (1992).
Article ADS CAS PubMed Google Scholar - Hinton,G. E., Dayan,P., Frey,B. J. & Neal,R. M. The “wake-sleep” algorithm for unsupervised neural networks. Science 268, 1158–1161 (1995).
Article ADS CAS PubMed Google Scholar - Salton,G. & McGill,M. J. Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983).
MATH Google Scholar - Landauer,T. K. & Dumais,S. T. The latent semantic analysis theory of knowledge. Psychol. Rev. 104, 211–240 (1997).
Article Google Scholar - Jutten,C. & Herault,J. Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture. Signal Proc. 24, 1–10 ( 1991).
Article MATH Google Scholar - Bell,A. J. & Sejnowski,T. J. An information maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129–1159 ( 1995).
Article CAS PubMed Google Scholar - Bartlett,M. S., Lades,H. M. & Sejnowski, T. J. Independent component representations for face recognition. Proc. SPIE 3299, 528–539 (1998).
Article ADS Google Scholar - Shepp,L. A. & Vardi,Y. Maximum likelihood reconstruction for emission tomography. IEEE Trans. Med. Imaging. 2, 113–122 (1982).
Article Google Scholar - Richardson,W. H. Bayesian-based iterative method of image restoration. J. Opt. Soc. Am. 62, 55–59 ( 1972).
Article ADS Google Scholar - Lucy,L. B. An iterative technique for the rectification of observed distributions. Astron. J. 74, 745–754 ( 1974).
Article ADS Google Scholar - Dempster,A. P., Laired,N. M. & Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. J. Royal Stat. Soc. 39, 1– 38 (1977).
MathSciNet MATH Google Scholar - Saul,L. & Pereira,F. Proceedings of the Second Conference on Empirical Methods n Natural Language Processing (eds Cardie, C. & Weischedel, R.) 81–89 (Morgan Kaufmann, San Francisco, 1997).
Google Scholar
Acknowledgements
We acknowledge the support of Bell Laboratories and MIT. C. Papageorgiou and T. Poggio provided us with the database of faces, and R. Sproat with the Grolier encyclopedia corpus. We thank L. Saul for convincing us of the advantages of EM-type algorithms. We have benefited from discussions with B. Anderson, K. Clarkson, R. Freund, L. Kaufman, E. Rietman, S. Roweis, N. Rubin, J. Tenenbaum, N. Tishby, M. Tsodyks, T. Tyson and M. Wright.
Author information
Authors and Affiliations
- Bell Laboratories, Lucent Technologies , Murray Hill, 07974, New Jersey, USA
Daniel D. Lee & H. Sebastian Seung - Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, 02139, Massachusetts, USA
H. Sebastian Seung
Authors
- Daniel D. Lee
You can also search for this author inPubMed Google Scholar - H. Sebastian Seung
You can also search for this author inPubMed Google Scholar
Rights and permissions
About this article
Cite this article
Lee, D., Seung, H. Learning the parts of objects by non-negative matrix factorization.Nature 401, 788–791 (1999). https://doi.org/10.1038/44565
- Received: 24 May 1999
- Accepted: 06 August 1999
- Issue Date: 21 October 1999
- DOI: https://doi.org/10.1038/44565