Identifying natural images from human brain activity (original) (raw)

Nature volume 452, pages 352–355 (2008)Cite this article

Abstract

A challenging goal in neuroscience is to be able to read out, or decode, mental content from brain activity. Recent functional magnetic resonance imaging (fMRI) studies have decoded orientation1,2, position3 and object category4,5 from activity in visual cortex. However, these studies typically used relatively simple stimuli (for example, gratings) or images drawn from fixed categories (for example, faces, houses), and decoding was based on previous measurements of brain activity evoked by those same stimuli or categories. To overcome these limitations, here we develop a decoding method based on quantitative receptive-field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas. These models describe the tuning of individual voxels for space, orientation and spatial frequency, and are estimated directly from responses evoked by natural images. We show that these receptive-field models make it possible to identify, from a large set of completely novel natural images, which specific image was seen by an observer. Identification is not a mere consequence of the retinotopic organization of visual areas; simpler receptive-field models that describe only spatial tuning yield much poorer identification performance. Our results suggest that it may soon be possible to reconstruct a picture of a person’s visual experience from measurements of brain activity alone.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 51 print issues and online access

$199.00 per year

only $3.90 per issue

Buy this article

Prices may be subject to local taxes which are calculated during checkout

Additional access options:

Similar content being viewed by others

References

  1. Haynes, J. D. & Rees, G. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nature Neurosci. 8, 686–691 (2005)
    Article CAS Google Scholar
  2. Kamitani, Y. & Tong, F. Decoding the visual and subjective contents of the human brain. Nature Neurosci. 8, 679–685 (2005)
    Article CAS Google Scholar
  3. Thirion, B. et al. Inverse retinotopy: inferring the visual content of images from brain activation patterns. Neuroimage 33, 1104–1116 (2006)
    Article Google Scholar
  4. Cox, D. D. & Savoy, R. L. Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. Neuroimage 19, 261–270 (2003)
    Article Google Scholar
  5. Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293, 2425–2430 (2001)
    Article ADS CAS Google Scholar
  6. Haynes, J. D. & Rees, G. Decoding mental states from brain activity in humans. Nature Rev. Neurosci. 7, 523–534 (2006)
    Article CAS Google Scholar
  7. Hung, C. P., Kreiman, G., Poggio, T. & DiCarlo, J. J. Fast readout of object identity from macaque inferior temporal cortex. Science 310, 863–866 (2005)
    Article ADS CAS Google Scholar
  8. Tsao, D. Y., Freiwald, W. A., Tootell, R. B. & Livingstone, M. S. A cortical region consisting entirely of face-selective cells. Science 311, 670–674 (2006)
    Article ADS CAS Google Scholar
  9. Simoncelli, E. P. & Olshausen, B. A. Natural image statistics and neural representation. Annu. Rev. Neurosci. 24, 1193–1216 (2001)
    Article CAS Google Scholar
  10. Wu, M. C., David, S. V. & Gallant, J. L. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neurosci. 29, 477–505 (2006)
    Article CAS Google Scholar
  11. Daugman, J. G. Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. J. Opt. Soc. Am. A 2, 1160–1169 (1985)
    Article ADS CAS Google Scholar
  12. Jones, J. P. & Palmer, L. A. An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. J. Neurophysiol. 58, 1233–1258 (1987)
    Article CAS Google Scholar
  13. Lee, T. S. Image representation using 2D Gabor wavelets. IEEE Trans. Pattern Anal. 18, 959–971 (1996)
    Article Google Scholar
  14. DeYoe, E. A. et al. Mapping striate and extrastriate visual areas in human cerebral cortex. Proc. Natl Acad. Sci. USA 93, 2382–2386 (1996)
    Article ADS CAS Google Scholar
  15. Dumoulin, S. O. & Wandell, B. A. Population receptive field estimates in human visual cortex. Neuroimage 39, 647–660 (2008)
    Article Google Scholar
  16. Engel, S. A. et al. fMRI of human visual cortex. Nature 369, 525 (1994)
    Article ADS CAS Google Scholar
  17. Hansen, K. A., David, S. V. & Gallant, J. L. Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response. Neuroimage 23, 233–241 (2004)
    Article Google Scholar
  18. Sereno, M. I. et al. Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging. Science 268, 889–893 (1995)
    Article ADS CAS Google Scholar
  19. Smith, A. T., Singh, K. D., Williams, A. L. & Greenlee, M. W. Estimating receptive field size from fMRI data in human striate and extrastriate visual cortex. Cereb. Cortex 11, 1182–1190 (2001)
    Article CAS Google Scholar
  20. Sasaki, Y. et al. The radial bias: a different slant on visual orientation sensitivity in human and nonhuman primates. Neuron 51, 661–670 (2006)
    Article CAS Google Scholar
  21. Olman, C. A., Ugurbil, K., Schrater, P. & Kersten, D. BOLD fMRI and psychophysical measurements of contrast response to broadband images. Vision Res. 44, 669–683 (2004)
    Article Google Scholar
  22. Singh, K. D., Smith, A. T. & Greenlee, M. W. Spatiotemporal frequency and direction sensitivities of human visual areas measured using fMRI. Neuroimage 12, 550–564 (2000)
    Article CAS Google Scholar
  23. Haynes, J. D. & Rees, G. Predicting the stream of consciousness from activity in human visual cortex. Curr. Biol. 15, 1301–1307 (2005)
    Article CAS Google Scholar
  24. Heeger, D. J. & Ress, D. What does fMRI tell us about neuronal activity? Nature Rev. Neurosci. 3, 142–151 (2002)
    Article CAS Google Scholar
  25. Logothetis, N. K. & Wandell, B. A. Interpreting the BOLD signal. Annu. Rev. Physiol. 66, 735–769 (2004)
    Article CAS Google Scholar
  26. Stanley, G. B., Li, F. F. & Dan, Y. Reconstruction of natural scenes from ensemble responses in the lateral geniculate nucleus. J. Neurosci. 19, 8036–8042 (1999)
    Article CAS Google Scholar
  27. Haynes, J. D., Lotto, R. B. & Rees, G. Responses of human visual cortex to uniform surfaces. Proc. Natl Acad. Sci. USA 101, 4286–4291 (2004)
    Article ADS CAS Google Scholar
  28. Rainer, G., Augath, M., Trinath, T. & Logothetis, N. K. Nonmonotonic noise tuning of BOLD fMRI signal to natural images in the visual cortex of the anesthetized monkey. Curr. Biol. 11, 846–854 (2001)
    Article CAS Google Scholar
  29. Salinas, E. & Abbott, L. F. Vector reconstruction from firing rates. J. Comput. Neurosci. 1, 89–107 (1994)
    Article CAS Google Scholar
  30. Zhang, K., Ginzburg, I., McNaughton, B. L. & Sejnowski, T. J. Interpreting neuronal population activity by reconstruction: unified framework with application to hippocampal place cells. J. Neurophysiol. 79, 1017–1044 (1998)
    Article CAS Google Scholar

Download references

Acknowledgements

This work was supported by a National Defense Science and Engineering Graduate fellowship (K.N.K.), the National Institutes of Health, and University of California, Berkeley intramural funds. We thank B. Inglis for assistance with MRI, K. Hansen for assistance with retinotopic mapping, D. Woods and X. Kang for acquisition of whole-brain anatomical data, and A. Rokem for assistance with scanner operation. We also thank C. Baker, M. D’Esposito, R. Ivry, A. Landau, M. Merolle and F. Theunissen for comments on the manuscript. Finally, we thank S. Nishimoto, R. Redfern, K. Schreiber, B. Willmore and B. Yu for their help in various aspects of this research.

Author Contributions K.N.K. designed and conducted the experiment and was first author on the paper. K.N.K. and T.N. analysed the data. R.J.P. provided mathematical ideas and assistance. J.L.G. provided guidance on all aspects of the project. All authors discussed the results and commented on the manuscript.

Author information

Authors and Affiliations

  1. Department of Psychology, University of California, Berkeley, California 94720, USA,
    Kendrick N. Kay & Jack L. Gallant
  2. Helen Wills Neuroscience Institute, University of California, Berkeley, California 94720, USA,
    Thomas Naselaris & Jack L. Gallant
  3. Department of Physics, University of California, Berkeley, California 94720, USA,
    Ryan J. Prenger

Authors

  1. Kendrick N. Kay
    You can also search for this author inPubMed Google Scholar
  2. Thomas Naselaris
    You can also search for this author inPubMed Google Scholar
  3. Ryan J. Prenger
    You can also search for this author inPubMed Google Scholar
  4. Jack L. Gallant
    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toJack L. Gallant.

Supplementary information

Supplementary Information

This file contains Supplementary Figures 1-11 with Legends, Supplementary Table 1, Supplementary Discussion, Supplementary Methods, and Supplementary Notes with additional references. (PDF 3551 kb)

Rights and permissions

About this article

Cite this article

Kay, K., Naselaris, T., Prenger, R. et al. Identifying natural images from human brain activity.Nature 452, 352–355 (2008). https://doi.org/10.1038/nature06713

Download citation

Editorial Summary

Reading the mind

Recent functional magnetic resonance imaging (fMRI) studies have shown that, based on patterns of activity evoked by different categories of visual images, it is possible to deduce simple features in the visual scene, or to which category it belongs. Kay et al. take this approach a tantalizing step further. Their newly developed decoding method, based on quantitative receptive field models that characterize the relationship between visual stimuli and fMRI activity in early visual areas, can identify with high accuracy which specific natural image an observer saw, even for an image chosen at random from 1,000 distinct images. This prompts the thought that it may soon be possible to decode subjective perceptual experiences such as visual imagery and dreams, an idea previously restricted to the realm of science fiction.