Humans integrate visual and haptic information in a statistically optimal fashion (original) (raw)

Nature volume 415, pages 429–433 (2002) Cite this article

Abstract

When a person looks at an object while exploring it with their hand, vision and touch both provide information for estimating the properties of the object. Vision frequently dominates the integrated visual–haptic percept, for example when judging size, shape or position1,2,3, but in some circumstances the percept is clearly affected by haptics4,5,6,7. Here we propose that a general principle, which minimizes variance in the final estimate, determines the degree to which vision or haptics dominates. This principle is realized by using maximum-likelihood estimation8,9,10,11,12,13,14,15 to combine the inputs. To investigate cue combination quantitatively, we first measured the variances associated with visual and haptic estimation of height. We then used these measurements to construct a maximum-likelihood integrator. This model behaved very similarly to humans in a visual–haptic task. Thus, the nervous system seems to combine visual and haptic information in a fashion that is similar to a maximum-likelihood integrator. Visual dominance occurs when the variance associated with visual estimation is lower than that associated with haptic estimation.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 51 print issues and online access

$199.00 per year

only $3.90 per issue

Buy this article

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Additional access options:

Figure 1: Maximum-likelihood estimation integration: two hypothetical situations.

The alternative text for this image may have been generated using AI.

Figure 2: Apparatus and stimuli.

The alternative text for this image may have been generated using AI.

Figure 3: Predictions and experimental data.

The alternative text for this image may have been generated using AI.

Similar content being viewed by others

References

  1. Rock, I. & Victor, J. Vision and touch: An experimentally created conflict between the two senses. Science 143, 594–596 (1964).
    Article ADS CAS Google Scholar
  2. Hay, J. C., Pick, H. L. Jr & Ikeda, K. Visual capture produced by prism spectacles. Psychonomic Sci. 2, 215–216 (1965).
    Article Google Scholar
  3. Warren, D. H. & Rossano, M. J. in The Psychology of Touch (eds Heller, M. A. & Schiff, W.) 119–137 (Erlbaum, Hillsdale, New Jersey, 1991).
    Google Scholar
  4. Power, R. P. The dominance of touch by vision: Sometimes incomplete. Perception 9, 457–466 (1980).
    Article CAS Google Scholar
  5. Welch, R. B. & Warren, D. H. Immediate perceptual response to intersensory discrepancy. Psychol. Bull. 88, 638–667 (1980).
    Article CAS Google Scholar
  6. Lederman, S. J. & Abbott, S. G. Texture perception: Studies of intersensory organization using a discrepancy paradigm, and visual versus tactual psychophysics. J. Exp. Psychol. Hum. Percept. Perform. 7, 902–915 (1981).
    Article CAS Google Scholar
  7. Heller, M. A. Haptic dominance in form perception with blurred vision. Perception 12, 607–613 (1983).
    Article CAS Google Scholar
  8. Clark, J. J. & Yuille, A. L. Data Fusion for Sensory Information Processing Systems (Kluwer Academic, Boston, 1990).
    Book Google Scholar
  9. Blake, A., Bülthoff, H. H. & Sheinberg, D. Shape from texture: Ideal observer and human psychophysics. Vision Res. 33, 1723–1737 (1993).
    Article CAS Google Scholar
  10. Landy, M. S., Maloney, L. T., Johnston, E. B. & Young, M. Measurement and modeling of depth cue combination: In defense of weak fusion. Vision Res. 35, 389–412 (1995).
    Article CAS Google Scholar
  11. Gharamani, Z., Wolpert, D. M. & Jordan, M. I. in Self-organization, Computational Maps, and Motor Control (eds Morasso, P. G. & Sanguineti, V.) 117–147 (Elsevier, Amsterdam, 1997).
    Book Google Scholar
  12. Knill, D. C. Discrimination of planar surface slant from texture: Human and ideal observers compared. Vision Res. 38, 1683–1697 (1998).
    Article CAS Google Scholar
  13. Backus, B. T. & Banks, M. S. Estimator reliability and distance scaling in stereoscopic slant perception. Perception 28, 417–442 (1999).
    Article Google Scholar
  14. van Beers, R. J., Sittig, A. C. & Denier van der Gon, J. J. Integration of proprioceptive and visual position information: An experimentally supported model. J. Neurophysiol. 81, 1355–1364 (1999).
    Article CAS Google Scholar
  15. Schrater, P. R. & Kersten, D. How optimal depth cue integration depends on the task. Int. J. Comp. Vis. 40, 71–89 (2000).
    Article Google Scholar
  16. Gibson, J. J. Adaptation, after-effect, and contrast in the perception of curved lines. J. Exp. Psychol. 16, 1–31 (1933).
    Article Google Scholar
  17. Festinger, L., Burnham, C. A., Ono, H. & Bamber, D. Efference and the conscious experience of perception. J. Exp. Psychol. 74 (4), 1–36 (1967).
    Article Google Scholar
  18. Singer, G. & Day, R. H. Visual capture of haptically judged depth. Percept. Psychophys. 5, 315–316 (1969).
    Article Google Scholar
  19. Tastevin, J. En partant de l’experience d’Aristote. L’Encephale 1, 57–84 (1937).
    Google Scholar
  20. Mon-Williams, M., Wann, J. P., Jenkinson, M. & Rushton, K. Synaesthesia in the normal limb. Proc. R. Soc. Lond. B 264, 1007–1010 (1997).
    Article ADS CAS Google Scholar
  21. Pavani, F., Spence, C. & Driver, J. Visual capture of touch: out-of-the-body experiences with rubber gloves. Psycholog. Sci. 11, 353–359 (2000).
    Article CAS Google Scholar
  22. Heller, M. A. Visual and tactual texture perception: Intersensory cooperation. Percept. Psychophys. 31, 339–344 (1982).
    Article CAS Google Scholar
  23. Banks, M. S. & Backus, B. T. Extra-retinal and perspective cues cause the small range of the induced effect. Vision Res. 38, 187–194 (1998).
    Article CAS Google Scholar
  24. Ernst, M. O., Banks, M. S. & Bülthoff, H. H. Touch can change visual slant perception. Nature Neurosci. 3, 69–73 (2000).
    Article CAS Google Scholar
  25. Peña, J. L. & Konishi, M. Auditory spatial receptive fields created by multiplication. Science 292, 249–252 (2001).
    Article ADS Google Scholar

Download references

Acknowledgements

We thank M. Landy for comments on the manuscript; and H. Ernst, X. Moncada, C. Alderson and S. Kashiwada for participating as observers. This research was supported by research grants from Air Force Office of Scientific Research and the National Institutes of Health, and by an equipment grant from Silicon Graphics.

Author information

Author notes

  1. Marc O. Ernst
    Present address: Max Planck Institute for Biological Cybernetics, Tübingen, 72076, Germany
  2. Marc O. Ernst: Correspondence and requests for materials should be addressed to M.O.E. (e-mail: marc.ernst@tuebingen.mpg.de).

Authors and Affiliations

  1. Vision Science Program/School of Optometry, University of California, Berkeley, 94720-2020, USA
    Marc O. Ernst & Martin S. Banks

Authors

  1. Marc O. Ernst
  2. Martin S. Banks

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Rights and permissions

About this article

Cite this article

Ernst, M., Banks, M. Humans integrate visual and haptic information in a statistically optimal fashion.Nature 415, 429–433 (2002). https://doi.org/10.1038/415429a

Download citation

This article is cited by