Hierarchy and adaptivity in segmenting visual scenes (original) (raw)

Nature volume 442, pages 810–813 (2006)Cite this article

Abstract

Finding salient, coherent regions in images is the basis for many visual tasks, and is especially important for object recognition. Human observers perform this task with ease, relying on a system in which hierarchical processing seems to have a critical role1. Despite many attempts, computerized algorithms2,3,4,5 have so far not demonstrated robust segmentation capabilities under general viewing conditions. Here we describe a new, highly efficient approach that determines all salient regions of an image and builds them into a hierarchical structure. Our algorithm, segmentation by weighted aggregation, is derived from algebraic multigrid solvers for physical systems6, and consists of fine-to-coarse pixel aggregation. Aggregates of various sizes, which may or may not overlap, are revealed as salient, without predetermining their number or scale. Results using this algorithm are markedly more accurate and significantly faster (linear in data size) than previous approaches.

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 51 print issues and online access

$199.00 per year

only $3.90 per issue

Buy this article

USD 39.95

Prices may be subject to local taxes which are calculated during checkout

Additional access options:

Figure 1: SWA.

The alternative text for this image may have been generated using AI.

Figure 2: The multiscale normalized cut graph approach.

The alternative text for this image may have been generated using AI.

Figure 3: Segmentation results for eight challenging images of animals on cluttered backgrounds.

The alternative text for this image may have been generated using AI.

Figure 4: Similarity search by parts.

The alternative text for this image may have been generated using AI.

Similar content being viewed by others

References

  1. Felleman, D. J. & Van Essen, D. C. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991)
    Article CAS Google Scholar
  2. Pietikainen, M., Rosenfeld, A. & Walter, I. Split-and-link algorithms for image segmentation. Patt. Recog. 15, 287–298 (1982)
    Article Google Scholar
  3. Comanicu, D. & Meer, P. Mean shift: A robust approach toward feature space analysis. IEEE Trans. Patt. Anal. Machine Intell. 24, 603–619 (2002)
    Article Google Scholar
  4. Malik, J., Belongie, S., Leung, T. & Shi, J. Contour and texture analysis for image segmentation. Int. J. Comp. Vision 43, 7–27 (2001)
    Article Google Scholar
  5. Felzenszwalb, P. & Huttenlocher, D. Efficient graph-based image segmentation. Int. J. Comput. Vision 59, 167–181 (2004)
    Article Google Scholar
  6. Brandt, A. Algebraic multigrid theory: the symmetric case. Appl. Math. Comput. 19, 23–56 (1986)
    MathSciNet MATH Google Scholar
  7. Sharon, E., Brandt, A. & Basri, R. Fast multiscale image segmentation. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 70–77 (2000)
    Google Scholar
  8. Hubel, D. H. & Wiesel, T. N. Receptive fields and functional architecture of monkey striate cortex. J. Physiol. 195, 215–243 (1968)
    Article CAS Google Scholar
  9. Tanaka, K. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139 (1996)
    Article CAS Google Scholar
  10. Shi, J. & Malik, J. Normalized cuts and image segmentation. IEEE Trans. Patt. Anal. Machine Intell. 22, 888–905 (2000)
    Article Google Scholar
  11. Golub, G. H. & Van Loan, C. F. Matrix Computations (Johns Hopkins Univ. Press, Baltimore, 1989)
    MATH Google Scholar
  12. Brandt, A., McCormick, S. & Ruge, J. In Sparsity and its Applications (ed. Evans, D. J.) 257–284 (Cambridge Univ. Press, Cambridge, 1984)
    Google Scholar
  13. Galun, M., Sharon, E., Basri, R. & Brandt, A. Texture segmentation by multiscale aggregation of filter responses and shape elements. Proc. Int. Conf. Comput. Vision 1, 469–476 (2003)
    Google Scholar
  14. Julesz, B. Textons, the elements of texture perception, and their interactions. Nature 290, 91–97 (1981)
    Article ADS CAS Google Scholar
  15. Voorhees, H. & Poggio, T. Computing texture boundaries from images. Nature 333, 364–367 (1988)
    Article ADS CAS Google Scholar
  16. Sharon, E., Brandt, A. & Basri, R. Segmentation and boundary detection using multiscale intensity measurements. Proc. IEEE Conf. Comput. Vision Patt. Recog. 1, 469–476 (2001)
    Google Scholar
  17. Stanley, D. A. & Rubin, N. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331 (2003)
    Article CAS Google Scholar

Download references

Acknowledgements

Research was supported in part by the European Commission Project Aim Shape, the Binational Science foundation, and by the German–Israeli Foundation. D.S. was supported by a grant from the National Institutes of Health. The research was conducted at the Moross Laboratory for Vision and Motor Control at the Weizmann Institute of Science. We thank N. Rubin and D. Jacobs for many useful remarks, and S. Geman for commenting on an earlier version of the manuscript. We are grateful to E. Borenstein for his help with constructing the sunglasses search system. We also thank M. Varma and R. Deitch for help with the comparisons presented in the Supplementary Information and N. Brandt for help with the graphics.

Author information

Authors and Affiliations

  1. Department of Computer Science and Applied Mathematics, The Weizmann Institute of Science, 76100, Rehovot, Israel
    Eitan Sharon, Meirav Galun, Ronen Basri & Achi Brandt
  2. Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts, 02129, USA
    Dahlia Sharon

Authors

  1. Eitan Sharon
  2. Meirav Galun
  3. Dahlia Sharon
  4. Ronen Basri
  5. Achi Brandt

Ethics declarations

Competing interests

Reprints and permissions information is available at npg.nature.com/reprintsandpermissions. The authors declare no competing financial interests. Correspondence and requests for materials should be addressed to E.S. (eitan.sharon@weizmann.ac.il).

Supplementary information

Rights and permissions

About this article

Cite this article

Sharon, E., Galun, M., Sharon, D. et al. Hierarchy and adaptivity in segmenting visual scenes.Nature 442, 810–813 (2006). https://doi.org/10.1038/nature04977

Download citation

This article is cited by

Editorial Summary

Seeing things

Humans usually can effortlessly find coherent regions even in noisy visual images, a task that is crucial for object recognition. Computer algorithms have been less successful at doing this in natural viewing conditions, in part because early work on the problem used only local computations on the image. Now a new approach has been developed, based on an image segmentation strategy that analyses all salient regions of an image and builds them into a hierarchical structure. This method is faster and more accurate than previous approaches, but the resulting algorithm is relatively simple to use. It is demonstrated in action by using it to find items within a large database of objects that match a target item.