State of the Art on Diffusion Models for Visual Computing (original) (raw)

Wang Yifan1*, Vladislav Golyanik2*, Kfir Aberman3, Jon Barron4, Amit H. Bermano5, Eric R. Chan1, Tali Dekel6, Aleksander Holynski4,7, Angjoo Kanazawa7, C. Karen Liu1, Lingjie Liu8, Ben Mildenhall4, Matthias Niessner9, Björn Ommer10, Christian Theobalt2, Peter Wonka11, Gordon Wetzstein1

1Stanford University, 2MPI for Informatics and VIA Center, 3Snap Inc., 4Google Research, 5Tel Aviv University, 6Weizmann Institute of Science, 7UC Berkley, 8University of Pennsylvania, 9TU Munich, 10LMU Munich, 11KAUST, *Equal contribution

Abstract

The field of visual computing is rapidly advancing due to the emergence of generative artificial intelligence (AI), which unlocks unprecedented capabilities for the generation, editing, and reconstruction of images, videos, and 3D scenes. In these domains, diffusion models are the generative AI architecture of choice. Within the last year alone, the literature on diffusion-based tools and applications has seen exponential growth and relevant papers are published across the computer graphics, computer vision, and AI communities with new works appearing daily on arXiv. This rapid growth of the field makes it difficult to keep up with all recent developments. The goal of this state-of-the-art report (STAR) is to introduce the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model, as well as overview important aspects of these generative AI tools, including personalization, conditioning, inversion, among others. Moreover, we give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing, categorized by the type of generated medium, including 2D images, videos, 3D objects, locomotion, and 4D scenes. Finally, we discuss available datasets, metrics, open challenges, and social implications. This STAR provides an intuitive starting point to explore this exciting topic for researchers, artists, and practitioners alike.

This state-of-the-art report discusses the theory and practice of diffusion models for visual computing. These models have recently become the de-facto standard for image, video, 3D, and 4D generation and editing.

BibTeX

@inproceedings{po2024state,
      title={State of the art on diffusion models for visual computing},
      author={Po, Ryan and Yifan, Wang and Golyanik, Vladislav and Aberman, Kfir and Barron, Jonathan T and Bermano, Amit and Chan, Eric and Dekel, Tali and Holynski, Aleksander and Kanazawa, Angjoo and others},
      booktitle={Computer Graphics Forum},
      volume={43},
      number={2},
      pages={e15063},
    year={2024},     
    }