GitHub - maxin-cn/Cinemo: [CVPR 2025] Consistent and Controllable Image Animation with Motion Diffusion Models (original) (raw)
Consistent and Controllable Image Animation with Motion Diffusion Models
Official PyTorch Implementation
Consistent and Controllable Image Animation with Motion Diffusion Models
Xin Ma, Yaohui Wang*†, Gengyun Jia, Xinyuan Chen, Tien-Tsin Wong, Yuan-Fang Li, Cunjian Chen*
(*Corresponding authors, †Project Lead)
This repo contains pre-trained weights, and sampling code of Cinemo. Please visit our project page for more results.
News
- 🔥 Feb. 27, 2025 💥 Our paper is accepted by CVPR 2025.
- 🔥 Jul. 29, 2024 💥 HuggingFace space is added, you can also launch gradio interface locally.
- 🔥 Jul. 23, 2024 💥 Our paper is released on arxiv.
- 🔥 Jun. 2, 2024 💥 The inference code is released. The checkpoint can be found here.
Setup
Download and set up the repo:
git clone https://github.com/maxin-cn/Cinemo cd Cinemo conda env create -f environment.yml conda activate cinemo
Animation
You can sample from our pre-trained Cinemo models with animation.py. Weights for our pre-trained Cinemo model can be found here. The script has various arguments for adjusting sampling steps, changing the classifier-free guidance scale, etc:
bash pipelines/animation.sh
Related model weights will be downloaded automatically, and the following results can be obtained,
| Input image | Output video | Input image | Output video |
|---|---|---|---|
 |
 |
 |
 |
| "People Walking" | "Sea Swell" | ||
 |
 |
 |
 |
| "Girl Dancing under the Stars" | "Dragon Glowing Eyes" | ||
 |
 |
 |
 |
| "Bubbles Floating upwards" | "Snowman Waving his Hand" |
Gradio interface
We also provide a local gradio interface, just run:
You can specify the --share and --server_name arguments to meet your requirement!
Other Applications
You can also utilize Cinemo for other applications, such as motion transfer and video editing:
bash pipelines/video_editing.sh
Related checkpoints will be downloaded automatically and following results will be obtained,
or motion transfer,
Contact Us
Xin Ma: xin.ma1@monash.edu, Yaohui Wang: wangyaohui@pjlab.org.cn
Citation
If you find this work useful for your research, please consider citing it.
@article{ma2024cinemo, title={Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models}, author={Ma, Xin and Wang, Yaohui and Jia, Gengyun and Chen, Xinyuan and Wong, Tien-Tsin and Li, Yuan-Fang and Chen, Cunjian}, journal={arXiv preprint arXiv:2407.15642}, year={2024} }
Acknowledgments
Cinemo has been greatly inspired by the following amazing works and teams: LaVie and SEINE, we thank all the contributors for open-sourcing.
License
The code and model weights are licensed under LICENSE.