GitHub - TencentARC/MotionCtrl at svd (original) (raw)

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

[SIGGRAPH 2024 CONFERENCE PROCEEDINGS]

Paper arXiv Project Page

🤗 HF Demo 🤗 HF Demo

More examples of MotionCtrl+SVD are in showcase_svd

motionctrl_small.mp4

🔥🔥 We release the codes, models and demos for MotionCtrl on Stable Video Diffusion (SVD).

Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.

Results of MotionCtrl+SVD

More results are in showcase_svd and our Project Page.

⚙️ Environment

conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt

💫 Inference

  1. Download the weights of MotionCtrl motionctrl_svd.pth and put it to ./checkpoints.
  2. Running scripts: sh configs/inference/run.sh

❗❗❗ Noted ❗❗❗

  1. If the motion control is not obvious, try to increase the speed in the run.sh or Motion Speed in the gradio demo.
  2. If the generated videos are distored severely, try to descrease the speed in the run.sh or Motion Speed in the gradio demo. Or increase FPS.

📚 Citation

If you make use of our work, please cite our paper.

@inproceedings{wang2024motionctrl, title={Motionctrl: A unified and flexible motion controller for video generation}, author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Li, Yaowei and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Ying}, booktitle={ACM SIGGRAPH 2024 Conference Papers}, pages={1--11}, year={2024} }

🤗 Acknowledgment

The current version of MotionCtrl is built on SVD. We appreciate the authors for sharing their awesome codebase.

❓ Contact

For any question, feel free to email wzhoux@connect.hku.hk or zhouzi1212@gmail.com.