GitHub - aigc-apps/VideoX-Fun: 📹 A more flexible framework that can generate videos at any resolution and creates videos from images. (original) (raw)

VideoX-Fun

😊 Welcome!

CogVideoX-Fun:

Wan-Fun:

Table of Contents
Introduction
Quick Start
Video Result
How to use
Model zoo
Reference
License

Introduction

VideoX-Fun is a video generation pipeline that can be used to generate AI images and videos, as well as to train baseline and Lora models for Diffusion Transformer. We support direct prediction from pre-trained baseline models to generate videos with different resolutions, durations, and FPS. Additionally, we also support users in training their own baseline and Lora models to perform specific style transformations.

We will support quick pull-ups from different platforms, refer to Quick Start.

What's New:

Update Wan2.1-Fun-V1.1: Support for 14B and 1.3B model Control + Reference Image models, support for camera control, and the Inpaint model has been retrained for improved performance. [2025.04.25]
Update Wan2.1-Fun-V1.0: Support I2V and Control models for 14B and 1.3B models, with support for start and end frame prediction. [2025.03.26]
Update CogVideoX-Fun-V1.5: Upload I2V model and related training/prediction code. [2024.12.16]
Reward Lora Support: Train Lora using reward backpropagation techniques to optimize generated videos, making them better aligned with human preferences. More Information. New version of the control model supports various control conditions such as Canny, Depth, Pose, MLSD, etc. [2024.11.21]
Diffusers Support: CogVideoX-Fun Control is now supported in diffusers. Thanks to a-r-r-o-w for contributing support in this PR. Check out the documentation for more details. [2024.10.16]
Update CogVideoX-Fun-V1.1: Retrain i2v model, add Noise to increase the motion amplitude of the video. Upload control model training code and Control model. [2024.09.29]
Update CogVideoX-Fun-V1.0: Initial code release! Now supports Windows and Linux. Supports video generation at arbitrary resolutions from 256x256x49 to 1024x1024x49 for 2B and 5B models. [2024.09.18]

Function：

Data Preprocessing
Train DiT
Video Generation

Our UI interface is as follows:

Quick Start

1. Cloud usage: AliyunDSW/Docker

a. From AliyunDSW

DSW has free GPU time, which can be applied once by a user and is valid for 3 months after applying.

Aliyun provide free GPU time in Freetier, get it and use in Aliyun PAI-DSW to start CogVideoX-Fun within 5min!

b. From ComfyUI

Our ComfyUI is as follows, please refer to ComfyUI README for details.

c. From docker

If you are using docker, please make sure that the graphics card driver and CUDA environment have been installed correctly in your machine.

Then execute the following commands in this way:

# pull image
docker pull mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun

# enter image
docker run -it -p 7860:7860 --network host --gpus all --security-opt seccomp:unconfined --shm-size 200g mybigpai-public-registry.cn-beijing.cr.aliyuncs.com/easycv/torch_cuda:cogvideox_fun

# clone code
git clone https://github.com/aigc-apps/VideoX-Fun.git

# enter VideoX-Fun's dir
cd VideoX-Fun

# download weights
mkdir models/Diffusion_Transformer
mkdir models/Personalized_Model

# Please use the hugginface link or modelscope link to download the model.
# CogVideoX-Fun
# https://huggingface.co/alibaba-pai/CogVideoX-Fun-V1.1-5b-InP
# https://modelscope.cn/models/PAI/CogVideoX-Fun-V1.1-5b-InP

# Wan
# https://huggingface.co/alibaba-pai/Wan2.1-Fun-V1.1-14B-InP
# https://modelscope.cn/models/PAI/Wan2.1-Fun-V1.1-14B-InP

2. Local install: Environment Check/Downloading/Installation

a. Environment Check

We have verified this repo execution on the following environment:

The detailed of Windows:

OS: Windows 10
python: python3.10 & python3.11
pytorch: torch2.2.0
CUDA: 11.8 & 12.1
CUDNN: 8+
GPU： Nvidia-3060 12G & Nvidia-3090 24G

The detailed of Linux:

OS: Ubuntu 20.04, CentOS
python: python3.10 & python3.11
pytorch: torch2.2.0
CUDA: 11.8 & 12.1
CUDNN: 8+
GPU：Nvidia-V100 16G & Nvidia-A10 24G & Nvidia-A100 40G & Nvidia-A100 80G

We need about 60GB available on disk (for saving weights), please check!

b. Weights

We'd better place the weights along the specified path:

Via ComfyUI: Put the models into the ComfyUI weights folder ComfyUI/models/Fun_Models/:

📦 ComfyUI/
├── 📂 models/
│   └── 📂 Fun_Models/
│       ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
│       ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
│       ├── 📂 Wan2.1-Fun-14B-InP
│       └── 📂 Wan2.1-Fun-1.3B-InP/

Run its own python file or UI interface:

📦 models/
├── 📂 Diffusion_Transformer/
│   ├── 📂 CogVideoX-Fun-V1.1-2b-InP/
│   ├── 📂 CogVideoX-Fun-V1.1-5b-InP/
│   ├── 📂 Wan2.1-Fun-14B-InP
│   └── 📂 Wan2.1-Fun-1.3B-InP/
├── 📂 Personalized_Model/
│   └── your trained trainformer model / your trained lora model (for UI load)

Video Result

Wan2.1-Fun-V1.1-14B-InP && Wan2.1-Fun-V1.1-1.3B-InP

inp_1.mp4	inp_2.mp4	inp_3.mp4	inp_4.mp4

inp_5.mp4	inp_6.mp4	inp_7.mp4	inp_8.mp4

Wan2.1-Fun-V1.1-14B-Control && Wan2.1-Fun-V1.1-1.3B-Control

Generic Control Video + Reference Image:

Reference Image	Control Video	Wan2.1-Fun-V1.1-14B-Control	Wan2.1-Fun-V1.1-1.3B-Control
	pose_control.mp4	14b_ref.mp4	1.3b_ref.mp4

Generic Control Video (Canny, Pose, Depth, etc.) and Trajectory Control:

Fun-Trajectory_00003.mp4	Fun-Trajectory-Merge_00003.mp4	Fun_00006.mp4

pose.mp4	canny.mp4	depth.mp4
pose_out.mp4	canny_out.mp4	depth_out.mp4

Wan2.1-Fun-V1.1-14B-Control-Camera && Wan2.1-Fun-V1.1-1.3B-Control-Camera

Pan Up	Pan Left	Pan Right
Pan_Up.mp4	Pan_Left.mp4	Pan_Right.mp4
Pan Down	Pan Up + Pan Left	Pan Up + Pan Right
Pan_Down.mp4	Pan_Left_Up.mp4	Pan_Right_Up.mp4

CogVideoX-Fun-V1.1-5B

Resolution-1024

00000005.mp4	00000006.mp4	00000009.mp4	00000010.mp4

Resolution-768

00000001.mp4	00000002.mp4	00000005.mp4	00000006.mp4

Resolution-512

00000036.mp4	00000035.mp4	00000034.mp4	00000033.mp4

CogVideoX-Fun-V1.1-5B-Control

demo_pose.mp4	demo_scribble.mp4	demo_depth.mp4
A young woman with beautiful clear eyes and blonde hair, wearing white clothes and twisting her body, with the camera focused on her face. High quality, masterpiece, best quality, high resolution, ultra-fine, dreamlike.	A young woman with beautiful clear eyes and blonde hair, wearing white clothes and twisting her body, with the camera focused on her face. High quality, masterpiece, best quality, high resolution, ultra-fine, dreamlike.	A young bear.
00000010.mp4	00000011.mp4	00000012.mp4

How to Use

1. Generation

a. GPU Memory Optimization

Since Wan2.1 has a very large number of parameters, we need to consider memory optimization strategies to adapt to consumer-grade GPUs. We provide GPU_memory_mode for each prediction file, allowing you to choose between model_cpu_offload, model_cpu_offload_and_qfloat8, and sequential_cpu_offload. This solution is also applicable to CogVideoX-Fun generation.

model_cpu_offload: The entire model is moved to the CPU after use, saving some GPU memory.
model_cpu_offload_and_qfloat8: The entire model is moved to the CPU after use, and the transformer model is quantized to float8, saving more GPU memory.
sequential_cpu_offload: Each layer of the model is moved to the CPU after use. It is slower but saves a significant amount of GPU memory.

qfloat8 may slightly reduce model performance but saves more GPU memory. If you have sufficient GPU memory, it is recommended to use model_cpu_offload.

b. Using ComfyUI

For details, refer to ComfyUI README.

c. Running Python Files

i. Single-GPU Inference:

Step 1: Download the corresponding weights and place them in the models folder.
Step 2: Use different files for prediction based on the weights and prediction goals. This library currently supports CogVideoX-Fun, Wan2.1, and Wan2.1-Fun. Different models are distinguished by folder names under the examples folder, and their supported features vary. Use them accordingly. Below is an example using CogVideoX-Fun:
- Text-to-Video:
  * Modify prompt, neg_prompt, guidance_scale, and seed in the file examples/cogvideox_fun/predict_t2v.py.
  * Run the file examples/cogvideox_fun/predict_t2v.py and wait for the results. The generated videos will be saved in the folder samples/cogvideox-fun-videos.
- Image-to-Video:
  * Modify validation_image_start, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the file examples/cogvideox_fun/predict_i2v.py.
  * validation_image_start is the starting image of the video, and validation_image_end is the ending image of the video.
  * Run the file examples/cogvideox_fun/predict_i2v.py and wait for the results. The generated videos will be saved in the folder samples/cogvideox-fun-videos_i2v.
- Video-to-Video:
  * Modify validation_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the file examples/cogvideox_fun/predict_v2v.py.
  * validation_video is the reference video for video-to-video generation. You can use the following demo video: Demo Video.
  * Run the file examples/cogvideox_fun/predict_v2v.py and wait for the results. The generated videos will be saved in the folder samples/cogvideox-fun-videos_v2v.
- Controlled Video Generation (Canny, Pose, Depth, etc.):
  * Modify control_video, validation_image_end, prompt, neg_prompt, guidance_scale, and seed in the file examples/cogvideox_fun/predict_v2v_control.py.
  * control_video is the control video extracted using operators such as Canny, Pose, or Depth. You can use the following demo video: Demo Video.
  * Run the file examples/cogvideox_fun/predict_v2v_control.py and wait for the results. The generated videos will be saved in the folder samples/cogvideox-fun-videos_v2v_control.
Step 3: If you want to integrate other backbones or Loras trained by yourself, modify lora_path and relevant paths in examples/{model_name}/predict_t2v.py or examples/{model_name}/predict_i2v.py as needed.

ii. Multi-GPU Inference:

When using multi-GPU inference, please make sure to install the xfuser. We recommend installing xfuser==0.4.2 and yunchang==0.6.2.

pip install xfuser==0.4.2 --progress-bar off -i https://mirrors.aliyun.com/pypi/simple/
pip install yunchang==0.6.2 --progress-bar off -i https://mirrors.aliyun.com/pypi/simple/

Please ensure that the product of ulysses_degree and ring_degree equals the number of GPUs being used. For example, if you are using 8 GPUs, you can set ulysses_degree=2 and ring_degree=4, or alternatively ulysses_degree=4 and ring_degree=2.

ulysses_degree performs parallelization after splitting across the heads.
ring_degree performs parallelization after splitting across the sequence.

Compared to ulysses_degree, ring_degree incurs higher communication costs. Therefore, when setting these parameters, you should take into account both the sequence length and the number of heads in the model.

Let’s take 8-GPU parallel inference as an example:

For Wan2.1-Fun-V1.1-14B-InP, which has 40 heads, ulysses_degree should be set to a divisor of 40 (e.g., 2, 4, 8, etc.). Thus, when using 8 GPUs for parallel inference, you can set ulysses_degree=8 and ring_degree=1.
For Wan2.1-Fun-V1.1-1.3B-InP, which has 12 heads, ulysses_degree should be set to a divisor of 12 (e.g., 2, 4, etc.). Thus, when using 8 GPUs for parallel inference, you can set ulysses_degree=4 and ring_degree=2.

After setting the parameters, run the following command for parallel inference:

torchrun --nproc-per-node=8 examples/wan2.1_fun/predict_t2v.py

d. Using the Web UI

The web UI supports text-to-video, image-to-video, video-to-video, and controlled video generation (Canny, Pose, Depth, etc.). This library currently supports CogVideoX-Fun, Wan2.1, and Wan2.1-Fun. Different models are distinguished by folder names under the examples folder, and their supported features vary. Use them accordingly. Below is an example using CogVideoX-Fun:

Step 1: Download the corresponding weights and place them in the models folder.
Step 2: Run the file examples/cogvideox_fun/app.py to access the Gradio interface.
Step 3: Select the generation model on the page, fill in prompt, neg_prompt, guidance_scale, and seed, click "Generate," and wait for the results. The generated videos will be saved in the sample folder.

2. Model Training

A complete model training pipeline should include data preprocessing and Video DiT training. The training process for different models is similar, and the data formats are also similar:

a. data preprocessing

We have provided a simple demo of training the Lora model through image data, which can be found in the wiki for details.

A complete data preprocessing link for long video segmentation, cleaning, and description can refer to README in the video captions section.

If you want to train a text to image and video generation model. You need to arrange the dataset in this format.

📦 project/
├── 📂 datasets/
│   ├── 📂 internal_datasets/
│       ├── 📂 train/
│       │   ├── 📄 00000001.mp4
│       │   ├── 📄 00000002.jpg
│       │   └── 📄 .....
│       └── 📄 json_of_internal_datasets.json

The json_of_internal_datasets.json is a standard JSON file. The file_path in the json can to be set as relative path, as shown in below:

[ { "file_path": "train/00000001.mp4", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "video" }, { "file_path": "train/00000002.jpg", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "image" }, ..... ]

You can also set the path as absolute path as follow:

[ { "file_path": "/mnt/data/videos/00000001.mp4", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "video" }, { "file_path": "/mnt/data/train/00000001.jpg", "text": "A group of young men in suits and sunglasses are walking down a city street.", "type": "image" }, ..... ]

b. Video DiT training

If the data format is relative path during data preprocessing, please set scripts/{model_name}/train.sh as follow.

export DATASET_NAME="datasets/internal_datasets/"
export DATASET_META_NAME="datasets/internal_datasets/json_of_internal_datasets.json"

If the data format is absolute path during data preprocessing, please set scripts/train.sh as follow.

export DATASET_NAME=""
export DATASET_META_NAME="/mnt/data/json_of_internal_datasets.json"

Then, we run scripts/train.sh.

For details on some parameter settings: Wan2.1-Fun can be found in Readme Train and Readme Lora. Wan2.1 can be found in Readme Train and Readme Lora. CogVideoX-Fun can be found in Readme Train and Readme Lora.

Model zoo

1. Wan2.1-Fun

V1.1:

Name	Storage Size	Hugging Face	Model Scope	Description
Wan2.1-Fun-V1.1-1.3B-InP	19.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-1.3B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction.
Wan2.1-Fun-V1.1-14B-InP	47.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-14B text-to-video generation weights, trained at multiple resolutions, supports start-end image prediction.
Wan2.1-Fun-V1.1-1.3B-Control	19.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-1.3B video control weights support various control conditions such as Canny, Depth, Pose, MLSD, etc., supports reference image + control condition-based control, and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction.
Wan2.1-Fun-V1.1-14B-Control	47.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-14B video control weights support various control conditions such as Canny, Depth, Pose, MLSD, etc., supports reference image + control condition-based control, and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction.
Wan2.1-Fun-V1.1-1.3B-Control-Camera	19.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-1.3B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction.
Wan2.1-Fun-V1.1-14B-Control-Camera	47.0 GB	🤗Link	😄Link	Wan2.1-Fun-V1.1-14B camera lens control weights. Supports multi-resolution (512, 768, 1024) video prediction, trained with 81 frames at 16 FPS, supports multilingual prediction.

V1.0:

Name	Storage Space	Hugging Face	Model Scope	Description
Wan2.1-Fun-1.3B-InP	19.0 GB	🤗Link	😄Link	Wan2.1-Fun-1.3B text-to-video weights, trained at multiple resolutions, supporting start and end frame prediction.
Wan2.1-Fun-14B-InP	47.0 GB	🤗Link	😄Link	Wan2.1-Fun-14B text-to-video weights, trained at multiple resolutions, supporting start and end frame prediction.
Wan2.1-Fun-1.3B-Control	19.0 GB	🤗Link	😄Link	Wan2.1-Fun-1.3B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support.
Wan2.1-Fun-14B-Control	47.0 GB	🤗Link	😄Link	Wan2.1-Fun-14B video control weights, supporting various control conditions such as Canny, Depth, Pose, MLSD, etc., and trajectory control. Supports multi-resolution (512, 768, 1024) video prediction at 81 frames, trained at 16 frames per second, with multilingual prediction support.

2. Wan2.1

Name	Hugging Face	Model Scope	Description
Wan2.1-T2V-1.3B	🤗Link	😄Link	Wanxiang 2.1-1.3B text-to-video weights
Wan2.1-T2V-14B	🤗Link	😄Link	Wanxiang 2.1-14B text-to-video weights
Wan2.1-I2V-14B-480P	🤗Link	😄Link	Wanxiang 2.1-14B-480P image-to-video weights
Wan2.1-I2V-14B-720P	🤗Link	😄Link	Wanxiang 2.1-14B-720P image-to-video weights

3. CogVideoX-Fun

V1.5:

Name	Storage Space	Hugging Face	Model Scope	Description
CogVideoX-Fun-V1.5-5b-InP	20.0 GB	🤗Link	😄Link	Our official graph-generated video model is capable of predicting videos at multiple resolutions (512, 768, 1024) and has been trained on 85 frames at a rate of 8 frames per second.
CogVideoX-Fun-V1.5-Reward-LoRAs	-	🤗Link	😄Link	The official reward backpropagation technology model optimizes the videos generated by CogVideoX-Fun-V1.5 to better match human preferences. ｜