GitHub - Vchitect/VBench: [CVPR2024 Highlight] VBench - We Evaluate Video Generation (original) (raw)

vbench_logo

How to Reach Us:

VBench Paper (CVPR 2024) VBench++ Report (arXiv) VBench-2.0 Report (arXiv) HuggingFace VBench Arena (View Generated Videos Here!) VBench-2.0 Arena (View Generated Videos Here!) Project Page Project Page Dataset Download PyPI Video Video Visitor

This repository contains the implementation of the following paper and its related serial works in progress. We evaluate video generative models!

VBench: Comprehensive Benchmark Suite for Video Generative Models
Ziqi Huang∗, Yinan He∗, Jiashuo Yu∗, Fan Zhang∗, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin+, Yu Qiao+, Ziwei Liu+
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Models
Ziqi Huang∗, Fan Zhang∗, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin+, Yu Qiao+, Ziwei Liu+

VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness
Dian Zheng∗, Ziqi Huang∗, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng+, Yu Qiao+, Ziwei Liu+

Table of Contents

🔥 Updates

📣 Overview

VBench-1.0

overall_structureWe propose VBench, a comprehensive benchmark suite for video generative models. We design a comprehensive and hierarchical Evaluation Dimension Suite to decompose "video generation quality" into multiple well-defined dimensions to facilitate fine-grained and objective evaluation. For each dimension and each content category, we carefully design a Prompt Suite as test cases, and sample Generated Videos from a set of video generation models. For each evaluation dimension, we specifically design an Evaluation Method Suite, which uses carefully crafted method or designated pipeline for automatic objective evaluation. We also conduct Human Preference Annotation for the generated videos for each dimension, and show that VBench evaluation results are well aligned with human perceptions. VBench can provide valuable insights from multiple perspectives. VBench++ supports a wide range of video generation tasks, including text-to-video and image-to-video, with an adaptive Image Suite for fair evaluation across different settings. It evaluates not only technical quality but also the trustworthiness of generative models, offering a comprehensive view of model performance. We continually incorporate more video generative models into VBench to inform the community about the evolving landscape of video generation.

VBench-2.0

overall_structureOverview of VBench-2.0. (a) Scope of VBench-2.0. Video generative models have progressed from achieving superficial faithfulness in fundamental technical aspects such as pixel fidelity and basic prompt adherence, to addressing more complex challenges associated with intrinsic faithfulness, including commonsense reasoning, physics-based realism, human motion, and creative composition. While VBench primarily assessed early-stage technical quality, VBench-2.0 expands the benchmarking framework to evaluate these advanced capabilities, ensuring a more comprehensive assessment of next-generation models. (b) Evaluation Dimension of VBench-2.0. VBench-2.0 introduces a structured evaluation suite comprising five broad categories and 18 fine-grained capability dimensions.

🎓 Evaluation Results

See our leaderboard for the most updated ranking and numerical results (with models like Gen-3, Kling, Pika). HuggingFace

We visualize the evaluation results of the 12 most recent top-performing long video generation models across 16 VBench dimensions.

Additionally, we present radar charts separately for the evaluation results of open-source and closed-source models. The results are normalized per dimension for clearer comparisons.

🏆 Leaderboard

See numeric values at our Leaderboard 🥇🥈🥉

📽️ Model Info

See model info for video generation models we used for evaluation.

🔨 Installation

Install with pip

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # or any other PyTorch version with CUDA<=12.1
pip install vbench

To evaluate some video generation ability aspects, you need to install detectron2 via:

pip install detectron2@git+https://github.com/facebookresearch/detectron2.git

If there is an error during detectron2 installation, see here. Detectron2 is working only with CUDA 12.1 or 11.X.

Download VBench_full_info.json to your running directory to read the benchmark prompt suites.

Install with git clone

git clone https://github.com/Vchitect/VBench.git
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu118 # or other version with CUDA<=12.1
pip install VBench

If there is an error during detectron2 installation, see here.

Usage

Use VBench to evaluate videos, and video generative models.

[New] Evaluate Your Own Videos

We support evaluating any video. Simply provide the path to the video file, or the path to the folder that contains your videos. There is no requirement on the videos' names.

To evaluate videos with customized input prompt, run our script with --mode=custom_input:

python evaluate.py \
    --dimension $DIMENSION \
    --videos_path /path/to/folder_or_video/ \
    --mode=custom_input

alternatively you can use our command:

vbench evaluate \
    --dimension $DIMENSION \
    --videos_path /path/to/folder_or_video/ \
    --mode=custom_input

To evaluate using multiple gpus, we can use the following commands:

torchrun --nproc_per_node=${GPUS} --standalone evaluate.py ...args...

or

vbench evaluate --ngpus=${GPUS} ...args...

Evaluation on the Standard Prompt Suite of VBench

Command Line

vbench evaluate --videos_path VIDEOPATH−−dimensionVIDEO_PATH --dimension VIDEOP​ATH−−dimensionDIMENSION

For example:

vbench evaluate --videos_path "sampled_videos/lavie/human_action" --dimension "human_action"

Python

from vbench import VBench my_VBench = VBench(device, <path/to/VBench_full_info.json>, <path/to/save/dir>) my_VBench.evaluate( videos_path = , name = , dimension_list = [, , ...], )

For example:

from vbench import VBench my_VBench = VBench(device, "vbench/VBench_full_info.json", "evaluation_results") my_VBench.evaluate( videos_path = "sampled_videos/lavie/human_action", name = "lavie_human_action", dimension_list = ["human_action"], )

Evaluation of Different Content Categories

command line

vbench evaluate
--videos_path $VIDEO_PATH
--dimension $DIMENSION
--mode=vbench_category
--category=$CATEGORY

or

python evaluate.py \
    --dimension $DIMENSION \
    --videos_path /path/to/folder_or_video/ \
    --mode=vbench_category

Example of Evaluating VideoCrafter-1.0

We have provided scripts to download VideoCrafter-1.0 samples, and the corresponding evaluation scripts.

# download sampled videos
sh scripts/download_videocrafter1.sh

# evaluate VideoCrafter-1.0
sh scripts/evaluate_videocrafter1.sh

Submit to Leaderboard

We have provided scripts for calculating the Total Score, Quality Score, and Semantic Score in the Leaderboard. You can run them locally to obtain the aggregate scores or as a final check before submitting to the Leaderboard.

Pack the evaluation results into a zip file.

cd evaluation_results zip -r ../evaluation_results.zip .

[Optional] get the total score of your submission file.

python scripts/cal_final_score.py --zip_file {path_to_evaluation_results.zip} --model_name {your_model_name}

You can submit the json file to HuggingFace

How to Calculate Total Score

To calculate the Total Score, we follow these steps:

  1. Normalization:
    Each dimension's results are normalized using the following formula:
    Normalized Score = (dim_score - min_val) / (max_val - min_val)
  2. Quality Score:
    The Quality Score is a weighted average of the following dimensions:
    subject consistency, background consistency, temporal flickering, motion smoothness, aesthetic quality, imaging quality, and dynamic degree.
  3. Semantic Score:
    The Semantic Score is a weighted average of the following dimensions:
    object class, multiple objects, human action, color, spatial relationship, scene, appearance style, temporal style, and overall consistency.
  4. Weighted Average Calculation:
    The Total Score is a weighted average of the Quality Score and Semantic Score:
    Total Score = w1 * Quality Score + w2 * Semantic Score

The minimum and maximum values used for normalization in each dimension, as well as the weighting coefficients for the average calculation, can be found in the scripts/constant.py file.

Total Score for VBench-I2V

For Total Score Calculation for VBench-I2V, you can refer to link.

💎 Pre-Trained Models

[Optional] Please download the pre-trained weights according to the guidance in the model_path.txt file for each model in the pretrained folder to ~/.cache/vbench.

📑 Prompt Suite

We provide prompt lists are at prompts/.

Check out details of prompt suites, and instructions for how to sample videos for evaluation.

📑 Sampled Videos

Dataset Download

To facilitate future research and to ensure full transparency, we release all the videos we sampled and used for VBench evaluation. You can download them on Google Drive.

See detailed explanations of the sampled videos here.

We also provide detailed setting for the models under evaluation here.

🏄 Evaluation Method Suite

To perform evaluation on one dimension, run this:

python evaluate.py --videos_path <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mi>I</mi><mi>D</mi><mi>E</mi><mi>O</mi><msub><mi>S</mi><mi>P</mi></msub><mi>A</mi><mi>T</mi><mi>H</mi><mo>−</mo><mo>−</mo><mi>d</mi><mi>i</mi><mi>m</mi><mi>e</mi><mi>n</mi><mi>s</mi><mi>i</mi><mi>o</mi><mi>n</mi></mrow><annotation encoding="application/x-tex">VIDEOS_PATH --dimension </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.22222em;">V</span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal" style="margin-right:0.02778em;">D</span><span class="mord mathnormal" style="margin-right:0.02778em;">EO</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em;">P</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.13889em;">T</span><span class="mord mathnormal" style="margin-right:0.08125em;">H</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.7778em;vertical-align:-0.0833em;"></span><span class="mord">−</span><span class="mord mathnormal">d</span><span class="mord mathnormal">im</span><span class="mord mathnormal">e</span><span class="mord mathnormal">n</span><span class="mord mathnormal">s</span><span class="mord mathnormal">i</span><span class="mord mathnormal">o</span><span class="mord mathnormal">n</span></span></span></span>DIMENSION

['subject_consistency', 'background_consistency', 'temporal_flickering', 'motion_smoothness', 'dynamic_degree', 'aesthetic_quality', 'imaging_quality', 'object_class', 'multiple_objects', 'human_action', 'color', 'spatial_relationship', 'scene', 'temporal_style', 'appearance_style', 'overall_consistency']

Alternatively, you can evaluate multiple models and multiple dimensions using this script:

vbench_videos/{model}/{dimension}/{prompt}-{index}.mp4/gif

Before evaluating the temporal flickering dimension, it is necessary to filter out the static videos first.

To filter static videos in the temporal flickering dimension, run this:

# This only filter out static videos whose prompt matches the prompt in the temporal_flickering.
python static_filter.py --videos_path $VIDEOS_PATH

You can adjust the filtering scope by:

# 1. Change the filtering scope to consider all files inside videos_path for filtering.
python static_filter.py --videos_path $VIDEOS_PATH --filter_scope all

# 2. Specify the path to a JSON file ($filename) to consider only videos whose prompts match those listed in $filename.
python static_filter.py --videos_path <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>V</mi><mi>I</mi><mi>D</mi><mi>E</mi><mi>O</mi><msub><mi>S</mi><mi>P</mi></msub><mi>A</mi><mi>T</mi><mi>H</mi><mo>−</mo><mo>−</mo><mi>f</mi><mi>i</mi><mi>l</mi><mi>t</mi><mi>e</mi><msub><mi>r</mi><mi>s</mi></msub><mi>c</mi><mi>o</mi><mi>p</mi><mi>e</mi></mrow><annotation encoding="application/x-tex">VIDEOS_PATH --filter_scope </annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:0.8333em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.22222em;">V</span><span class="mord mathnormal" style="margin-right:0.07847em;">I</span><span class="mord mathnormal" style="margin-right:0.02778em;">D</span><span class="mord mathnormal" style="margin-right:0.02778em;">EO</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.05764em;">S</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.3283em;"><span style="top:-2.55em;margin-left:-0.0576em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.13889em;">P</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">A</span><span class="mord mathnormal" style="margin-right:0.13889em;">T</span><span class="mord mathnormal" style="margin-right:0.08125em;">H</span><span class="mspace" style="margin-right:0.2222em;"></span><span class="mbin">−</span><span class="mspace" style="margin-right:0.2222em;"></span></span><span class="base"><span class="strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord">−</span><span class="mord mathnormal" style="margin-right:0.10764em;">f</span><span class="mord mathnormal">i</span><span class="mord mathnormal">lt</span><span class="mord mathnormal">e</span><span class="mord"><span class="mord mathnormal" style="margin-right:0.02778em;">r</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:-0.0278em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">s</span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">co</span><span class="mord mathnormal">p</span><span class="mord mathnormal">e</span></span></span></span>filename

✒️ Citation

If you find our repo useful for your research, please consider citing our paper:

@InProceedings{huang2023vbench, title={{VBench}: Comprehensive Benchmark Suite for Video Generative Models}, author={Huang, Ziqi and He, Yinan and Yu, Jiashuo and Zhang, Fan and Si, Chenyang and Jiang, Yuming and Zhang, Yuanhan and Wu, Tianxing and Jin, Qingyang and Chanpaisit, Nattapol and Wang, Yaohui and Chen, Xinyuan and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, year={2024} }

@article{huang2024vbench++, title={{VBench++}: Comprehensive and Versatile Benchmark Suite for Video Generative Models}, author={Huang, Ziqi and Zhang, Fan and Xu, Xiaojie and He, Yinan and Yu, Jiashuo and Dong, Ziyue and Ma, Qianli and Chanpaisit, Nattapol and Si, Chenyang and Jiang, Yuming and Wang, Yaohui and Chen, Xinyuan and Chen, Ying-Cong and Wang, Limin and Lin, Dahua and Qiao, Yu and Liu, Ziwei}, journal={arXiv preprint arXiv:2411.13503}, year={2024} }

@article{zheng2025vbench2, title={{VBench-2.0}: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness}, author={Zheng, Dian and Huang, Ziqi and Liu, Hongbo and Zou, Kai and He, Yinan and Zhang, Fan and Zhang, Yuanhan and He, Jingwen and Zheng, Wei-Shi and Qiao, Yu and Liu, Ziwei}, journal={arXiv preprint arXiv:2503.21755}, year={2025} }

♥️ Acknowledgement

💪 VBench Contributors

Order is based on the time joining the project:

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Nattapol Chanpaisit, Xiaojie Xu, Qianli Ma, Ziyue Dong, Dian Zheng, Hongbo Liu, Kai Zou

🤗 Open-Sourced Repositories

This project wouldn't be possible without the following open-sourced repositories:AMT, UMT, RAM, CLIP, RAFT, GRiT, IQA-PyTorch, ViCLIP, and LAION Aesthetic Predictor.

We are putting together Awesome-Evaluation-of-Visual-Generation, which collects works for evaluating visual generation.

Our related projects: Evaluation Agent

@article{zhang2024evaluationagent, title = {Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models}, author = {Zhang, Fan and Tian, Shulin and Huang, Ziqi and Qiao, Yu and Liu, Ziwei}, journal={arXiv preprint arXiv:2412.09645}, year = {2024} }