GitHub - OpenGVLab/VideoChat-Flash: VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling (original) (raw)
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, and Limin Wang
π€ Model & Data ο½ π₯οΈ Demo | π Paper | π Blog
π₯ Updates
- 2025/05/10:π₯π₯π₯ We release most video of our training data, Hope it can be of help to you!
- 2025/03/27:π₯π₯ We release our dataset and evaluation codes for single-hop and multi-hop needle-in-a-haystack!
- 2025/03/09:π₯π₯ We release our weights of each training stage here, try to build your VideoChat-Flash on them!
- 2025/02/25:π₯π₯ We release our training data, training codes based LLaVA for VideoChat-Flash and training codes based XTuner for finetuning InternVideo2.5.
- 2025/02/12: πππOur VideoChat-Flash-7B@448 has achieved first place on the latest Video Detail Caption Benchmark, AuroraCap.
- 2025/01/15: We provide evaluation codes for QA & Grounding Benchmark.
- 2025/01/12: π₯π₯π₯Release VideoChat2-Flash, a powerfull MLLM built on video encoder (InternVideo) and LLM (Qwen).
- We offer five models, VideoChat2-Flash-2B@224 (Small LLM), VideoChat2-Flash-7B@224, VideoChat2-Flash-7B@448 (Overall best), VideoChat-Flash-Qwen2_5-7B-1M (Super long video input) and VideoChat-Flash-Qwen2_5-7B_InternVideo2-1B (Stronger short-term temporal understanding).
π Future Plan
- lmdeploy/vllm support for videochat-flash and InternVideo2.5
- LoRA finetuning training code for videochat-flash and InternVideo2.5
- Mixing image/video training code for InternVideo2.5
- Faster training code with XTuner for VideoChat-Flash
As I am currently very busy with work and find it difficult to complete the above plans quickly, I sincerely ask friends in the community to join in and submit a PR.
π¦ Introduction
πState-of-the-art performance in short and long video understanding, with temporal localization capabilities comparable to expert models.
πSupports ultra-long video inputs, achieving a groundbreaking needle-in-a-haystack evaluation accuracy of 99.1% on 10,000 frames, capable of processing videos up to three hours long.
β‘Highly efficient model architecture with exceptional inference speed, encoding each video frame into just 16 tokens, making it 5β10 times faster than the previous model.
Demo & Inference
Refer to hf README to inference our model.
Evaluation
See evaluation codes. And lmms-eval have supported our model, you also could use it to evaluate our model on varous benchmarks.
Training
See training codes based LLaVA for VideoChat-Flash and training codes based XTuner for finetuning InternVideo2.5.
π NIAH
See xtuner-eval_niah for evaluation of Single-Hop NIAH-Video and Multi-Hop NIAH-Video.
π Citation
If you find this project useful in your research, please consider cite:
@article{li2024videochat, title={VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling}, author={Li, Xinhao and Wang, Yi and Yu, Jiashuo and Zeng, Xiangyu and Zhu, Yuhan and Huang, Haian and Gao, Jianfei and Li, Kunchang and He, Yinan and Wang, Chenting and Qiao, Yu and Wang, Yali and Wang, Limin}, journal={arXiv preprint arXiv:2501.00574}, year={2024} }
π« Acknowledgement
Thanks to the open source of the following projects: InternVideo, UMT, Qwen, LLaVA-VL, lmms-eval, Ask-Anything, ToMe, LongVLM, FastV, LLaVolta, PyramidDrop, LongVA, their implementation provides valuable reference experience for our project.