GitHub - OpenGVLab/InternVL: [CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型 (original) (raw)

News 🚀🚀🚀

Documents

🌟 Get Started

🏆 InternVL Family

Model Zoo

Multimodal Large Language Model (InternVL 2.5)

Model Name Vision Part Language Part HF Link MS Link
InternVL2_5-1B InternViT‑300M‑448px‑V2_5 Qwen2.5‑0.5B‑Instruct 🤗 link 🤖 link
InternVL2_5-2B InternViT-300M-448px-V2_5 internlm2_5-1_8b-chat 🤗 link 🤖 link
InternVL2_5-4B InternViT-300M-448px-V2_5 Qwen2.5-3B-Instruct 🤗 link 🤖 link
InternVL2_5-8B InternViT-300M-448px-V2_5 internlm2_5-7b-chat 🤗 link 🤖 link
InternVL2_5-26B InternViT-6B-448px-V2_5 internlm2_5-20b-chat 🤗 link 🤖 link
InternVL2_5-38B InternViT-6B-448px-V2_5 Qwen2.5-32B-Instruct 🤗 link 🤖 link
InternVL2_5-78B InternViT-6B-448px-V2_5 Qwen2.5-72B-Instruct 🤗 link 🤖 link
Model Name Vision Part Language Part HF Link MS Link
InternVL2_5-1B-MPO InternViT‑300M‑448px‑V2_5 Qwen2.5‑0.5B‑Instruct 🤗 link 🤖 link
InternVL2_5-2B-MPO InternViT-300M-448px-V2_5 internlm2_5-1_8b-chat 🤗 link 🤖 link
InternVL2_5-4B-MPO InternViT-300M-448px-V2_5 Qwen2.5-3B-Instruct 🤗 link 🤖 link
InternVL2_5-8B-MPO InternViT-300M-448px-V2_5 internlm2_5-7b-chat 🤗 link 🤖 link
InternVL2_5-26B-MPO InternViT-6B-448px-V2_5 internlm2_5-20b-chat 🤗 link 🤖 link
InternVL2_5-38B-MPO InternViT-6B-448px-V2_5 Qwen2.5-32B-Instruct 🤗 link 🤖 link
InternVL2_5-78B-MPO InternViT-6B-448px-V2_5 Qwen2.5-72B-Instruct 🤗 link 🤖 link

Multimodal Large Language Model (InternVL 2.0)

Model Name Vision Part Language Part HF Link MS Link
InternVL2-1B InternViT-300M-448px Qwen2-0.5B-Instruct 🤗 link 🤖 link
InternVL2-2B InternViT-300M-448px internlm2-chat-1-8b 🤗 link 🤖 link
InternVL2-4B InternViT-300M-448px Phi‑3‑mini‑128k‑instruct 🤗 link 🤖 link
InternVL2-8B InternViT-300M-448px internlm2_5-7b-chat 🤗 link 🤖 link
InternVL2-26B InternViT-6B-448px-V1-5 internlm2-chat-20b 🤗 link 🤖 link
InternVL2-40B InternViT‑6B‑448px‑V1‑5 Nous‑Hermes‑2‑Yi‑34B 🤗 link 🤖 link
InternVL2‑Llama3-76B InternViT-6B-448px-V1-5 Hermes‑2‑Theta‑ Llama‑3‑70B 🤗 link 🤖 link

Multimodal Large Language Model (InternVL 1.0-1.5)

Model Date HF Link MS Link Note
Mini‑InternVL‑Chat‑4B‑V1‑5 2024.05.28 🤗 link 🤖 link 🚀🚀 16% of the model size, 90% of the performance
Mini-InternVL-Chat-2B-V1-5 2024.05.19 🤗 link 🤖 link 🚀 8% of the model size, 80% of the performance
InternVL-Chat-V1-5 2024.04.18 🤗 link 🤖 link support 4K image; super strong OCR; Approaching the performance of GPT-4V and Gemini Pro on various benchmarks like MMMU, DocVQA, ChartQA, MathVista, etc.
InternVL-Chat-V1-2-Plus 2024.02.21 🤗 link 🤖 link more SFT data and stronger
InternVL-Chat-V1-2 2024.02.11 🤗 link 🤖 link scaling up LLM to 34B
InternVL-Chat-V1-1 2024.01.24 🤗 link 🤖 link support Chinese and stronger OCR
InternVL-Chat-19B 2023.12.25 🤗 link 🤖 link English multimodal dialogue
InternVL-Chat-13B 2023.12.25 🤗 link 🤖 link English multimodal dialogue

CLIP-like Model (InternVL 1.0-2.5)

Model Date HF Link MS Link Note
InternViT-300M-448px-V2_5 2024.12.05 🤗 link 🤖 link 🚀🚀 A more powerful lightweight visual encoder. (🔥new)
InternViT-6B-448px-V2_5 2024.12.05 🤗 link 🤖 link 🚀🚀 A stronger visual encoder to extract visual features. (🔥new)
InternViT-300M-448px 2024.05.25 🤗 link 🤖 link distilled small vision foundation model with 300M parameters
InternViT‑6B‑448px‑V1‑5 2024.04.20 🤗 link 🤖 link support dynamic resolution and super strong OCR feature extraction capability by incremental pre-training
InternViT-6B-448px-V1-2 2024.02.11 🤗 link 🤖 link support 448 resolution by incremental pre-training
InternViT-6B-448px-V1-0 2024.01.30 🤗 link 🤖 link support 448 resolution by incremental pre-training
InternViT-6B-224px 2023.12.22 🤗 link 🤖 link the first version of InternViT-6B, extracted from InternVL‑14B‑224px

Vision-Language Foundation Model (InternVL 1.0)

Model Date HF Link MS Link Note
InternVL‑14B‑224px 2023.12.22 🤗 link 🤖 link vision-language foundation model, InternViT-6B + QLLaMA, can be used for image-text retrieval like CLIP

TODO List

What can InternVL do?

Visual Perception (click to expand)

Quick Start with HuggingFace

using InternViT-6B for visual feature extraction (click to expand)

import torch from PIL import Image from transformers import AutoModel, CLIPImageProcessor

model = AutoModel.from_pretrained( 'OpenGVLab/InternViT-6B-448px-V2_5', torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True).cuda().eval()

image = Image.open('./examples/image1.jpg').convert('RGB')

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values pixel_values = pixel_values.to(torch.bfloat16).cuda()

outputs = model(pixel_values)

using InternVL-C(ontrastive) and InternVL-G(enerative) for cross-modal retrieval (click to expand)

import torch from PIL import Image from transformers import AutoModel, CLIPImageProcessor from transformers import AutoTokenizer

model = AutoModel.from_pretrained( 'OpenGVLab/InternVL-14B-224px', torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True).cuda().eval()

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternVL-14B-224px')

tokenizer = AutoTokenizer.from_pretrained( 'OpenGVLab/InternVL-14B-224px', use_fast=False, add_eos_token=True) tokenizer.pad_token_id = 0 # set pad_token_id to 0

images = [ Image.open('./examples/image1.jpg').convert('RGB'), Image.open('./examples/image2.jpg').convert('RGB'), Image.open('./examples/image3.jpg').convert('RGB') ] prefix = 'summarize:' texts = [ prefix + 'a photo of a red panda', # English prefix + '一张熊猫的照片', # Chinese prefix + '二匹の猫の写真' # Japanese ]

pixel_values = image_processor(images=images, return_tensors='pt').pixel_values pixel_values = pixel_values.to(torch.bfloat16).cuda() input_ids = tokenizer(texts, return_tensors='pt', max_length=80, truncation=True, padding='max_length').input_ids.cuda()

InternVL-C

logits_per_image, logits_per_text = model( image=pixel_values, text=input_ids, mode='InternVL-C') probs = logits_per_image.softmax(dim=-1)

tensor([[9.9609e-01, 5.2185e-03, 6.0070e-08],

[2.2949e-02, 9.7656e-01, 5.9903e-06],

[3.2932e-06, 7.4863e-05, 1.0000e+00]], device='cuda:0',

dtype=torch.bfloat16, grad_fn=)

InternVL-G

logits_per_image, logits_per_text = model( image=pixel_values, text=input_ids, mode='InternVL-G') probs = logits_per_image.softmax(dim=-1)

tensor([[9.9609e-01, 3.1738e-03, 3.6322e-08],

[8.6060e-03, 9.9219e-01, 2.8759e-06],

[1.7583e-06, 3.1233e-05, 1.0000e+00]], device='cuda:0',

dtype=torch.bfloat16, grad_fn=)

please set add_eos_token to False for generation

tokenizer.add_eos_token = False image = Image.open('./examples/image1.jpg').convert('RGB') pixel_values = image_processor(images=image, return_tensors='pt').pixel_values pixel_values = pixel_values.to(torch.bfloat16).cuda()

tokenized = tokenizer("English caption:", return_tensors='pt') pred = model.generate( pixel_values=pixel_values, input_ids=tokenized.input_ids.cuda(), attention_mask=tokenized.attention_mask.cuda(), num_beams=5, min_new_tokens=8, ) caption = tokenizer.decode(pred[0].cpu(), skip_special_tokens=True).strip()

English caption: a red panda sitting on top of a wooden platform

using InternVL 2.5 for multimodal chat (click to expand)

Here, we take the smaller OpenGVLab/InternVL2_5-8B as an example:

import numpy as np import torch import torchvision.transforms as T from decord import VideoReader, cpu from PIL import Image from torchvision.transforms.functional import InterpolationMode from transformers import AutoModel, AutoTokenizer

IMAGENET_MEAN = (0.485, 0.456, 0.406) IMAGENET_STD = (0.229, 0.224, 0.225)

def build_transform(input_size): MEAN, STD = IMAGENET_MEAN, IMAGENET_STD transform = T.Compose([ T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img), T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC), T.ToTensor(), T.Normalize(mean=MEAN, std=STD) ]) return transform

def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size): best_ratio_diff = float('inf') best_ratio = (1, 1) area = width * height for ratio in target_ratios: target_aspect_ratio = ratio[0] / ratio[1] ratio_diff = abs(aspect_ratio - target_aspect_ratio) if ratio_diff < best_ratio_diff: best_ratio_diff = ratio_diff best_ratio = ratio elif ratio_diff == best_ratio_diff: if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]: best_ratio = ratio return best_ratio

def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False): orig_width, orig_height = image.size aspect_ratio = orig_width / orig_height

# calculate the existing image aspect ratio
target_ratios = set(
    (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
    i * j <= max_num and i * j >= min_num)
target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])

# find the closest aspect ratio to the target
target_aspect_ratio = find_closest_aspect_ratio(
    aspect_ratio, target_ratios, orig_width, orig_height, image_size)

# calculate the target width and height
target_width = image_size * target_aspect_ratio[0]
target_height = image_size * target_aspect_ratio[1]
blocks = target_aspect_ratio[0] * target_aspect_ratio[1]

# resize the image
resized_img = image.resize((target_width, target_height))
processed_images = []
for i in range(blocks):
    box = (
        (i % (target_width // image_size)) * image_size,
        (i // (target_width // image_size)) * image_size,
        ((i % (target_width // image_size)) + 1) * image_size,
        ((i // (target_width // image_size)) + 1) * image_size
    )
    # split the image
    split_img = resized_img.crop(box)
    processed_images.append(split_img)
assert len(processed_images) == blocks
if use_thumbnail and len(processed_images) != 1:
    thumbnail_img = image.resize((image_size, image_size))
    processed_images.append(thumbnail_img)
return processed_images

def load_image(image_file, input_size=448, max_num=12): image = Image.open(image_file).convert('RGB') transform = build_transform(input_size=input_size) images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num) pixel_values = [transform(image) for image in images] pixel_values = torch.stack(pixel_values) return pixel_values

If you have an 80G A100 GPU, you can put the entire model on a single GPU.

Otherwise, you need to load a model using multiple GPUs, please refer to the Multiple GPUs section.

path = 'OpenGVLab/InternVL2_5-8B' model = AutoModel.from_pretrained( path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, trust_remote_code=True).eval().cuda() tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

set the max number of tiles in max_num

pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda() generation_config = dict(max_new_tokens=1024, do_sample=False)

pure-text conversation (纯文本对话)

question = 'Hello, who are you?' response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}')

question = 'Can you tell me a story?' response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}')

single-image single-round conversation (单图单轮对话)

question = '\nPlease describe the image shortly.' response = model.chat(tokenizer, pixel_values, question, generation_config) print(f'User: {question}\nAssistant: {response}')

single-image multi-round conversation (单图多轮对话)

question = '\nPlease describe the image in detail.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}')

question = 'Please write a poem according to the image.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}')

multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)

pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda() pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda() pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)

question = '\nDescribe the two images in detail.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}')

question = 'What are the similarities and differences between these two images.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}')

multi-image multi-round conversation, separate images (多图多轮对话，独立图像)

pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda() pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda() pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0) num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]

question = 'Image-1: \nImage-2: \nDescribe the two images in detail.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}')

question = 'What are the similarities and differences between these two images.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}')

batch inference, single image per sample (单图批处理)

pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda() pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda() num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)] pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)

questions = ['\nDescribe the image in detail.'] * len(num_patches_list) responses = model.batch_chat(tokenizer, pixel_values, num_patches_list=num_patches_list, questions=questions, generation_config=generation_config) for question, response in zip(questions, responses): print(f'User: {question}\nAssistant: {response}')

video multi-round conversation (视频多轮对话)

def get_index(bound, fps, max_frame, first_idx=0, num_segments=32): if bound: start, end = bound[0], bound[1] else: start, end = -100000, 100000 start_idx = max(first_idx, round(start * fps)) end_idx = min(round(end * fps), max_frame) seg_size = float(end_idx - start_idx) / num_segments frame_indices = np.array([ int(start_idx + (seg_size / 2) + np.round(seg_size * idx)) for idx in range(num_segments) ]) return frame_indices

def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32): vr = VideoReader(video_path, ctx=cpu(0), num_threads=1) max_frame = len(vr) - 1 fps = float(vr.get_avg_fps())

pixel_values_list, num_patches_list = [], []
transform = build_transform(input_size=input_size)
frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)
for frame_index in frame_indices:
    img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')
    img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)
    pixel_values = [transform(tile) for tile in img]
    pixel_values = torch.stack(pixel_values)
    num_patches_list.append(pixel_values.shape[0])
    pixel_values_list.append(pixel_values)
pixel_values = torch.cat(pixel_values_list)
return pixel_values, num_patches_list

video_path = './examples/red-panda.mp4' pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1) pixel_values = pixel_values.to(torch.bfloat16).cuda() video_prefix = ''.join([f'Frame-{i+1}: \n' for i in range(len(num_patches_list))]) question = video_prefix + 'What is the red panda doing?'

Frame1: \nFrame2: \n...\nFrame8: \n{question}

response, history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}')

question = 'Describe this video in detail.' response, history = model.chat(tokenizer, pixel_values, question, generation_config, num_patches_list=num_patches_list, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}')

License

This project is released under the MIT license. Parts of this project contain code and models from other sources, which are subject to their respective licenses.

Citation

If you find this project useful in your research, please consider cite:

@article{chen2024expanding, title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling}, author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others}, journal={arXiv preprint arXiv:2412.05271}, year={2024} } @article{wang2024mpo, title={Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization}, author={Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Zhu, Jinguo and Zhu, Xizhou and Lu, Lewei and Qiao, Yu and Dai, Jifeng}, journal={arXiv preprint arXiv:2411.10442}, year={2024} } @article{gao2024mini, title={Mini-InternVL: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance}, author={Gao, Zhangwei and Chen, Zhe and Cui, Erfei and Ren, Yiming and Wang, Weiyun and Zhu, Jinguo and Tian, Hao and Ye, Shenglong and He, Junjun and Zhu, Xizhou and others}, journal={Visual Intelligence}, volume={2}, number={1}, pages={1--17}, year={2024}, publisher={Springer} } @article{chen2024far, title={How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites}, author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others}, journal={Science China Information Sciences}, volume={67}, number={12}, pages={220101}, year={2024}, publisher={Springer} } @inproceedings{chen2024internvl, title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks}, author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={24185--24198}, year={2024} }

Acknowledgement

InternVL is built with reference to the code of the following projects: OpenAI CLIP, Open CLIP, CLIP Benchmark, EVA, InternImage, ViT-Adapter, MMSegmentation, Transformers, DINOv2, BLIP-2, Qwen-VL, and LLaVA-1.5. Thanks for their awesome work!

Scan the following QR Code, join our WeChat group.

image