Multimodal Inputs - vLLM (original) (raw)

This page teaches you how to pass multi-modal inputs to multi-modal models in vLLM.

Note

We are actively iterating on multi-modal support. See this RFC for upcoming changes, and open an issue on GitHub if you have any feedback or feature requests.

Tip

When serving multi-modal models, consider setting --allowed-media-domains to restrict domain that vLLM can access to prevent it from accessing arbitrary endpoints that can potentially be vulnerable to Server-Side Request Forgery (SSRF) attacks. You can provide a list of domains for this arg. For example: --allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com

Also, consider setting VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 to prevent HTTP redirects from being followed to bypass domain restrictions.

This restriction is especially important if you run vLLM in a containerized environment where the vLLM pods may have unrestricted access to internal networks.

Offline Inference

To input multi-modal data, follow this schema in [vllm.inputs.PromptType](../../api/vllm/inputs/#vllm.inputs.PromptType " PromptType = DecoderOnlyPrompt | EncoderDecoderPrompt

  module-attribute

"):

Image Inputs

You can pass a single image to the 'image' field of the multi-modal dictionary, as shown in the following examples:

Code

[](#%5F%5Fcodelineno-0-1)from vllm import LLM [](#%5F%5Fcodelineno-0-2) [](#%5F%5Fcodelineno-0-3)llm = LLM(model="llava-hf/llava-1.5-7b-hf") [](#%5F%5Fcodelineno-0-4) [](#%5F%5Fcodelineno-0-5)# Refer to the HuggingFace repo for the correct format to use [](#%5F%5Fcodelineno-0-6)prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:" [](#%5F%5Fcodelineno-0-7) [](#%5F%5Fcodelineno-0-8)# Load the image using PIL.Image [](#%5F%5Fcodelineno-0-9)image = PIL.Image.open(...) [](#%5F%5Fcodelineno-0-10) [](#%5F%5Fcodelineno-0-11)# Single prompt inference [](#%5F%5Fcodelineno-0-12)outputs = llm.generate({ [](#%5F%5Fcodelineno-0-13) "prompt": prompt, [](#%5F%5Fcodelineno-0-14) "multi_modal_data": {"image": image}, [](#%5F%5Fcodelineno-0-15)}) [](#%5F%5Fcodelineno-0-16) [](#%5F%5Fcodelineno-0-17)for o in outputs: [](#%5F%5Fcodelineno-0-18) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-0-19) print(generated_text) [](#%5F%5Fcodelineno-0-20) [](#%5F%5Fcodelineno-0-21)# Batch inference [](#%5F%5Fcodelineno-0-22)image_1 = PIL.Image.open(...) [](#%5F%5Fcodelineno-0-23)image_2 = PIL.Image.open(...) [](#%5F%5Fcodelineno-0-24)outputs = llm.generate( [](#%5F%5Fcodelineno-0-25) [ [](#%5F%5Fcodelineno-0-26) { [](#%5F%5Fcodelineno-0-27) "prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:", [](#%5F%5Fcodelineno-0-28) "multi_modal_data": {"image": image_1}, [](#%5F%5Fcodelineno-0-29) }, [](#%5F%5Fcodelineno-0-30) { [](#%5F%5Fcodelineno-0-31) "prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:", [](#%5F%5Fcodelineno-0-32) "multi_modal_data": {"image": image_2}, [](#%5F%5Fcodelineno-0-33) } [](#%5F%5Fcodelineno-0-34) ] [](#%5F%5Fcodelineno-0-35)) [](#%5F%5Fcodelineno-0-36) [](#%5F%5Fcodelineno-0-37)for o in outputs: [](#%5F%5Fcodelineno-0-38) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-0-39) print(generated_text)

Full example: examples/generate/multimodal/vision_language_offline.py

To substitute multiple images inside the same text prompt, you can pass in a list of images instead:

Code

[](#%5F%5Fcodelineno-1-1)from vllm import LLM [](#%5F%5Fcodelineno-1-2) [](#%5F%5Fcodelineno-1-3)llm = LLM( [](#%5F%5Fcodelineno-1-4) model="microsoft/Phi-3.5-vision-instruct", [](#%5F%5Fcodelineno-1-5) trust_remote_code=True, # Required to load Phi-3.5-vision [](#%5F%5Fcodelineno-1-6) max_model_len=4096, # Otherwise, it may not fit in smaller GPUs [](#%5F%5Fcodelineno-1-7) limit_mm_per_prompt={"image": 2}, # The maximum number to accept [](#%5F%5Fcodelineno-1-8)) [](#%5F%5Fcodelineno-1-9) [](#%5F%5Fcodelineno-1-10)# Refer to the HuggingFace repo for the correct format to use [](#%5F%5Fcodelineno-1-11)prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n" [](#%5F%5Fcodelineno-1-12) [](#%5F%5Fcodelineno-1-13)# Load the images using PIL.Image [](#%5F%5Fcodelineno-1-14)image1 = PIL.Image.open(...) [](#%5F%5Fcodelineno-1-15)image2 = PIL.Image.open(...) [](#%5F%5Fcodelineno-1-16) [](#%5F%5Fcodelineno-1-17)outputs = llm.generate({ [](#%5F%5Fcodelineno-1-18) "prompt": prompt, [](#%5F%5Fcodelineno-1-19) "multi_modal_data": {"image": [image1, image2]}, [](#%5F%5Fcodelineno-1-20)}) [](#%5F%5Fcodelineno-1-21) [](#%5F%5Fcodelineno-1-22)for o in outputs: [](#%5F%5Fcodelineno-1-23) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-1-24) print(generated_text)

Full example: examples/generate/multimodal/vision_language_multi_image_offline.py

If using the LLM.chat method, you can pass images directly in the message content using various formats: image URLs, PIL Image objects, or pre-computed embeddings:

Code

[](#%5F%5Fcodelineno-2-1)from vllm import LLM [](#%5F%5Fcodelineno-2-2)from vllm.assets.image import ImageAsset [](#%5F%5Fcodelineno-2-3) [](#%5F%5Fcodelineno-2-4)llm = LLM(model="llava-hf/llava-1.5-7b-hf") [](#%5F%5Fcodelineno-2-5)image_url = "https://picsum.photos/id/32/512/512" [](#%5F%5Fcodelineno-2-6)image_pil = ImageAsset('cherry_blossom').pil_image [](#%5F%5Fcodelineno-2-7)image_embeds = torch.load(...) [](#%5F%5Fcodelineno-2-8) [](#%5F%5Fcodelineno-2-9)conversation = [ [](#%5F%5Fcodelineno-2-10) {"role": "system", "content": "You are a helpful assistant"}, [](#%5F%5Fcodelineno-2-11) {"role": "user", "content": "Hello"}, [](#%5F%5Fcodelineno-2-12) {"role": "assistant", "content": "Hello! How can I assist you today?"}, [](#%5F%5Fcodelineno-2-13) { [](#%5F%5Fcodelineno-2-14) "role": "user", [](#%5F%5Fcodelineno-2-15) "content": [ [](#%5F%5Fcodelineno-2-16) { [](#%5F%5Fcodelineno-2-17) "type": "image_url", [](#%5F%5Fcodelineno-2-18) "image_url": {"url": image_url}, [](#%5F%5Fcodelineno-2-19) }, [](#%5F%5Fcodelineno-2-20) { [](#%5F%5Fcodelineno-2-21) "type": "image_pil", [](#%5F%5Fcodelineno-2-22) "image_pil": image_pil, [](#%5F%5Fcodelineno-2-23) }, [](#%5F%5Fcodelineno-2-24) { [](#%5F%5Fcodelineno-2-25) "type": "image_embeds", [](#%5F%5Fcodelineno-2-26) "image_embeds": image_embeds, [](#%5F%5Fcodelineno-2-27) }, [](#%5F%5Fcodelineno-2-28) { [](#%5F%5Fcodelineno-2-29) "type": "text", [](#%5F%5Fcodelineno-2-30) "text": "What's in these images?", [](#%5F%5Fcodelineno-2-31) }, [](#%5F%5Fcodelineno-2-32) ], [](#%5F%5Fcodelineno-2-33) }, [](#%5F%5Fcodelineno-2-34)] [](#%5F%5Fcodelineno-2-35) [](#%5F%5Fcodelineno-2-36)# Perform inference and log output. [](#%5F%5Fcodelineno-2-37)outputs = llm.chat(conversation) [](#%5F%5Fcodelineno-2-38) [](#%5F%5Fcodelineno-2-39)for o in outputs: [](#%5F%5Fcodelineno-2-40) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-2-41) print(generated_text)

Multi-image input can be extended to perform video captioning. We show this with Qwen2-VL as it supports videos:

Code

[](#%5F%5Fcodelineno-3-1)from vllm import LLM [](#%5F%5Fcodelineno-3-2) [](#%5F%5Fcodelineno-3-3)# Specify the maximum number of frames per video to be 4. This can be changed. [](#%5F%5Fcodelineno-3-4)llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4}) [](#%5F%5Fcodelineno-3-5) [](#%5F%5Fcodelineno-3-6)# Create the request payload. [](#%5F%5Fcodelineno-3-7)video_frames = ... # load your video making sure it only has the number of frames specified earlier. [](#%5F%5Fcodelineno-3-8)message = { [](#%5F%5Fcodelineno-3-9) "role": "user", [](#%5F%5Fcodelineno-3-10) "content": [ [](#%5F%5Fcodelineno-3-11) { [](#%5F%5Fcodelineno-3-12) "type": "text", [](#%5F%5Fcodelineno-3-13) "text": "Describe this set of frames. Consider the frames to be a part of the same video.", [](#%5F%5Fcodelineno-3-14) }, [](#%5F%5Fcodelineno-3-15) ], [](#%5F%5Fcodelineno-3-16)} [](#%5F%5Fcodelineno-3-17)for i in range(len(video_frames)): [](#%5F%5Fcodelineno-3-18) base64_image = encode_image(video_frames[i]) # base64 encoding. [](#%5F%5Fcodelineno-3-19) new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}} [](#%5F%5Fcodelineno-3-20) message["content"].append(new_image) [](#%5F%5Fcodelineno-3-21) [](#%5F%5Fcodelineno-3-22)# Perform inference and log output. [](#%5F%5Fcodelineno-3-23)outputs = llm.chat([message]) [](#%5F%5Fcodelineno-3-24) [](#%5F%5Fcodelineno-3-25)for o in outputs: [](#%5F%5Fcodelineno-3-26) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-3-27) print(generated_text)

Custom RGBA Background Color

When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the rgba_background_color parameter in media_io_kwargs.

Code

[](#%5F%5Fcodelineno-4-1)from vllm import LLM [](#%5F%5Fcodelineno-4-2) [](#%5F%5Fcodelineno-4-3)# Default white background (no configuration needed) [](#%5F%5Fcodelineno-4-4)llm = LLM(model="llava-hf/llava-1.5-7b-hf") [](#%5F%5Fcodelineno-4-5) [](#%5F%5Fcodelineno-4-6)# Custom black background for dark theme [](#%5F%5Fcodelineno-4-7)llm = LLM( [](#%5F%5Fcodelineno-4-8) model="llava-hf/llava-1.5-7b-hf", [](#%5F%5Fcodelineno-4-9) media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}}, [](#%5F%5Fcodelineno-4-10)) [](#%5F%5Fcodelineno-4-11) [](#%5F%5Fcodelineno-4-12)# Custom brand color background (e.g., blue) [](#%5F%5Fcodelineno-4-13)llm = LLM( [](#%5F%5Fcodelineno-4-14) model="llava-hf/llava-1.5-7b-hf", [](#%5F%5Fcodelineno-4-15) media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}}, [](#%5F%5Fcodelineno-4-16))

Note

Moondream3 Prompt Recipes

Moondream3ForCausalLM supports two task-specific prompt formats:

[](#%5F%5Fcodelineno-5-1)from vllm import LLM, SamplingParams [](#%5F%5Fcodelineno-5-2)from vllm.assets.image import ImageAsset [](#%5F%5Fcodelineno-5-3) [](#%5F%5Fcodelineno-5-4)llm = LLM( [](#%5F%5Fcodelineno-5-5) model="moondream/moondream3-preview", [](#%5F%5Fcodelineno-5-6) tokenizer="moondream/starmie-v1", [](#%5F%5Fcodelineno-5-7) trust_remote_code=True, [](#%5F%5Fcodelineno-5-8) max_model_len=2048, [](#%5F%5Fcodelineno-5-9) limit_mm_per_prompt={"image": 1}, [](#%5F%5Fcodelineno-5-10)) [](#%5F%5Fcodelineno-5-11) [](#%5F%5Fcodelineno-5-12)image = ImageAsset("stop_sign").pil_image [](#%5F%5Fcodelineno-5-13) [](#%5F%5Fcodelineno-5-14) [](#%5F%5Fcodelineno-5-15)def make_query_prompt(question: str) -> str: [](#%5F%5Fcodelineno-5-16) return ( [](#%5F%5Fcodelineno-5-17) "<|endoftext|><image><|md_reserved_0|>query<|md_reserved_1|>" [](#%5F%5Fcodelineno-5-18) f"{question}<|md_reserved_2|>" [](#%5F%5Fcodelineno-5-19) ) [](#%5F%5Fcodelineno-5-20) [](#%5F%5Fcodelineno-5-21) [](#%5F%5Fcodelineno-5-22)def make_caption_prompt(length: str = "normal") -> str: [](#%5F%5Fcodelineno-5-23) return ( [](#%5F%5Fcodelineno-5-24) "<|endoftext|><image><|md_reserved_0|>" [](#%5F%5Fcodelineno-5-25) f"describe<|md_reserved_1|>{length}<|md_reserved_2|>" [](#%5F%5Fcodelineno-5-26) ) [](#%5F%5Fcodelineno-5-27) [](#%5F%5Fcodelineno-5-28) [](#%5F%5Fcodelineno-5-29)query_out = llm.generate( [](#%5F%5Fcodelineno-5-30) { [](#%5F%5Fcodelineno-5-31) "prompt": make_query_prompt("What is shown in this image?"), [](#%5F%5Fcodelineno-5-32) "multi_modal_data": {"image": image}, [](#%5F%5Fcodelineno-5-33) }, [](#%5F%5Fcodelineno-5-34) SamplingParams(max_tokens=64, temperature=0), [](#%5F%5Fcodelineno-5-35))[0].outputs[0].text [](#%5F%5Fcodelineno-5-36) [](#%5F%5Fcodelineno-5-37)caption_out = llm.generate( [](#%5F%5Fcodelineno-5-38) { [](#%5F%5Fcodelineno-5-39) "prompt": make_caption_prompt(), [](#%5F%5Fcodelineno-5-40) "multi_modal_data": {"image": image}, [](#%5F%5Fcodelineno-5-41) }, [](#%5F%5Fcodelineno-5-42) SamplingParams(max_tokens=100, temperature=0), [](#%5F%5Fcodelineno-5-43))[0].outputs[0].text [](#%5F%5Fcodelineno-5-44) [](#%5F%5Fcodelineno-5-45)print("query:", query_out) [](#%5F%5Fcodelineno-5-46)print("caption:", caption_out)

Note

The native Moondream3 model also has detect and point skills. Those require custom coordinate decoding and are not exposed by this vLLM implementation.

Video Inputs

You can pass a list of NumPy arrays directly to the 'video' field of the multi-modal dictionary instead of using multi-image input.

Instead of NumPy arrays, you can also pass 'torch.Tensor' instances, as shown in this example using Qwen2.5-VL:

Code

[](#%5F%5Fcodelineno-6-1)from transformers import AutoProcessor [](#%5F%5Fcodelineno-6-2)from vllm import LLM, SamplingParams [](#%5F%5Fcodelineno-6-3)from qwen_vl_utils import process_vision_info [](#%5F%5Fcodelineno-6-4) [](#%5F%5Fcodelineno-6-5)model_path = "Qwen/Qwen2.5-VL-3B-Instruct" [](#%5F%5Fcodelineno-6-6)video_path = "https://content.pexels.com/videos/free-videos.mp4" [](#%5F%5Fcodelineno-6-7) [](#%5F%5Fcodelineno-6-8)llm = LLM( [](#%5F%5Fcodelineno-6-9) model=model_path, [](#%5F%5Fcodelineno-6-10) gpu_memory_utilization=0.8, [](#%5F%5Fcodelineno-6-11) enforce_eager=True, [](#%5F%5Fcodelineno-6-12) limit_mm_per_prompt={"video": 1}, [](#%5F%5Fcodelineno-6-13)) [](#%5F%5Fcodelineno-6-14) [](#%5F%5Fcodelineno-6-15)sampling_params = SamplingParams(max_tokens=1024) [](#%5F%5Fcodelineno-6-16) [](#%5F%5Fcodelineno-6-17)video_messages = [ [](#%5F%5Fcodelineno-6-18) { [](#%5F%5Fcodelineno-6-19) "role": "system", [](#%5F%5Fcodelineno-6-20) "content": "You are a helpful assistant.", [](#%5F%5Fcodelineno-6-21) }, [](#%5F%5Fcodelineno-6-22) { [](#%5F%5Fcodelineno-6-23) "role": "user", [](#%5F%5Fcodelineno-6-24) "content": [ [](#%5F%5Fcodelineno-6-25) {"type": "text", "text": "describe this video."}, [](#%5F%5Fcodelineno-6-26) { [](#%5F%5Fcodelineno-6-27) "type": "video", [](#%5F%5Fcodelineno-6-28) "video": video_path, [](#%5F%5Fcodelineno-6-29) "total_pixels": 20480 * 28 * 28, [](#%5F%5Fcodelineno-6-30) "min_pixels": 16 * 28 * 28, [](#%5F%5Fcodelineno-6-31) }, [](#%5F%5Fcodelineno-6-32) ] [](#%5F%5Fcodelineno-6-33) }, [](#%5F%5Fcodelineno-6-34)] [](#%5F%5Fcodelineno-6-35) [](#%5F%5Fcodelineno-6-36)messages = video_messages [](#%5F%5Fcodelineno-6-37)processor = AutoProcessor.from_pretrained(model_path) [](#%5F%5Fcodelineno-6-38)prompt = processor.apply_chat_template( [](#%5F%5Fcodelineno-6-39) messages, [](#%5F%5Fcodelineno-6-40) tokenize=False, [](#%5F%5Fcodelineno-6-41) add_generation_prompt=True, [](#%5F%5Fcodelineno-6-42)) [](#%5F%5Fcodelineno-6-43) [](#%5F%5Fcodelineno-6-44)image_inputs, video_inputs = process_vision_info(messages) [](#%5F%5Fcodelineno-6-45)mm_data = {} [](#%5F%5Fcodelineno-6-46)if video_inputs is not None: [](#%5F%5Fcodelineno-6-47) mm_data["video"] = video_inputs [](#%5F%5Fcodelineno-6-48) [](#%5F%5Fcodelineno-6-49)llm_inputs = { [](#%5F%5Fcodelineno-6-50) "prompt": prompt, [](#%5F%5Fcodelineno-6-51) "multi_modal_data": mm_data, [](#%5F%5Fcodelineno-6-52)} [](#%5F%5Fcodelineno-6-53) [](#%5F%5Fcodelineno-6-54)outputs = llm.generate([llm_inputs], sampling_params=sampling_params) [](#%5F%5Fcodelineno-6-55)for o in outputs: [](#%5F%5Fcodelineno-6-56) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-6-57) print(generated_text)

Note

'process_vision_info' is only applicable to Qwen2.5-VL and similar models.

Full example: examples/generate/multimodal/vision_language_offline.py

Audio Inputs

You can pass a tuple (array, sampling_rate) to the 'audio' field of the multi-modal dictionary.

Full example: examples/generate/multimodal/audio_language_offline.py

Chunking Long Audio for Transcription

Speech-to-text models like Whisper have a maximum audio length they can process (typically 30 seconds). For longer audio files, vLLM provides a utility to intelligently split audio into chunks at quiet points to minimize cutting through speech.

[](#%5F%5Fcodelineno-7-1)from vllm import LLM, SamplingParams [](#%5F%5Fcodelineno-7-2)from vllm.multimodal.audio import split_audio [](#%5F%5Fcodelineno-7-3)from vllm.multimodal.media.audio import load_audio [](#%5F%5Fcodelineno-7-4) [](#%5F%5Fcodelineno-7-5)# Load long audio file [](#%5F%5Fcodelineno-7-6)audio, sr = load_audio("long_audio.wav", sr=16000) [](#%5F%5Fcodelineno-7-7) [](#%5F%5Fcodelineno-7-8)# Split into chunks at low-energy (quiet) regions [](#%5F%5Fcodelineno-7-9)chunks = split_audio( [](#%5F%5Fcodelineno-7-10) audio_data=audio, [](#%5F%5Fcodelineno-7-11) sample_rate=sr, [](#%5F%5Fcodelineno-7-12) max_clip_duration_s=30.0, # Maximum chunk length in seconds [](#%5F%5Fcodelineno-7-13) overlap_duration_s=1.0, # Search window for finding quiet split points [](#%5F%5Fcodelineno-7-14) min_energy_window_size=1600, # Window size for energy calculation (~100ms at 16kHz) [](#%5F%5Fcodelineno-7-15)) [](#%5F%5Fcodelineno-7-16) [](#%5F%5Fcodelineno-7-17)# Initialize Whisper model [](#%5F%5Fcodelineno-7-18)llm = LLM(model="openai/whisper-large-v3-turbo") [](#%5F%5Fcodelineno-7-19)sampling_params = SamplingParams(temperature=0, max_tokens=256) [](#%5F%5Fcodelineno-7-20) [](#%5F%5Fcodelineno-7-21)# Transcribe each chunk [](#%5F%5Fcodelineno-7-22)transcriptions = [] [](#%5F%5Fcodelineno-7-23)for chunk in chunks: [](#%5F%5Fcodelineno-7-24) outputs = llm.generate({ [](#%5F%5Fcodelineno-7-25) "prompt": "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>", [](#%5F%5Fcodelineno-7-26) "multi_modal_data": {"audio": (chunk, sr)}, [](#%5F%5Fcodelineno-7-27) }, sampling_params) [](#%5F%5Fcodelineno-7-28) transcriptions.append(outputs[0].outputs[0].text) [](#%5F%5Fcodelineno-7-29) [](#%5F%5Fcodelineno-7-30)# Combine results [](#%5F%5Fcodelineno-7-31)full_transcription = " ".join(transcriptions)

The split_audio function:

Automatic Audio Channel Normalization

vLLM automatically normalizes audio channels for models that require specific audio formats. When loading audio with libraries like torchaudio, stereo files return shape [channels, time], but many audio models (particularly Whisper-based models) expect mono audio with shape [time].

Supported models with automatic mono conversion:

For these models, vLLM automatically:

  1. Detects if the model requires mono audio via the feature extractor
  2. Converts multi-channel audio to mono using channel averaging
  3. Handles both (channels, time) format (torchaudio) and (time, channels) format (soundfile)

Example with stereo audio:

[](#%5F%5Fcodelineno-8-1)import torchaudio [](#%5F%5Fcodelineno-8-2)from vllm import LLM [](#%5F%5Fcodelineno-8-3) [](#%5F%5Fcodelineno-8-4)# Load stereo audio file - returns (channels, time) shape [](#%5F%5Fcodelineno-8-5)audio, sr = torchaudio.load("stereo_audio.wav") [](#%5F%5Fcodelineno-8-6)print(f"Original shape: {audio.shape}") # e.g., torch.Size([2, 16000]) [](#%5F%5Fcodelineno-8-7) [](#%5F%5Fcodelineno-8-8)# vLLM automatically converts to mono for Whisper-based models [](#%5F%5Fcodelineno-8-9)llm = LLM(model="openai/whisper-large-v3") [](#%5F%5Fcodelineno-8-10) [](#%5F%5Fcodelineno-8-11)outputs = llm.generate({ [](#%5F%5Fcodelineno-8-12) "prompt": "", [](#%5F%5Fcodelineno-8-13) "multi_modal_data": {"audio": (audio.numpy(), sr)}, [](#%5F%5Fcodelineno-8-14)})

No manual conversion is needed - vLLM handles the channel normalization automatically based on the model's requirements.

Embedding Inputs

To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model, pass a tensor of shape (..., hidden_size of LM) to the corresponding field of the multi-modal dictionary. The exact shape depends on the model being used.

You must enable this feature via enable_mm_embeds=True.

Warning

The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users!

Image Embeddings

Code

[](#%5F%5Fcodelineno-9-1)from vllm import LLM [](#%5F%5Fcodelineno-9-2) [](#%5F%5Fcodelineno-9-3)# Inference with image embeddings as input [](#%5F%5Fcodelineno-9-4)llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True) [](#%5F%5Fcodelineno-9-5) [](#%5F%5Fcodelineno-9-6)# Refer to the HuggingFace repo for the correct format to use [](#%5F%5Fcodelineno-9-7)prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:" [](#%5F%5Fcodelineno-9-8) [](#%5F%5Fcodelineno-9-9)# For most models, `image_embeds` has shape: (num_images, image_feature_size, hidden_size) [](#%5F%5Fcodelineno-9-10)image_embeds = torch.load(...) [](#%5F%5Fcodelineno-9-11) [](#%5F%5Fcodelineno-9-12)outputs = llm.generate({ [](#%5F%5Fcodelineno-9-13) "prompt": prompt, [](#%5F%5Fcodelineno-9-14) "multi_modal_data": {"image": image_embeds}, [](#%5F%5Fcodelineno-9-15)}) [](#%5F%5Fcodelineno-9-16) [](#%5F%5Fcodelineno-9-17)for o in outputs: [](#%5F%5Fcodelineno-9-18) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-9-19) print(generated_text) [](#%5F%5Fcodelineno-9-20) [](#%5F%5Fcodelineno-9-21)# Additional examples for models that require extra fields [](#%5F%5Fcodelineno-9-22)llm = LLM( [](#%5F%5Fcodelineno-9-23) "Qwen/Qwen2-VL-2B-Instruct", [](#%5F%5Fcodelineno-9-24) limit_mm_per_prompt={"image": 4}, [](#%5F%5Fcodelineno-9-25) enable_mm_embeds=True, [](#%5F%5Fcodelineno-9-26)) [](#%5F%5Fcodelineno-9-27)mm_data = { [](#%5F%5Fcodelineno-9-28) "image": { [](#%5F%5Fcodelineno-9-29) # Shape: (total_feature_size, hidden_size) [](#%5F%5Fcodelineno-9-30) # total_feature_size = sum(image_feature_size for image in images) [](#%5F%5Fcodelineno-9-31) "image_embeds": torch.load(...), [](#%5F%5Fcodelineno-9-32) # Shape: (num_images, 3) [](#%5F%5Fcodelineno-9-33) # image_grid_thw is needed to calculate positional encoding. [](#%5F%5Fcodelineno-9-34) "image_grid_thw": torch.load(...), [](#%5F%5Fcodelineno-9-35) } [](#%5F%5Fcodelineno-9-36)} [](#%5F%5Fcodelineno-9-37) [](#%5F%5Fcodelineno-9-38)llm = LLM( [](#%5F%5Fcodelineno-9-39) "openbmb/MiniCPM-V-2_6", [](#%5F%5Fcodelineno-9-40) trust_remote_code=True, [](#%5F%5Fcodelineno-9-41) limit_mm_per_prompt={"image": 4}, [](#%5F%5Fcodelineno-9-42) enable_mm_embeds=True, [](#%5F%5Fcodelineno-9-43)) [](#%5F%5Fcodelineno-9-44)mm_data = { [](#%5F%5Fcodelineno-9-45) "image": { [](#%5F%5Fcodelineno-9-46) # Shape: (num_images, num_slices, hidden_size) [](#%5F%5Fcodelineno-9-47) # num_slices can differ for each image [](#%5F%5Fcodelineno-9-48) "image_embeds": [torch.load(...) for image in images], [](#%5F%5Fcodelineno-9-49) # Shape: (num_images, 2) [](#%5F%5Fcodelineno-9-50) # image_sizes is needed to calculate details of the sliced image. [](#%5F%5Fcodelineno-9-51) "image_sizes": [image.size for image in images], [](#%5F%5Fcodelineno-9-52) } [](#%5F%5Fcodelineno-9-53)}

For Qwen3-VL, the image_embeds should contain both the base image embedding and deepstack features.

Audio Embedding Inputs

You can pass pre-computed audio embeddings similar to image embeddings:

Code

[](#%5F%5Fcodelineno-10-1)from vllm import LLM [](#%5F%5Fcodelineno-10-2)import torch [](#%5F%5Fcodelineno-10-3) [](#%5F%5Fcodelineno-10-4)# Enable audio embeddings support [](#%5F%5Fcodelineno-10-5)llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True) [](#%5F%5Fcodelineno-10-6) [](#%5F%5Fcodelineno-10-7)# Refer to the HuggingFace repo for the correct format to use [](#%5F%5Fcodelineno-10-8)prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:" [](#%5F%5Fcodelineno-10-9) [](#%5F%5Fcodelineno-10-10)# Load pre-computed audio embeddings, usually with shape: [](#%5F%5Fcodelineno-10-11)# (num_audios, audio_feature_size, hidden_size of LM) [](#%5F%5Fcodelineno-10-12)audio_embeds = torch.load(...) [](#%5F%5Fcodelineno-10-13) [](#%5F%5Fcodelineno-10-14)outputs = llm.generate({ [](#%5F%5Fcodelineno-10-15) "prompt": prompt, [](#%5F%5Fcodelineno-10-16) "multi_modal_data": {"audio": audio_embeds}, [](#%5F%5Fcodelineno-10-17)}) [](#%5F%5Fcodelineno-10-18) [](#%5F%5Fcodelineno-10-19)for o in outputs: [](#%5F%5Fcodelineno-10-20) generated_text = o.outputs[0].text [](#%5F%5Fcodelineno-10-21) print(generated_text)

Cached Inputs

When using multi-modal inputs, vLLM normally hashes each media item by content to enable caching across requests. You can optionally pass multi_modal_uuids to provide your own stable IDs for each item so caching can reuse work across requests without rehashing the raw content.

Code

[](#%5F%5Fcodelineno-11-1)from vllm import LLM [](#%5F%5Fcodelineno-11-2)from PIL import Image [](#%5F%5Fcodelineno-11-3) [](#%5F%5Fcodelineno-11-4)# Qwen2.5-VL example with two images [](#%5F%5Fcodelineno-11-5)llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct") [](#%5F%5Fcodelineno-11-6) [](#%5F%5Fcodelineno-11-7)prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:" [](#%5F%5Fcodelineno-11-8)img_a = Image.open("/path/to/a.jpg") [](#%5F%5Fcodelineno-11-9)img_b = Image.open("/path/to/b.jpg") [](#%5F%5Fcodelineno-11-10) [](#%5F%5Fcodelineno-11-11)outputs = llm.generate({ [](#%5F%5Fcodelineno-11-12) "prompt": prompt, [](#%5F%5Fcodelineno-11-13) "multi_modal_data": {"image": [img_a, img_b]}, [](#%5F%5Fcodelineno-11-14) # Provide stable IDs for caching. [](#%5F%5Fcodelineno-11-15) # Requirements (matched by this example): [](#%5F%5Fcodelineno-11-16) # - Include every modality present in multi_modal_data. [](#%5F%5Fcodelineno-11-17) # - For lists, provide the same number of entries. [](#%5F%5Fcodelineno-11-18) # - Use None to fall back to content hashing for that item. [](#%5F%5Fcodelineno-11-19) "multi_modal_uuids": {"image": ["sku-1234-a", None]}, [](#%5F%5Fcodelineno-11-20)}) [](#%5F%5Fcodelineno-11-21) [](#%5F%5Fcodelineno-11-22)for o in outputs: [](#%5F%5Fcodelineno-11-23) print(o.outputs[0].text)

Using UUIDs, you can also skip sending media data entirely if you expect cache hits for respective items. Note that the request will fail if the skipped media doesn't have a corresponding UUID, or if the UUID fails to hit the cache.

Code

[](#%5F%5Fcodelineno-12-1)from vllm import LLM [](#%5F%5Fcodelineno-12-2)from PIL import Image [](#%5F%5Fcodelineno-12-3) [](#%5F%5Fcodelineno-12-4)# Qwen2.5-VL example with two images [](#%5F%5Fcodelineno-12-5)llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct") [](#%5F%5Fcodelineno-12-6) [](#%5F%5Fcodelineno-12-7)prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:" [](#%5F%5Fcodelineno-12-8)img_b = Image.open("/path/to/b.jpg") [](#%5F%5Fcodelineno-12-9) [](#%5F%5Fcodelineno-12-10)outputs = llm.generate({ [](#%5F%5Fcodelineno-12-11) "prompt": prompt, [](#%5F%5Fcodelineno-12-12) "multi_modal_data": {"image": [None, img_b]}, [](#%5F%5Fcodelineno-12-13) # Since img_a is expected to be cached, we can skip sending the actual [](#%5F%5Fcodelineno-12-14) # image entirely. [](#%5F%5Fcodelineno-12-15) "multi_modal_uuids": {"image": ["sku-1234-a", None]}, [](#%5F%5Fcodelineno-12-16)}) [](#%5F%5Fcodelineno-12-17) [](#%5F%5Fcodelineno-12-18)for o in outputs: [](#%5F%5Fcodelineno-12-19) print(o.outputs[0].text)

Warning

If both multimodal processor caching and prefix caching are disabled, user-provided multi_modal_uuids are ignored.

Online Serving

Our OpenAI-compatible server accepts multi-modal data via the Chat Completions API. Media inputs also support optional UUIDs users can provide to uniquely identify each media, which is used to cache the media results across requests.

Important

A chat template is required to use Chat Completions API. For HF format models, the default chat template is defined inside chat_template.json or tokenizer_config.json.

If no default chat template is available, we will first look for a built-in fallback in vllm/transformers_utils/chat_templates/registry.py. If no fallback is available, an error is raised and you have to provide the chat template manually via the --chat-template argument.

For certain models, we provide alternative chat templates inside examples. For example, VLM2Vec uses examples/pooling/embed/template/vlm2vec_phi3v.jinja which is different from the default one for Phi-3-Vision.

Image Inputs

Image input is supported according to OpenAI Vision API. Here is a simple example using Phi-3.5-Vision.

First, launch the OpenAI-compatible server:

[](#%5F%5Fcodelineno-13-1)vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \ [](#%5F%5Fcodelineno-13-2) --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2

Then, you can use the OpenAI client as follows:

Code

[](#%5F%5Fcodelineno-14-1)import os [](#%5F%5Fcodelineno-14-2)from openai import OpenAI [](#%5F%5Fcodelineno-14-3) [](#%5F%5Fcodelineno-14-4)openai_api_key = "EMPTY" [](#%5F%5Fcodelineno-14-5)openai_api_base = "http://localhost:8000/v1" [](#%5F%5Fcodelineno-14-6) [](#%5F%5Fcodelineno-14-7)client = OpenAI( [](#%5F%5Fcodelineno-14-8) api_key=openai_api_key, [](#%5F%5Fcodelineno-14-9) base_url=openai_api_base, [](#%5F%5Fcodelineno-14-10)) [](#%5F%5Fcodelineno-14-11) [](#%5F%5Fcodelineno-14-12)# Single-image input inference [](#%5F%5Fcodelineno-14-13) [](#%5F%5Fcodelineno-14-14)# Public image URL for testing remote image processing [](#%5F%5Fcodelineno-14-15)image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" [](#%5F%5Fcodelineno-14-16) [](#%5F%5Fcodelineno-14-17)# Create chat completion with remote image [](#%5F%5Fcodelineno-14-18)chat_response = client.chat.completions.create( [](#%5F%5Fcodelineno-14-19) model="microsoft/Phi-3.5-vision-instruct", [](#%5F%5Fcodelineno-14-20) messages=[ [](#%5F%5Fcodelineno-14-21) { [](#%5F%5Fcodelineno-14-22) "role": "user", [](#%5F%5Fcodelineno-14-23) "content": [ [](#%5F%5Fcodelineno-14-24) # NOTE: The prompt formatting with the image token `<image>` is not needed [](#%5F%5Fcodelineno-14-25) # since the prompt will be processed automatically by the API server. [](#%5F%5Fcodelineno-14-26) { [](#%5F%5Fcodelineno-14-27) "type": "text", [](#%5F%5Fcodelineno-14-28) "text": "What’s in this image?", [](#%5F%5Fcodelineno-14-29) }, [](#%5F%5Fcodelineno-14-30) { [](#%5F%5Fcodelineno-14-31) "type": "image_url", [](#%5F%5Fcodelineno-14-32) "image_url": {"url": image_url}, [](#%5F%5Fcodelineno-14-33) "uuid": image_url, # Optional [](#%5F%5Fcodelineno-14-34) }, [](#%5F%5Fcodelineno-14-35) ], [](#%5F%5Fcodelineno-14-36) } [](#%5F%5Fcodelineno-14-37) ], [](#%5F%5Fcodelineno-14-38)) [](#%5F%5Fcodelineno-14-39)print("Chat completion output:", chat_response.choices[0].message.content) [](#%5F%5Fcodelineno-14-40) [](#%5F%5Fcodelineno-14-41)# Local image file path (update this to point to your actual image file) [](#%5F%5Fcodelineno-14-42)image_file = "/path/to/image.jpg" [](#%5F%5Fcodelineno-14-43) [](#%5F%5Fcodelineno-14-44)# Create chat completion with local image file [](#%5F%5Fcodelineno-14-45)# Launch the API server/engine with the --allowed-local-media-path argument. [](#%5F%5Fcodelineno-14-46)if os.path.exists(image_file): [](#%5F%5Fcodelineno-14-47) chat_completion_from_local_image_url = client.chat.completions.create( [](#%5F%5Fcodelineno-14-48) model="microsoft/Phi-3.5-vision-instruct", [](#%5F%5Fcodelineno-14-49) messages=[ [](#%5F%5Fcodelineno-14-50) { [](#%5F%5Fcodelineno-14-51) "role": "user", [](#%5F%5Fcodelineno-14-52) "content": [ [](#%5F%5Fcodelineno-14-53) { [](#%5F%5Fcodelineno-14-54) "type": "text", [](#%5F%5Fcodelineno-14-55) "text": "What’s in this image?", [](#%5F%5Fcodelineno-14-56) }, [](#%5F%5Fcodelineno-14-57) { [](#%5F%5Fcodelineno-14-58) "type": "image_url", [](#%5F%5Fcodelineno-14-59) "image_url": {"url": f"file://{image_file}"}, [](#%5F%5Fcodelineno-14-60) }, [](#%5F%5Fcodelineno-14-61) ], [](#%5F%5Fcodelineno-14-62) } [](#%5F%5Fcodelineno-14-63) ], [](#%5F%5Fcodelineno-14-64) ) [](#%5F%5Fcodelineno-14-65) result = chat_completion_from_local_image_url.choices[0].message.content [](#%5F%5Fcodelineno-14-66) print("Chat completion output from local image file:\n", result) [](#%5F%5Fcodelineno-14-67)else: [](#%5F%5Fcodelineno-14-68) print(f"Local image file not found at {image_file}, skipping local file test.") [](#%5F%5Fcodelineno-14-69) [](#%5F%5Fcodelineno-14-70)# Multi-image input inference [](#%5F%5Fcodelineno-14-71)image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg" [](#%5F%5Fcodelineno-14-72)image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg" [](#%5F%5Fcodelineno-14-73) [](#%5F%5Fcodelineno-14-74)chat_response = client.chat.completions.create( [](#%5F%5Fcodelineno-14-75) model="microsoft/Phi-3.5-vision-instruct", [](#%5F%5Fcodelineno-14-76) messages=[ [](#%5F%5Fcodelineno-14-77) { [](#%5F%5Fcodelineno-14-78) "role": "user", [](#%5F%5Fcodelineno-14-79) "content": [ [](#%5F%5Fcodelineno-14-80) { [](#%5F%5Fcodelineno-14-81) "type": "text", [](#%5F%5Fcodelineno-14-82) "text": "What are the animals in these images?", [](#%5F%5Fcodelineno-14-83) }, [](#%5F%5Fcodelineno-14-84) { [](#%5F%5Fcodelineno-14-85) "type": "image_url", [](#%5F%5Fcodelineno-14-86) "image_url": {"url": image_url_duck}, [](#%5F%5Fcodelineno-14-87) "uuid": image_url_duck, # Optional [](#%5F%5Fcodelineno-14-88) }, [](#%5F%5Fcodelineno-14-89) { [](#%5F%5Fcodelineno-14-90) "type": "image_url", [](#%5F%5Fcodelineno-14-91) "image_url": {"url": image_url_lion}, [](#%5F%5Fcodelineno-14-92) "uuid": image_url_lion, # Optional [](#%5F%5Fcodelineno-14-93) }, [](#%5F%5Fcodelineno-14-94) ], [](#%5F%5Fcodelineno-14-95) } [](#%5F%5Fcodelineno-14-96) ], [](#%5F%5Fcodelineno-14-97)) [](#%5F%5Fcodelineno-14-98)print("Chat completion output:", chat_response.choices[0].message.content)

Full example: examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py

Tip

Loading from local file paths is also supported on vLLM: You can specify the allowed local media path via --allowed-local-media-path when launching the API server/engine, and pass the file path as url in the API request.

Tip

There is no need to place image placeholders in the text content of the API request - they are already represented by the image content. In fact, you can place image placeholders in the middle of the text by interleaving text and image content.

Note

By default, the timeout for fetching images through HTTP URL is 5 seconds. You can override this by setting the environment variable:

[](#%5F%5Fcodelineno-15-1)export VLLM_IMAGE_FETCH_TIMEOUT=<timeout>

Video Inputs

Instead of image_url, you can pass a video file via video_url. Here is a simple example using LLaVA-OneVision.

First, launch the OpenAI-compatible server:

[](#%5F%5Fcodelineno-16-1)vllm serve llava-hf/llava-onevision-qwen2-0.5b-ov-hf --runner generate --max-model-len 8192

Then, you can use the OpenAI client as follows:

Code

[](#%5F%5Fcodelineno-17-1)from openai import OpenAI [](#%5F%5Fcodelineno-17-2) [](#%5F%5Fcodelineno-17-3)openai_api_key = "EMPTY" [](#%5F%5Fcodelineno-17-4)openai_api_base = "http://localhost:8000/v1" [](#%5F%5Fcodelineno-17-5) [](#%5F%5Fcodelineno-17-6)client = OpenAI( [](#%5F%5Fcodelineno-17-7) api_key=openai_api_key, [](#%5F%5Fcodelineno-17-8) base_url=openai_api_base, [](#%5F%5Fcodelineno-17-9)) [](#%5F%5Fcodelineno-17-10) [](#%5F%5Fcodelineno-17-11)video_url = "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4" [](#%5F%5Fcodelineno-17-12) [](#%5F%5Fcodelineno-17-13)## Use video url in the payload [](#%5F%5Fcodelineno-17-14)chat_completion_from_url = client.chat.completions.create( [](#%5F%5Fcodelineno-17-15) messages=[ [](#%5F%5Fcodelineno-17-16) { [](#%5F%5Fcodelineno-17-17) "role": "user", [](#%5F%5Fcodelineno-17-18) "content": [ [](#%5F%5Fcodelineno-17-19) { [](#%5F%5Fcodelineno-17-20) "type": "text", [](#%5F%5Fcodelineno-17-21) "text": "What's in this video?", [](#%5F%5Fcodelineno-17-22) }, [](#%5F%5Fcodelineno-17-23) { [](#%5F%5Fcodelineno-17-24) "type": "video_url", [](#%5F%5Fcodelineno-17-25) "video_url": {"url": video_url}, [](#%5F%5Fcodelineno-17-26) "uuid": video_url, # Optional [](#%5F%5Fcodelineno-17-27) }, [](#%5F%5Fcodelineno-17-28) ], [](#%5F%5Fcodelineno-17-29) } [](#%5F%5Fcodelineno-17-30) ], [](#%5F%5Fcodelineno-17-31) model=model, [](#%5F%5Fcodelineno-17-32) max_completion_tokens=64, [](#%5F%5Fcodelineno-17-33)) [](#%5F%5Fcodelineno-17-34) [](#%5F%5Fcodelineno-17-35)result = chat_completion_from_url.choices[0].message.content [](#%5F%5Fcodelineno-17-36)print("Chat completion output from image url:", result)

Full example: examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py

Note

By default, the timeout for fetching videos through HTTP URL is 30 seconds. You can override this by setting the environment variable:

[](#%5F%5Fcodelineno-18-1)export VLLM_VIDEO_FETCH_TIMEOUT=<timeout>

Video Frame Recovery

For improved robustness when processing potentially corrupted or truncated video files, vLLM supports optional frame recovery using a dynamic window forward-scan approach. When enabled, if a target frame fails to load during sequential reading, the next successfully grabbed frame (before the next target frame) will be used in its place.

To enable video frame recovery, pass the frame_recovery parameter via --media-io-kwargs:

[](#%5F%5Fcodelineno-19-1)# Example: Enable frame recovery [](#%5F%5Fcodelineno-19-2)vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \ [](#%5F%5Fcodelineno-19-3) --media-io-kwargs '{"video": {"frame_recovery": true}}'

Parameters:

How it works:

  1. The system reads frames sequentially
  2. If a target frame fails to grab, it's marked as "failed"
  3. The next successfully grabbed frame (before reaching the next target) is used to recover the failed frame
  4. This approach handles both mid-video corruption and end-of-video truncation

Works with common video formats like MP4 when using OpenCV backends.

When you extract video frames on the client side and send them as video/jpeg (base64-concatenated JPEG frames), you can preserve the original video metadata by using media_io_kwargs in your request. This enables more accurate video understanding by preserving temporal information that would otherwise be lost during client-side frame extraction.

Supported Parameters:

Parameter Type Description
fps float Frame rate of the original video
frames_indices list[int] Indices of the actually sampled frames
total_num_frames int Total frame count of the original video
duration float Duration of the original video in seconds
do_sample_frames bool Whether to perform frame sampling

Code

[](#%5F%5Fcodelineno-20-1)from openai import OpenAI [](#%5F%5Fcodelineno-20-2) [](#%5F%5Fcodelineno-20-3)client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") [](#%5F%5Fcodelineno-20-4) [](#%5F%5Fcodelineno-20-5)# Client-side frame extraction [](#%5F%5Fcodelineno-20-6)frames = extract_frames(video_path, num_frames=32) [](#%5F%5Fcodelineno-20-7)frames_b64 = ",".join([encode_image(f) for f in frames]) [](#%5F%5Fcodelineno-20-8)video_url = f"data:video/jpeg;base64,{frames_b64}" [](#%5F%5Fcodelineno-20-9) [](#%5F%5Fcodelineno-20-10)# Pass video metadata via media_io_kwargs [](#%5F%5Fcodelineno-20-11)response = client.chat.completions.create( [](#%5F%5Fcodelineno-20-12) model="your-multimodal-model", [](#%5F%5Fcodelineno-20-13) messages=[{ [](#%5F%5Fcodelineno-20-14) "role": "user", [](#%5F%5Fcodelineno-20-15) "content": [ [](#%5F%5Fcodelineno-20-16) {"type": "video_url", "video_url": {"url": video_url}}, [](#%5F%5Fcodelineno-20-17) {"type": "text", "text": "Describe what happens in this video."} [](#%5F%5Fcodelineno-20-18) ] [](#%5F%5Fcodelineno-20-19) }], [](#%5F%5Fcodelineno-20-20) extra_body={ [](#%5F%5Fcodelineno-20-21) "media_io_kwargs": { [](#%5F%5Fcodelineno-20-22) "video": { [](#%5F%5Fcodelineno-20-23) "fps": 30.0, [](#%5F%5Fcodelineno-20-24) "frames_indices": [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, [](#%5F%5Fcodelineno-20-25) 100, 110, 120, 130, 140, 150, 160, 170, [](#%5F%5Fcodelineno-20-26) 180, 190, 200, 210, 220, 230, 240, 250, [](#%5F%5Fcodelineno-20-27) 260, 270, 280, 290, 300, 310], [](#%5F%5Fcodelineno-20-28) "total_num_frames": 900, [](#%5F%5Fcodelineno-20-29) "duration": 30.0, [](#%5F%5Fcodelineno-20-30) } [](#%5F%5Fcodelineno-20-31) } [](#%5F%5Fcodelineno-20-32) }, [](#%5F%5Fcodelineno-20-33)) [](#%5F%5Fcodelineno-20-34) [](#%5F%5Fcodelineno-20-35)print(response.choices[0].message.content)

Why use media_io_kwargs?

When extracting frames client-side, the server loses important context about the original video:

By passing this metadata, the model can better understand the temporal distribution of the sampled frames and whether important moments might have been skipped.

Custom RGBA Background Color

To use a custom background color for RGBA images, pass the rgba_background_color parameter via --media-io-kwargs:

[](#%5F%5Fcodelineno-21-1)# Example: Black background for dark theme [](#%5F%5Fcodelineno-21-2)vllm serve llava-hf/llava-1.5-7b-hf \ [](#%5F%5Fcodelineno-21-3) --media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}' [](#%5F%5Fcodelineno-21-4) [](#%5F%5Fcodelineno-21-5)# Example: Custom gray background [](#%5F%5Fcodelineno-21-6)vllm serve llava-hf/llava-1.5-7b-hf \ [](#%5F%5Fcodelineno-21-7) --media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'

Audio Inputs

Audio input is supported according to OpenAI Audio API. Here is a simple example using Ultravox-v0.5-1B.

First, launch the OpenAI-compatible server:

[](#%5F%5Fcodelineno-22-1)vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b

Then, you can use the OpenAI client as follows:

Code

[](#%5F%5Fcodelineno-23-1)import base64 [](#%5F%5Fcodelineno-23-2)import requests [](#%5F%5Fcodelineno-23-3)from openai import OpenAI [](#%5F%5Fcodelineno-23-4)from vllm.assets.audio import AudioAsset [](#%5F%5Fcodelineno-23-5) [](#%5F%5Fcodelineno-23-6)def encode_base64_content_from_url(content_url: str) -> str: [](#%5F%5Fcodelineno-23-7) """Encode a content retrieved from a remote url to base64 format.""" [](#%5F%5Fcodelineno-23-8) [](#%5F%5Fcodelineno-23-9) with requests.get(content_url) as response: [](#%5F%5Fcodelineno-23-10) response.raise_for_status() [](#%5F%5Fcodelineno-23-11) result = base64.b64encode(response.content).decode('utf-8') [](#%5F%5Fcodelineno-23-12) [](#%5F%5Fcodelineno-23-13) return result [](#%5F%5Fcodelineno-23-14) [](#%5F%5Fcodelineno-23-15)openai_api_key = "EMPTY" [](#%5F%5Fcodelineno-23-16)openai_api_base = "http://localhost:8000/v1" [](#%5F%5Fcodelineno-23-17) [](#%5F%5Fcodelineno-23-18)client = OpenAI( [](#%5F%5Fcodelineno-23-19) api_key=openai_api_key, [](#%5F%5Fcodelineno-23-20) base_url=openai_api_base, [](#%5F%5Fcodelineno-23-21)) [](#%5F%5Fcodelineno-23-22) [](#%5F%5Fcodelineno-23-23)# Any format supported by soundfile/PyAV is supported [](#%5F%5Fcodelineno-23-24)audio_url = AudioAsset("winning_call").url [](#%5F%5Fcodelineno-23-25)audio_base64 = encode_base64_content_from_url(audio_url) [](#%5F%5Fcodelineno-23-26) [](#%5F%5Fcodelineno-23-27)chat_completion_from_base64 = client.chat.completions.create( [](#%5F%5Fcodelineno-23-28) messages=[ [](#%5F%5Fcodelineno-23-29) { [](#%5F%5Fcodelineno-23-30) "role": "user", [](#%5F%5Fcodelineno-23-31) "content": [ [](#%5F%5Fcodelineno-23-32) { [](#%5F%5Fcodelineno-23-33) "type": "text", [](#%5F%5Fcodelineno-23-34) "text": "What's in this audio?", [](#%5F%5Fcodelineno-23-35) }, [](#%5F%5Fcodelineno-23-36) { [](#%5F%5Fcodelineno-23-37) "type": "input_audio", [](#%5F%5Fcodelineno-23-38) "input_audio": { [](#%5F%5Fcodelineno-23-39) "data": audio_base64, [](#%5F%5Fcodelineno-23-40) "format": "wav", [](#%5F%5Fcodelineno-23-41) }, [](#%5F%5Fcodelineno-23-42) "uuid": audio_url, # Optional [](#%5F%5Fcodelineno-23-43) }, [](#%5F%5Fcodelineno-23-44) ], [](#%5F%5Fcodelineno-23-45) }, [](#%5F%5Fcodelineno-23-46) ], [](#%5F%5Fcodelineno-23-47) model=model, [](#%5F%5Fcodelineno-23-48) max_completion_tokens=64, [](#%5F%5Fcodelineno-23-49)) [](#%5F%5Fcodelineno-23-50) [](#%5F%5Fcodelineno-23-51)result = chat_completion_from_base64.choices[0].message.content [](#%5F%5Fcodelineno-23-52)print("Chat completion output from input audio:", result)

Alternatively, you can pass audio_url, which is the audio counterpart of image_url for image input:

Code

[](#%5F%5Fcodelineno-24-1)chat_completion_from_url = client.chat.completions.create( [](#%5F%5Fcodelineno-24-2) messages=[ [](#%5F%5Fcodelineno-24-3) { [](#%5F%5Fcodelineno-24-4) "role": "user", [](#%5F%5Fcodelineno-24-5) "content": [ [](#%5F%5Fcodelineno-24-6) { [](#%5F%5Fcodelineno-24-7) "type": "text", [](#%5F%5Fcodelineno-24-8) "text": "What's in this audio?", [](#%5F%5Fcodelineno-24-9) }, [](#%5F%5Fcodelineno-24-10) { [](#%5F%5Fcodelineno-24-11) "type": "audio_url", [](#%5F%5Fcodelineno-24-12) "audio_url": {"url": audio_url}, [](#%5F%5Fcodelineno-24-13) "uuid": audio_url, # Optional [](#%5F%5Fcodelineno-24-14) }, [](#%5F%5Fcodelineno-24-15) ], [](#%5F%5Fcodelineno-24-16) } [](#%5F%5Fcodelineno-24-17) ], [](#%5F%5Fcodelineno-24-18) model=model, [](#%5F%5Fcodelineno-24-19) max_completion_tokens=64, [](#%5F%5Fcodelineno-24-20)) [](#%5F%5Fcodelineno-24-21) [](#%5F%5Fcodelineno-24-22)result = chat_completion_from_url.choices[0].message.content [](#%5F%5Fcodelineno-24-23)print("Chat completion output from audio url:", result)

Full example: examples/generate/multimodal/openai_chat_completion_client_for_multimodal.py

Note

By default, the timeout for fetching audios through HTTP URL is 10 seconds. You can override this by setting the environment variable:

[](#%5F%5Fcodelineno-25-1)export VLLM_AUDIO_FETCH_TIMEOUT=<timeout>

Embedding Inputs

To input pre-computed embeddings belonging to a data type (i.e. image, video, or audio) directly to the language model, pass a tensor of shape (..., hidden_size of LM) for each item to the corresponding field of the multi-modal dictionary.

Important

Unlike offline inference, the embeddings for each item must be passed separately in order for placeholder tokens to be applied correctly by the chat template.

You must enable this feature via the --enable-mm-embeds flag in vllm serve.

Warning

The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users!

Image Embedding Inputs

For image embeddings, you can pass the base64-encoded tensor to the image_embeds field. The following example demonstrates how to pass image embeddings to the OpenAI server:

Code

[](#%5F%5Fcodelineno-26-1)from vllm.utils.serial_utils import tensor2base64 [](#%5F%5Fcodelineno-26-2) [](#%5F%5Fcodelineno-26-3)client = OpenAI( [](#%5F%5Fcodelineno-26-4) # defaults to os.environ.get("OPENAI_API_KEY") [](#%5F%5Fcodelineno-26-5) api_key=openai_api_key, [](#%5F%5Fcodelineno-26-6) base_url=openai_api_base, [](#%5F%5Fcodelineno-26-7)) [](#%5F%5Fcodelineno-26-8) [](#%5F%5Fcodelineno-26-9)# Basic usage - this is equivalent to the LLaVA example for offline inference [](#%5F%5Fcodelineno-26-10)model = "llava-hf/llava-1.5-7b-hf" [](#%5F%5Fcodelineno-26-11)embeds = { [](#%5F%5Fcodelineno-26-12) "type": "image_embeds", [](#%5F%5Fcodelineno-26-13) "image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size) [](#%5F%5Fcodelineno-26-14) "uuid": image_url, # Optional [](#%5F%5Fcodelineno-26-15)} [](#%5F%5Fcodelineno-26-16) [](#%5F%5Fcodelineno-26-17) [](#%5F%5Fcodelineno-26-18)# Additional examples for models that require extra fields [](#%5F%5Fcodelineno-26-19)model = "Qwen/Qwen2-VL-2B-Instruct" [](#%5F%5Fcodelineno-26-20)embeds = { [](#%5F%5Fcodelineno-26-21) "type": "image_embeds", [](#%5F%5Fcodelineno-26-22) "image_embeds": { [](#%5F%5Fcodelineno-26-23) "image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size) [](#%5F%5Fcodelineno-26-24) "image_grid_thw": tensor2base64(torch.load(...)), # Shape: (3,) [](#%5F%5Fcodelineno-26-25) }, [](#%5F%5Fcodelineno-26-26) "uuid": image_url, # Optional [](#%5F%5Fcodelineno-26-27)} [](#%5F%5Fcodelineno-26-28) [](#%5F%5Fcodelineno-26-29)model = "openbmb/MiniCPM-V-2_6" [](#%5F%5Fcodelineno-26-30)embeds = { [](#%5F%5Fcodelineno-26-31) "type": "image_embeds", [](#%5F%5Fcodelineno-26-32) "image_embeds": { [](#%5F%5Fcodelineno-26-33) "image_embeds": tensor2base64(torch.load(...)), # Shape: (num_slices, hidden_size) [](#%5F%5Fcodelineno-26-34) "image_sizes": tensor2base64(torch.load(...)), # Shape: (2,) [](#%5F%5Fcodelineno-26-35) }, [](#%5F%5Fcodelineno-26-36) "uuid": image_url, # Optional [](#%5F%5Fcodelineno-26-37)} [](#%5F%5Fcodelineno-26-38) [](#%5F%5Fcodelineno-26-39)# Single image input [](#%5F%5Fcodelineno-26-40)chat_completion = client.chat.completions.create( [](#%5F%5Fcodelineno-26-41) messages=[ [](#%5F%5Fcodelineno-26-42) { [](#%5F%5Fcodelineno-26-43) "role": "system", [](#%5F%5Fcodelineno-26-44) "content": "You are a helpful assistant.", [](#%5F%5Fcodelineno-26-45) }, [](#%5F%5Fcodelineno-26-46) { [](#%5F%5Fcodelineno-26-47) "role": "user", [](#%5F%5Fcodelineno-26-48) "content": [ [](#%5F%5Fcodelineno-26-49) { [](#%5F%5Fcodelineno-26-50) "type": "text", [](#%5F%5Fcodelineno-26-51) "text": "What's in this image?", [](#%5F%5Fcodelineno-26-52) }, [](#%5F%5Fcodelineno-26-53) embeds, [](#%5F%5Fcodelineno-26-54) ], [](#%5F%5Fcodelineno-26-55) }, [](#%5F%5Fcodelineno-26-56) ], [](#%5F%5Fcodelineno-26-57) model=model, [](#%5F%5Fcodelineno-26-58)) [](#%5F%5Fcodelineno-26-59) [](#%5F%5Fcodelineno-26-60)# Multi image input [](#%5F%5Fcodelineno-26-61)chat_completion = client.chat.completions.create( [](#%5F%5Fcodelineno-26-62) messages=[ [](#%5F%5Fcodelineno-26-63) { [](#%5F%5Fcodelineno-26-64) "role": "system", [](#%5F%5Fcodelineno-26-65) "content": "You are a helpful assistant.", [](#%5F%5Fcodelineno-26-66) }, [](#%5F%5Fcodelineno-26-67) { [](#%5F%5Fcodelineno-26-68) "role": "user", [](#%5F%5Fcodelineno-26-69) "content": [ [](#%5F%5Fcodelineno-26-70) { [](#%5F%5Fcodelineno-26-71) "type": "text", [](#%5F%5Fcodelineno-26-72) "text": "What's in this image?", [](#%5F%5Fcodelineno-26-73) }, [](#%5F%5Fcodelineno-26-74) embeds, [](#%5F%5Fcodelineno-26-75) embeds, [](#%5F%5Fcodelineno-26-76) ], [](#%5F%5Fcodelineno-26-77) }, [](#%5F%5Fcodelineno-26-78) ], [](#%5F%5Fcodelineno-26-79) model=model, [](#%5F%5Fcodelineno-26-80)) [](#%5F%5Fcodelineno-26-81) [](#%5F%5Fcodelineno-26-82)# Multi image input (interleaved) [](#%5F%5Fcodelineno-26-83)chat_completion = client.chat.completions.create( [](#%5F%5Fcodelineno-26-84) messages=[ [](#%5F%5Fcodelineno-26-85) { [](#%5F%5Fcodelineno-26-86) "role": "system", [](#%5F%5Fcodelineno-26-87) "content": "You are a helpful assistant.", [](#%5F%5Fcodelineno-26-88) }, [](#%5F%5Fcodelineno-26-89) { [](#%5F%5Fcodelineno-26-90) "role": "user", [](#%5F%5Fcodelineno-26-91) "content": [ [](#%5F%5Fcodelineno-26-92) embeds, [](#%5F%5Fcodelineno-26-93) { [](#%5F%5Fcodelineno-26-94) "type": "text", [](#%5F%5Fcodelineno-26-95) "text": "What's in this image?", [](#%5F%5Fcodelineno-26-96) }, [](#%5F%5Fcodelineno-26-97) embeds, [](#%5F%5Fcodelineno-26-98) ], [](#%5F%5Fcodelineno-26-99) }, [](#%5F%5Fcodelineno-26-100) ], [](#%5F%5Fcodelineno-26-101) model=model, [](#%5F%5Fcodelineno-26-102))

Cached Inputs

Just like with offline inference, you can skip sending media if you expect cache hits with provided UUIDs. You can do so by sending media like this:

Code

[](#%5F%5Fcodelineno-27-1) # Image/video/audio URL: [](#%5F%5Fcodelineno-27-2) { [](#%5F%5Fcodelineno-27-3) "type": "image_url", [](#%5F%5Fcodelineno-27-4) "image_url": None, [](#%5F%5Fcodelineno-27-5) "uuid": image_uuid, [](#%5F%5Fcodelineno-27-6) }, [](#%5F%5Fcodelineno-27-7) [](#%5F%5Fcodelineno-27-8) # image_embeds [](#%5F%5Fcodelineno-27-9) { [](#%5F%5Fcodelineno-27-10) "type": "image_embeds", [](#%5F%5Fcodelineno-27-11) "image_embeds": None, [](#%5F%5Fcodelineno-27-12) "uuid": image_uuid, [](#%5F%5Fcodelineno-27-13) }, [](#%5F%5Fcodelineno-27-14) [](#%5F%5Fcodelineno-27-15) # input_audio: [](#%5F%5Fcodelineno-27-16) { [](#%5F%5Fcodelineno-27-17) "type": "input_audio", [](#%5F%5Fcodelineno-27-18) "input_audio": None, [](#%5F%5Fcodelineno-27-19) "uuid": audio_uuid, [](#%5F%5Fcodelineno-27-20) }, [](#%5F%5Fcodelineno-27-21) [](#%5F%5Fcodelineno-27-22) # PIL Image: [](#%5F%5Fcodelineno-27-23) { [](#%5F%5Fcodelineno-27-24) "type": "image_pil", [](#%5F%5Fcodelineno-27-25) "image_pil": None, [](#%5F%5Fcodelineno-27-26) "uuid": image_uuid, [](#%5F%5Fcodelineno-27-27) },