dynamo/docs/features/multimodal/multimodal-vllm.md at main · nvyutwu/dynamo

title	vLLM Multimodal

This document provides a comprehensive guide for multimodal inference using the vLLM backend in Dynamo.

**Security Requirement**: All multimodal workers require the `--enable-multimodal` flag to be explicitly set at startup. This prevents unintended processing of multimodal data from untrusted sources. Media requests are rejected when the flag is absent, and workers configured with a multimodal role fail at startup. This flag is analogous to `--enable-mm-embeds` in vLLM serve but also extends it to all multimodal content (URL, embeddings, and base64 data).

Support Matrix

Modality	Aggregated	P/D	Separate encode worker
Image	Yes	Yes	Legacy entry point only
Video	Yes	Yes	Processed by the language-model worker
Audio	Yes	Yes, with decode reload	Not routed to the separate encoder

Supported URL Formats

Format	Example	Description
HTTP/HTTPS	`http://example.com/image.jpg`	Remote media files
Data URL	`data:image/jpeg;base64,/9j/4AAQ...`	Base64-encoded inline data

Deployment Patterns

The main multimodal vLLM launchers in this repo are:

Pattern	Device	Launch script	Unified selection	Best for
Aggregated	CUDA	`agg_multimodal.sh`	`--unified`	Simplest image/video serving from one worker
Aggregated	XPU	`xpu/agg_multimodal_xpu.sh`	No	Image/video serving on XPU devices
P/D	CUDA	`disagg_multimodal_p_d.sh`	`--unified`	Prefill/decode separation without a dedicated encoder
E/PD (Encode + PD)	CUDA	`disagg_multimodal_e_pd.sh`	No	Separate encoder and embedding-cache workflows
E/P/D (Full Disaggregation)	CUDA	`disagg_multimodal_epd.sh`	No	Separate encode, prefill, and decode workers

Image/Video Serving

Dynamo supports multimodal image and video requests for Vision Language Models (VLMs). Qwen/Qwen3-VL-2B-Instruct is a good example because the same model can handle both image_url and video_url requests through the standard OpenAI chat endpoint.

Aggregated Serving

Use the single-worker aggregated launcher for the simplest image/video setup:

cd $DYNAMO_HOME/examples/backends/vllm

# GPU deployment
bash launch/agg_multimodal.sh --model Qwen/Qwen3-VL-2B-Instruct

# Unified backend
bash launch/agg_multimodal.sh --unified --model Qwen/Qwen3-VL-2B-Instruct

# XPU deployment
bash launch/xpu/agg_multimodal_xpu.sh --model Qwen/Qwen3-VL-2B-Instruct

Image request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "Qwen/Qwen3-VL-2B-Instruct",
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "What is in this image?"
            },
            {
              "type": "image_url",
              "image_url": {
                "url": "http://images.cocodataset.org/test2017/000000155781.jpg"
              }
            }
          ]
        }
      ],
      "max_tokens": 64,
      "temperature": 0.0,
      "stream": false
    }'

Video request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "Qwen/Qwen3-VL-2B-Instruct",
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "Describe the video in detail"
            },
            {
              "type": "video_url",
              "video_url": {
                "url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-Omni/demo/draw.mp4"
              }
            }
          ]
        }
      ],
      "max_tokens": 64,
      "stream": false
    }' | jq

P/D Serving

Use the P/D launcher to separate prefill and decode without deploying a dedicated multimodal encoder:

cd $DYNAMO_HOME/examples/backends/vllm

# Legacy vLLM worker path
bash launch/disagg_multimodal_p_d.sh --model Qwen/Qwen3-VL-2B-Instruct

# Unified vLLM worker path
bash launch/disagg_multimodal_p_d.sh --unified \
  --model Qwen/Qwen3-VL-2B-Instruct

For Qwen-VL images, prefill sends grid and embedding-shape metadata so decode can construct schema-valid placeholder embeddings and initialize mRoPE. Other model families use the expanded prompt token IDs produced during prefill.

The P/D handoff does not carry video embeddings. Video and audio inputs are loaded again on the decode worker. This preserves current behavior but adds media download and processing work. Mixed image-and-video P/D requests retain the same model-specific limitations as the legacy vLLM path.

Unified vLLM Backend

Pass --unified to the aggregated or P/D launchers to run python -m dynamo.vllm.unified_main. The unified path supports HTTP URLs, data URLs, frontend-decoded images, mm_processor_kwargs, frontend-provided multimodal hashes, and Kimi-style vision_chunk inputs.

The Python vLLM frontend can pre-render multimodal processor inputs and send them to an aggregated unified worker. Shared memory is the same-node default; NIXL supports the transfer channel used by cross-node deployments:

# Same-node shared-memory transfer
DYN_CHAT_PROCESSOR=vllm DYNAMO_MM_TRANSFER=shm \
  bash launch/agg_multimodal.sh --unified \
  --model Qwen/Qwen3-VL-2B-Instruct

# NIXL transfer
DYN_CHAT_PROCESSOR=vllm DYNAMO_MM_TRANSFER=nixl \
  bash launch/agg_multimodal.sh --unified \
  --model Qwen/Qwen3-VL-2B-Instruct

The frontend includes the original media references when transfer preparation is unavailable or partial. Fully transferred requests omit those references to avoid duplicating large inline data URIs in the backend payload. A receiver-side failure after a full transfer does not currently have a raw-media fallback.

P/D prefill deliberately uses the original media because it still needs raw-media-derived metadata for the decode handoff.

The unified vLLM entry point does not provide a separate Encode worker and rejects both `--disaggregation-mode encode` and `--route-to-encoder`. Use the legacy E/PD or E/P/D launchers when a dedicated encoder is required.

E/PD Serving (Encode + PD)

Use disagg_multimodal_e_pd.sh when you want a separate encode worker and a combined prefill/decode worker. This path is primarily useful for image-centric workloads and embedding-cache experiments.

When a separate encode worker is deployed with the current vLLM path, only `image_url` inputs are routed to it. `video_url` inputs are still processed on the combined PD worker.

cd $DYNAMO_HOME/examples/backends/vllm

# Multi-GPU deployment
bash launch/disagg_multimodal_e_pd.sh --model Qwen/Qwen3-VL-2B-Instruct

# Single-GPU (functional testing with small models)
bash launch/disagg_multimodal_e_pd.sh --model Qwen/Qwen3-VL-2B-Instruct --single-gpu

E/P/D Serving (Full Disaggregation)

Use disagg_multimodal_epd.sh when you want separate encode, prefill, and decode workers for multimodal workloads.

In the current vLLM implementation, the separate encode worker is only used for `image_url` inputs. `video_url` inputs are still processed on the prefill worker, not on the encode worker.

cd $DYNAMO_HOME/examples/backends/vllm

# Multi-GPU deployment
bash launch/disagg_multimodal_epd.sh --model Qwen/Qwen3-VL-2B-Instruct

# Single-GPU (functional testing with small models)
bash launch/disagg_multimodal_epd.sh --model Qwen/Qwen3-VL-2B-Instruct --single-gpu

Audio Serving

Dynamo supports audio_url requests for audio-capable models. Audio is loaded by the backend worker via vLLM's AudioMediaIO at native sample rate — vLLM's model-specific processor handles resampling and feature extraction internally. Omni models can handle image_url, video_url, and audio_url in the same request.

Aggregated Serving

Use the same aggregated multimodal launcher with an audio-capable model:

pip install 'vllm[audio]'  # installs librosa and other audio dependencies
cd $DYNAMO_HOME/examples/backends/vllm

# GPU deployment
bash launch/agg_multimodal.sh --model Qwen/Qwen3-Omni-30B-A3B-Instruct

# XPU deployment
DYN_CHAT_PROCESSOR=vllm \
  bash launch/xpu/agg_multimodal_xpu.sh --model Qwen/Qwen3-Omni-30B-A3B-Instruct

flowchart LR
  HTTP --> frontend
  frontend --> HTTP
  frontend --audio_url--> vllm_worker
  vllm_worker --> frontend

Audio request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
      "model": "Qwen/Qwen3-Omni-30B-A3B-Instruct",
      "messages": [
        {
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "What sound is this?"
            },
            {
              "type": "audio_url",
              "audio_url": {
                "url": "https://raw.githubusercontent.com/yuekaizhang/Triton-ASR-Client/main/datasets/mini_en/wav/1221-135766-0002.wav"
              }
            }
          ]
        }
      ],
      "max_tokens": 100,
      "stream": false
    }' | jq

Embedding Cache

Dynamo supports embedding cache in both aggregated and disaggregated settings:

Setting	Implementation	Launch Script
Aggregated	Supported via vLLM ECConnector in vLLM 0.17+	`agg_multimodal.sh` (or with `vllm serve` directly)
Disaggregated encoder	Dynamo-managed cache in the worker layer on top of vLLM engine	`disagg_multimodal_e_pd.sh`

Aggregated Worker

A single vLLM instance caches encoded embeddings on CPU so repeated images skip encoding entirely. Supported natively with vLLM 0.17+.

---
title: Embedding Cache — Aggregated Encoder (e.g. aggregated EP or EPD node)
---
flowchart LR
  req[Multimodal Request] --> gpu{GPU Encoder Cache<br/>hit?}
  gpu -- yes --> skip[Use cached GPU embedding<br/>no encoder, no connector]
  gpu -- no --> cpu{CPU Embedding Cache<br/>hit?}
  cpu -- yes --> load[Load: CPU → GPU<br/>skip encoder]
  cpu -- no --> encode[Run Encoder]
  encode -- save: GPU → CPU --> store[(CPU Embedding Cache<br/>LRU)]

Launch with Dynamo:

bash examples/backends/vllm/launch/agg_multimodal.sh \
    --unified \
    --model Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
    --multimodal-embedding-cache-capacity-gb 10

Both dynamo.vllm and dynamo.vllm.unified_main automatically configure ec_both mode with DynamoMultimodalEmbeddingCacheConnector when capacity is greater than zero. A capacity of zero disables the CPU cache. Frontend-provided multimodal hashes are reused as cache identities so routing and embedding-cache lookups agree.

Launch with vllm serve (standalone, no Dynamo):

vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
    --ec-transfer-config "{
        \"ec_role\": \"ec_both\",
        \"ec_connector\": \"DynamoMultimodalEmbeddingCacheConnector\",
        \"ec_connector_module_path\": \"dynamo.vllm.multimodal_utils.multimodal_embedding_cache_connector\",
        \"ec_connector_extra_config\": {\"multimodal_embedding_cache_capacity_gb\": 10}
    }"

The multimodal_embedding_cache_capacity_gb parameter controls the CPU-side LRU cache size in GB (0 = disabled). Requires vLLM 0.17+.

Disaggregated Encoder (Embedding Cache in Prefill Worker)

In the disaggregated setting, the Prefill Worker (P) owns a CPU-side LRU embedding cache (EmbeddingCacheManager). On each request P checks the cache first — on a hit, the Encode Worker is skipped entirely. On a miss, P routes to the Encode Worker (E), receives embeddings via NIXL, saves them to the cache, and then feeds the embeddings along with the request into the vLLM Instance for prefill.

---
title: Embedding Cache — Disaggregated Encoder
---
flowchart LR
    req[Request] --> cpu_check{"CPU cache hit?<br/>(EmbeddingCacheManager)"}

    subgraph P ["Prefill Worker (P)"]
        cpu_check -. hit .-> use[Use cached embedding]
        use --> vllm[vLLM Instance]
    end

    cpu_check -- miss --> E["Encode Worker (E)"]
    E -- "embeddings via NIXL" --> save["Save to cache"]
    save --> vllm

Launch:

cd $DYNAMO_HOME/examples/backends/vllm
bash launch/disagg_multimodal_e_pd.sh --multimodal-embedding-cache-capacity-gb 10

Client: Use the same image_url request format shown in Aggregated Serving.

LoRA Adapters on Multimodal Workers

Multimodal workers support dynamic loading and unloading of LoRA adapters at runtime via the management API. This enables serving fine-tuned multimodal models alongside the base model.

Loading a LoRA Adapter

Load an adapter on a running multimodal worker via the load_lora endpoint:

# For components workers (URI-based, requires DYN_LORA_ENABLED=true)
curl -X POST http://<worker-host>:<port>/load_lora \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "my-vlm-adapter",
    "source": {"uri": "s3://my-bucket/adapters/my-vlm-adapter"}
  }'

# For example workers (path-based)
curl -X POST http://<worker-host>:<port>/load_lora \
  -H "Content-Type: application/json" \
  -d '{
    "lora_name": "my-vlm-adapter",
    "lora_path": "/path/to/adapter"
  }'

Sending Requests with a LoRA

Set the model field in the request to the LoRA adapter name:

curl -X POST http://<frontend-host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my-vlm-adapter",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "Describe this image"},
        {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}}
      ]}
    ]
  }'

Requests without a LoRA name (or with the base model name) will use the base model.

Unloading a LoRA Adapter

curl -X POST http://<worker-host>:<port>/unload_lora \
  -H "Content-Type: application/json" \
  -d '{"lora_name": "my-vlm-adapter"}'

Listing Loaded Adapters

curl -X POST http://<worker-host>:<port>/list_loras

Disaggregated Mode

In disaggregated (prefill/decode) deployments, the same LoRA adapter must be loaded on both the prefill and decode workers. The LoRA identity (model field) is automatically propagated from the prefill worker to the decode worker in the forwarded request.

# Load on prefill worker
curl -X POST http://<prefill-worker>/load_lora \
  -d '{"lora_name": "my-adapter", "source": {"uri": "s3://bucket/adapter"}}'

# Load on decode worker (same adapter)
curl -X POST http://<decode-worker>/load_lora \
  -d '{"lora_name": "my-adapter", "source": {"uri": "s3://bucket/adapter"}}'

If a LoRA is loaded on the prefill worker but not on the decode worker, the decode worker will fall back to the base model for that request.

Supported Models

For a list of multimodal models supported by vLLM, see vLLM Supported Multimodal Models. Models listed there should generally work with aggregated serving, though they may not all be explicitly tested in this repo.

FilesExpand file tree

multimodal-vllm.md

Latest commit

History

multimodal-vllm.md

File metadata and controls

Support Matrix

Supported URL Formats

Deployment Patterns

Image/Video Serving

Aggregated Serving

P/D Serving

Unified vLLM Backend

E/PD Serving (Encode + PD)

E/P/D Serving (Full Disaggregation)

Audio Serving

Aggregated Serving

Embedding Cache

Aggregated Worker

Disaggregated Encoder (Embedding Cache in Prefill Worker)

LoRA Adapters on Multimodal Workers

Loading a LoRA Adapter

Sending Requests with a LoRA

Unloading a LoRA Adapter

Listing Loaded Adapters

Disaggregated Mode

Supported Models