Skip to content

README.md

Latest commit

 

History

History
136 lines (102 loc) · 5.79 KB

README.md

File metadata and controls

136 lines (102 loc) · 5.79 KB
Raw
title vLLM
subtitle vLLM engines run in Dynamo's distributed runtime with disaggregated serving, NIXL KV transfer, and KV-aware routing.

Dynamo vLLM integrates vLLM engines into Dynamo's distributed runtime, enabling disaggregated serving, KV-aware routing, and request cancellation while maintaining full compatibility with vLLM's native engine arguments. Dynamo leverages vLLM's native KV cache events, NIXL-based transfer mechanisms, and metric reporting to enable KV-aware routing and P/D disaggregation.

Installation

Install Latest Release

We recommend using uv to install:

uv venv --python 3.12 --seed
uv pip install "ai-dynamo[vllm]"

This installs Dynamo with the compatible vLLM version.

Container

We have public images available on NGC Catalog:

docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version>
./container/run.sh -it --framework VLLM --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:<version>
python container/render.py --framework vllm --output-short-filename
docker build -f container/rendered.Dockerfile -t dynamo:latest-vllm .
./container/run.sh -it --framework VLLM [--mount-workspace]

Development Setup

For development, use the devcontainer which has all dependencies pre-installed.

Feature Support Matrix

Feature Status Notes
Disaggregated Serving ✅ Prefill/decode separation with NIXL KV transfer
KV-Aware Routing ✅
SLA-Based Planner ✅
KVBM ✅
LMCache ✅ CUDA 12.9 and arm64/aarch64 containers may require building LMCache from source
FlexKV ✅
Multimodal Support ✅ Aggregated and P/D image/video serving on legacy and unified Python backends; separate Encode workers use the legacy path
Observability ✅ Metrics and monitoring
WideEP ✅ Support for DeepEP
DP Rank Routing ✅ Hybrid load balancing via external DP rank control
LoRA ✅ Dynamic loading/unloading from S3-compatible storage
GB200 Support ✅ Container functional on main

Quick Start

Start infrastructure services for local development:

docker compose -f dev/docker-compose.yml up -d

Launch an aggregated serving deployment:

cd $DYNAMO_HOME/examples/backends/vllm
bash launch/agg.sh

Running launch scripts standalone. The launch/*.sh scripts expect etcd and NATS to be reachable on localhost. Bring them up first (run from the repo root, or use the absolute path shown):

docker compose -f "$DYNAMO_HOME/dev/docker-compose.yml" up -d

Then run the launch script. Without these, workers register but the frontend cannot discover them and requests hang.

Rust Backend Preview

The Python vLLM backend remains the recommended entry point for production deployments and examples. The Rust backend is a development preview for validating the Rust LLMEngine integration with vLLM's engine-core client. Use it when working on the Rust backend contract, cancellation, metrics, or P/D wiring; use python -m dynamo.vllm or python -m dynamo.vllm.unified_main for the most complete vLLM feature coverage.

Note

The Rust backend depends on vLLM's engine-core crates, which are not yet published to crates.io and are pulled as git dependencies. They are gated behind the off-by-default vllm_rs cargo feature, so the default workspace build does not require the git sources and the crate is excluded from the published Dynamo crates. You must pass --features vllm_rs to build or run it.

To run the Rust backend locally, start the same infrastructure services and frontend, then launch the Rust worker in another terminal:

docker compose -f dev/docker-compose.yml up -d

python -m dynamo.frontend --http-port 8000
DYN_SYSTEM_PORT=8081 cargo run -p dynamo-vllm-rs-backend --features vllm_rs -- Qwen/Qwen3-0.6B -- \
  --enforce-eager \
  --max-model-len 4096

The Rust worker starts a managed vLLM engine-core process and registers with the Dynamo frontend using the same discovery path as the Python unified backend. The Rust backend is expected to become the default only after it reaches feature and operational parity with the Python vLLM backend.

Next Steps