2.2.3-Backend&colon-vLLM.md

vLLM

Handle: vllm
URL: http://localhost:33911

A high-throughput and memory-efficient inference and serving engine for LLMs

Starting

# [Optional] pre-build the vLLM image
harbor build vllm

# Start the vLLM service
harbor up vllm

• Harbor builds custom vllm image with bitsandbytes
• vllm will be connected to webui, aider, boost, chatui and some other services when running together
• Official docker images require specific CUDA versions - beware
• When ROCm is detected, Harbor mounts AMD devices and starts vLLM explicitly via python3 -m vllm.entrypoints.openai.api_server, which supports AMD ROCm images that do not provide an OpenAI-server entrypoint

Integrations

vllm is a high-throughput OpenAI-compatible inference server (listening on port 8000 inside the container) that many Harbor frontends and satellite tools auto-discover and use when started together. Wiring is performed exclusively through the cross-compose overlay files (compose.x.*.vllm.yml and a few always-on config mergers).

Frontends and tools that auto-configure for vllm

• webui — mounts ./services/webui/configs/config.vllm.json (registers OpenAI provider at http://vllm:8000/v1 with key sk-vllm).
• chatui — mounts ./services/chatui/configs/chatui.vllm.yml (adds a MODELS entry for vllm using the same internal URL/key).
• aider — mounts ./services/aider/configs/aider.vllm.yml (OpenAI base + key; model chosen via HARBOR_AIDER_MODEL).
• boost — injects HARBOR_BOOST_OPENAI_URL_VLLM and HARBOR_BOOST_OPENAI_KEY_VLLM so boost can target vllm as one of its backends.
• litellm — mounts ./services/litellm/litellm.vllm.yaml (adds a model entry under the vllm name; the example model can be changed by the user).
• opint (Open Interpreter) — completely overrides the container entrypoint to pass --api_base http://vllm:8000/v1 --api_key sk-vllm plus the user command.
• optillm — sets OPTILLM_BASE_URL=http://vllm:8000/v1 and the matching API key.
• raglite — sets VLLM_BASE_URL=http://vllm:8000 (note: no /v1 suffix for this client).
• astrbot — sets HARBOR_ASTRBOT_VLLM_ENDPOINT (comment in the overlay also reminds users they can configure it in the AstrBot WebUI).
• cognee — declares depends_on: [vllm], forces LLM_PROVIDER=openai, and wires LLM_ENDPOINT, LLM_API_KEY, plus LLM_MODEL=${HARBOR_VLLM_MODEL}.
• mindsdb — sets LLM_FUNCTION_BASE_URL=http://vllm:8000/v1 and the OpenAI key so MindsDB functions can call the model.
• mi — depends_on: [vllm], sets OPENAI_BASE_URL=http://vllm:8000 (no /v1) + OPENAI_API_KEY and MODEL=${HARBOR_MI_VLLM_MODEL} (plus a second overlay for the harbor mi variant that supplies a default model).
• ml-intern — depends_on: [vllm], supplies both VLLM_* and LOCAL_LLM_* pairs plus ML_INTERN_MODEL=vllm/${HARBOR_ML_INTERN_VLLM_MODEL}.
• npcsh — depends_on: [vllm], configures a large set of *MODEL / *PROVIDER variables (all pointing at the same vllm model via HARBOR_NPCSH_VLLM_MODEL) together with multiple OpenAI-compatible base URLs and the key.
• open-design — depends_on: [vllm], sets protocol, base URL, key and model defaults using the HARBOR_OPEN_DESIGN_VLLM_MODEL and HARBOR_OPEN_DESIGN_* family.
• aichat — the custom start_aichat.sh + yaml merger always concatenates every aichat.*.yml file present in the image (including aichat.vllm.yml). This gives aichat an openai-compatible client for vllm (using ${HARBOR_AICHAT_MODEL}) whenever the vllm service is reachable on the network.

Reverse proxy / external access

• When traefik is also running, compose.x.traefik.vllm.yml (auto-generated) attaches the standard labels so vllm becomes available at the public URL https://vllm.${HARBOR_TRAEFIK_DOMAIN} on the websecure entrypoint. It joins the traefik-public network in addition to the internal harbor-network.

Hardware acceleration (GPU / accelerator)

• NVIDIA: compose.x.vllm.nvidia.yml requests count: all GPUs with the nvidia driver and gpu capability.
• ROCm / AMD: compose.x.vllm.rocm.yml mounts the two device nodes and forces the explicit python3 -m vllm.entrypoints.openai.api_server entrypoint (required for some ROCm images).
• CDI (modern container device interface): compose.x.vllm.cdi.yml (generated) uses the cdi driver with nvidia.com/gpu=all.
• The image itself is always produced by the local services/vllm/Dockerfile (which does pip install bitsandbytes on top of whatever HARBOR_VLLM_IMAGE:HARBOR_VLLM_VERSION you have chosen). GPU overlays are applied after the build step.

Source builds

Running harbor build vllm (recommended before first GPU use or when you change the version) executes the local Dockerfile. The resulting image carries the bitsandbytes addition and is tagged the same way the official image would have been, so every consumer overlay and GPU compose file continues to work without modification.

Host / volume interactions

• Model weights (Hugging Face cache): ${HARBOR_HF_CACHE:-./services/vllm/.hf-cache} on the host is bind-mounted to /root/.cache/huggingface inside the container. All harbor hf download, harbor pull, and harbor vllm model commands populate this cache; vLLM reads its models directly from it.
• vLLM runtime cache: ${HARBOR_VLLM_CACHE} → /root/.cache/vllm (stores compiled CUDA kernels, prefix caches, etc.).
• The same vLLM cache volume is also mounted by ollama, localai, sglang and kobold (shared cache directory convention).
• ipc: host gives the container direct access to host IPC namespace — required for optimal multi-GPU and shared-memory performance with vLLM.
• No persistent ./services/vllm/data volume (vLLM loads a single model via the specifier or extra args; multi-model serving is usually done via an external router such as litellm or boost).

When you execute a command such as harbor up vllm webui chatui aider boost cognee the cross-compose machinery guarantees that every listed frontend immediately sees a working vLLM backend at the internal address with the conventional key, and any GPU/ROCm/CDI requirements are satisfied automatically.

Models

• Official vLLM models
• Quantization kernels

Once you've found a model you want to run, you can configure it with Harbor:

# Quickly lookup some of the compatible quants
harbor hf find awq
harbor hf find gptq

# This propagates the settings
# to the relevant configuration files
harbor vllm model Qwen/Qwen3.5-4B

# To run a gated model, ensure that you've
# also set your Huggingface API Token
harbor hf token <your-token>

Configuration

• Engine CLI Arguments
• Environment Variables

You can configure specific portions of vllm via Harbor CLI:

# See original CLI help
harbor run vllm --help

# Get/Set the extra arguments
harbor vllm args
harbor vllm args '--dtype bfloat16 --code-revision 3.5'

harbor config set vllm.host.port 4090

Version and update

# Get/set desired vLLM version
harbor vllm version
# Command accepts a docker tag
harbor vllm version latest
# Customize docker image
harbor config set vllm.image custom/vllm
# Force-pull new version of the base image
# if you have set version to "latest"
docker pull $(harbor config get vllm.image):$(harbor config get vllm.version)

ROCm / AMD

Use AMD ROCm images instead of CUDA images when running on AMD hardware. Pin to an image that supports your GPU architecture when possible. Harbor includes a ROCm compose override that mounts /dev/kfd and /dev/dri and starts vLLM explicitly with the OpenAI API server entrypoint; some AMD images do not provide that entrypoint themselves.

# Example for AMD Strix Halo / gfx1151
harbor config set vllm.image rocm/vllm-dev
harbor vllm version preview_v0.20.0_20260429
harbor build vllm

Some newer multimodal Qwen checkpoints need text-only mode for chat inference to avoid multimodal warmup/profile memory spikes:

harbor vllm model Qwen/Qwen3.6-35B-A3B
harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype half --language-model-only --attention-backend TRITON_ATTN'

If graph capture fails, add --enforce-eager as a fallback. If it succeeds, leaving eager mode off is usually faster.

AMD Strix Halo / Radeon 8060S / gfx1151 notes

Strix Halo works with vLLM, but it is not yet a normal upstream ROCm experience. Treat vLLM on this hardware as a throughput-oriented, version-sensitive setup rather than the fastest single-user chat backend.

Key learnings from local testing:

• rocm/vllm:latest may be unusable on gfx1151; a tiny PyTorch GPU operation segfaulted in testing.
• rocm/vllm-dev:preview_v0.20.0_20260429 worked for Qwen/Qwen3.6-35B-A3B HF safetensors.
• vLLM does not currently load the Qwen3.6 GGUF path used by llama.cpp: qwen35moe GGUF is not supported.
• ROCm AITER MoE did not work on gfx1151 in the tested image; use Triton/auto MoE fallback unless you are using a Strix-specific patched stack.
• Single-request latency is still behind llama.cpp GGUF. vLLM becomes more interesting for concurrent/batched OpenAI-compatible serving.

A known Strix-specific stack is kyuz0/vllm-therock-gfx1151, built on TheRock ROCm nightlies with gfx1151 patches. It can be run outside Harbor for comparison:

docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/vllm-strix:/root/.cache/vllm \
  kyuz0/vllm-therock-gfx1151:stable \
  vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 1 \
    --dtype auto \
    --trust-remote-code \
    --language-model-only \
    --attention-backend TRITON_ATTN \
    --mm-encoder-attn-backend TRITON_ATTN

For better vLLM single-stream speed on Strix Halo, an AWQ checkpoint was faster than BF16 in local testing:

docker run --rm -it \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  --security-opt seccomp=unconfined \
  --ipc=host \
  -p 8000:8000 \
  -e VLLM_USE_TRITON_AWQ=1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ~/.cache/vllm-strix-awq:/root/.cache/vllm \
  kyuz0/vllm-therock-gfx1151:stable \
  vllm serve cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --max-num-seqs 8 \
    --dtype auto \
    --trust-remote-code \
    --language-model-only \
    --attention-backend TRITON_ATTN \
    --mm-encoder-attn-backend TRITON_ATTN

Measured locally on a Strix Halo / Radeon 8060S system:

Setup	Single request	Concurrent aggregate
rocm/vllm-dev + Qwen3.6 BF16	~15 tok/s	not measured
rocm/vllm-dev + Qwen3.6 BF16 + MTP	~21-22 tok/s	not measured
kyuz0/vllm-therock-gfx1151 + Qwen3.6 BF16	~14-20 tok/s, depending on MTP	not measured
kyuz0/vllm-therock-gfx1151 + Qwen3.6 AWQ 4-bit	~25-26 tok/s	~53 tok/s at concurrency 8
llama.cpp + Qwen3.6 Q8_0 GGUF on Vulkan/RADV	~48-49 tok/s	not measured
llama.cpp + Qwen3.6 Q4_0-class GGUF on Vulkan/RADV	~75 tok/s	not measured

For interactive local chat on Strix Halo, prefer llama.cpp. Use vLLM when you need its OpenAI-compatible server, batching, multi-client concurrency, or vLLM-specific model support.

Speculative decoding / MTP

Speculative decoding is configured through vLLM's JSON-style --speculative-config flags. It is model- and backend-sensitive; benchmark before making it your default.

For checkpoints that include MTP weights, such as Qwen/Qwen3.6-35B-A3B, the target model can draft its own next token:

harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype half --language-model-only --attention-backend TRITON_ATTN --max-num-seqs 256 --speculative-config.method mtp --speculative-config.num_speculative_tokens 1'

Draft models must use the same tokenizer vocabulary as the target model. For example, Qwen/Qwen3-0.6B is not a compatible draft for Qwen/Qwen3.6-35B-A3B because the vocabulary sizes differ. Qwen/Qwen3.5-0.8B has a matching vocabulary, but vLLM 0.20 currently treats Qwen3.5 draft models as MTP modules on this path and fails to load them as ordinary draft models when their hidden size differs from the target.

DFlash is another option when a matching drafter exists. On ROCm, do not force TRITON_ATTN for DFlash, and use bfloat16 to avoid drafter dtype mismatches:

harbor vllm args '--max-model-len 4096 --gpu-memory-utilization 0.80 --dtype bfloat16 --language-model-only --max-num-seqs 256 --trust-remote-code --speculative-config.method dflash --speculative-config.model z-lab/Qwen3.6-35B-A3B-DFlash --speculative-config.num_speculative_tokens 8 --max-num-batched-tokens 32768'

ngram speculative decoding does not require an extra model, but it mainly helps prompt-copying/repetitive workloads and may be slower for general chat.

You can specify more options directly in services/vllm/override.env.

Following options can be set via harbor config:

# The port on the host machine where the vLLM service
# will be available
HARBOR_VLLM_HOST_PORT          33911

# Docker image to use for vLLM
HARBOR_VLLM_IMAGE              vllm/vllm-openai

# Docker image tag to use for vLLM
HARBOR_VLLM_VERSION            v0.8.5

# Default model to load on startup
HARBOR_VLLM_MODEL              Qwen/Qwen3.5-4B

VRAM

Below are some steps to take if running out of VRAM (no magic, though).

Limit Context Length

You can limit the context length to reduce the memory footprint. This can be done via the --max-model-len flag.

harbor vllm args --max-model-len 2048

Quantization

vLLM supports many different quantization formats. You would typically configure this via --load-format and --quantization flags. For example:

harbor vllm args --load-format bitsandbytes --quantization bitsandbytes

Offloading

vLLM supports partial offloading to the CPU, similar to llama.cpp and some other backends. This can be configured via the --cpu-offload-gb flag.

harbor vllm args --cpu-offload-gb 4

Disable CUDA Graphs

When loading the model, VRAM usage can spike when computing the CUDA graphs. This can be disabled via --enforce-eager flag.

harbor vllm args --enforce-eager

GPU Memory Utilization

Reduce the amount of VRAM allocated for the model executor. Can be ranged from 0 to 1.0, 0.9 by default.

harbor vllm args --gpu-memory-utilization 0

Run on CPU

You can move to CPU by setting the --device cpu flag.

harbor vllm args --device cpu