[Bug] Warmup memory miscalculation causes OOM with hybrid GDN models

[biondogs]

Bug Description

The warmup process miscalculates KV cache memory for hybrid GDN (GatedDeltaNet) + FullAttention models, causing OOM during graph compilation. The warmup graph compilation for 300+ buckets requires ~20-30 GiB peak, but the KV cache for hybrid models with long context (262K) consumes most of the available HBM, leaving insufficient memory for warmup graphs.

Environment

• Image: vault.habana.ai/gaudi-docker/1.24.1/ubuntu24.04/habanalabs/vllm-0.24.0-ptupstream-2.11.0:latest (also tested on v0.26.0)
• vLLM version: v0.24.0 and v0.26.0
• Gaudi software: v1.24.1 (PyTorch 2.11)
• Model: bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 (hybrid: FullAttention + GDN layers)
• Configuration: TP=4, max-model-len=262144, kv-cache-dtype=fp8_inc

Problem Details

With a hybrid GDN model and 262K context:

• Total HBM per card: ~126.5 GiB
• Model weights: ~7.1 GiB
• KV cache (without VLLM_COMPACT_GDN): ~108 GiB — GDN state scales with block count, effectively doubling KV cache size
• Remaining for warmup: ~18 GiB
• Warmup graph compilation (300+ buckets): requires ~20-30 GiB peak → OOM

Even with VLLM_COMPACT_GDN=true and VLLM_GRAPH_RESERVED_MEM=0.25 (reserving 25% of HBM for warmup graphs), the warmup barely fits. However, reserving 25% of HBM (~31 GiB) for warmup graphs is extremely wasteful — that memory could be used for KV cache and serving more concurrent requests.

Error

allocation failed for 768MB

(OOM during warmup graph compilation when KV cache consumed most of HBM)

Workaround

Set VLLM_SKIP_WARMUP=true to skip warmup entirely. Graphs compile on-demand for real request shapes, which uses much less memory since real request batch sizes are smaller than the worst-case warmup buckets.

Suggested Fix

The warmup memory budget calculation should account for the GDN state cache memory overhead in hybrid models. Either:

1. Correctly estimate GDN state memory when calculating available memory for warmup graphs
2. Allow warmup to use a dynamic memory budget that adapts to the actual KV cache + GDN state usage
3. Provide a more granular way to limit warmup memory without reserving a fixed percentage of total HBM

Related

• [Feature]: Support Qwen3.5-27B (GatedDeltaNet hybrid architecture) on Intel Gaudi 2 #1284 — Original feature request for Qwen3.5 GDN support on Gaudi
• [Bug] Synapse graph compilation failure (synStatus 26) with hybrid GDN models on v0.26.0 #1734 — Synapse graph compilation failure with hybrid GDN models on v0.26.0
• [Bug] KV cache index_copy_ slot mapping mismatch in eager mode with GDN models on v0.26.0 #1735 — KV cache index_copy_ slot mapping mismatch in eager mode on v0.26.0

[shepark]

1. GDN memory estimation — The hybrid GDN KV-cache sizing adjustment in determine_available_memory exists, but by design it is applied only when compact GDN is OFF (if has_attn and has_gdn and not compact_gdn). These models auto-enable compact GDN (VLLM_COMPACT_GDN=1), where the GDN state is a small fixed allocation independent of num_blocks, so the adjustment is intentionally skipped and GDN state is not the source of the OOM. The actual OOM is on the prompt warmup path: the largest prompt graph (query = max_num_batched_tokens, context dim = ceil((max_model_len − max_num_batched_tokens)/block_size), e.g. 8192 × 1984 blocks at 262k context) exceeds the fixed graph reserve (VLLM_GRAPH_RESERVED_MEM, default 10% of usable HBM) while KV takes the other ~90%. warmup_graphs compiles every bucket largest-first with no memory-budget guard, so it OOMs. Reproduction confirms it's independent of decode/GDN: clamping decode blocks and disabling prefix caching both still OOM at the same point. So this is a separate problem from the GDN estimation.
2. & 3. Dynamic / more granular warmup budget — Reasonable long-term optimizations (peak-aware graph reserve, or a memory-budgeted warmup that skips over-sized buckets and defers them to on-demand compile). But they mainly automate a trade-off that is already tunable via config, so no code change is strictly required today:
   VLLM_GRAPH_RESERVED_MEM ↑ (e.g. 0.1 → 0.15–0.2) — more graph headroom, less KV.
   --max-num-batched-tokens ↓ — shrinks the prompt graph's query dim.
   --max-model-len ↓ — shrinks the prompt context dim.