GLM-5.2-FP8 with DSA enablement

[jkaniecki]

Enable GLM-5.2 with DSA (DeepSeek Sparse Attention) on HPU

What changed:

hpu_sparse_attn_indexer.py (new): per-request Q·K BF16 scoring + torch.topk to select top-2048 KV cache slots per decode step
hpu_attn.py: forward_mqa_sparse — gathers top-K latent KV, decompresses, runs masked MLA decode attention
oot_mla.py: dispatches to forward_mqa_sparse when use_sparse=True
deepseek_v2.py: BF16 indexer cache, HPU-safe Indexer.forward, SparseAttnIndexer dispatch
platform.py / hpu_model_runner.py: DSA routing + indexer cache layer index fix
Validated on 8× Gaudi3 (GLM-5.2-FP8, TP=8):

GSM8K 5-shot (1319 samples): 93.6% exact match
AIME 2026 - 100% - on pair with model card's target (99,2 % mean for multiple reruns)

[Copilot]

Pull request overview

Enable GLM-5.2 FP8 with DeepSeek Sparse Attention (DSA) execution on Intel Gaudi (HPU) by wiring sparse indexer execution and adding a sparse decode attention path in the HPU MLA backend.

Changes:

• Add an HPU SparseAttnIndexer implementation and route upstream SparseAttnIndexer.forward_native to it for HPU.
• Introduce an HPU MLA sparse decode path (forward_mqa_sparse) and dispatch to it from the OOT MLA wrapper when use_sparse=True.
• Extend KV-cache spec discovery / platform routing to recognize attention modules that provide their own KV-cache spec and to log sparse-attention selection.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
vllm_gaudi/v1/worker/hpu_model_runner.py	Adds KV-cache spec extraction for attention layers implementing AttentionLayerBase.get_kv_cache_spec.
vllm_gaudi/platform.py	Updates sparse-attention backend selection logging and adds a runner KV-cache multi-layer hook.
vllm_gaudi/ops/hpu_sparse_attn_indexer.py	New HPU indexer implementation used by DSA to produce top-k cache slot indices.
vllm_gaudi/models/deepseek_v2.py	Monkey-patches DeepSeek indexer/cache paths to use BF16 and dispatch SparseAttnIndexer to the HPU implementation.
vllm_gaudi/attention/oot_mla.py	Routes decode to forward_mqa_sparse when sparse mode is active and passes the top-k index buffer through.
vllm_gaudi/attention/backends/hpu_attn.py	Adds forward_mqa_sparse implementation that gathers/decompresses top-k KV and runs sparse decode attention.

Suppressed comments (2)

vllm_gaudi/ops/hpu_sparse_attn_indexer.py:9

• The docstring claims "BF16 matmul + torch.topk", but the implementation currently does not do Q·K scoring or torch.topk, and hidden_states/weights are unused. This is misleading for maintainers and reviewers.

def forward_hpu(self, hidden_states, q, k, weights):
    """HPU SparseAttnIndexer: BF16 matmul + torch.topk."""
    forward_context = get_forward_context()
    attn_metadata = forward_context.attn_metadata

vllm_gaudi/ops/hpu_sparse_attn_indexer.py:40

• Decode-path slot selection uses the batch-wide attn_metadata.block_list and then expands the same all_slots to every sequence in the batch. In this codebase block_list is a flattened list across requests and block_groups indicates which request each block belongs to (see hpu_model_runner.py:2446-2451). As written, this can mix KV slots between different requests, which is both a correctness issue and a potential cross-request data exposure.

    # Decode: use first-N physical slots from the request's context blocks.
    # Full top-K logit scoring deferred to kernel optimization phase.
    batch_size = q.shape[0]
    block_list = attn_metadata.block_list
    pos_range = torch.arange(block_size, device=block_list.device)
    all_slots = (block_list.unsqueeze(1) * block_size + pos_range.unsqueeze(0)).reshape(-1)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.