Skip to content

[Bugfix] Fix fp8_ds_mla config leak in DSpark speculative decoding#48415

Closed
pranavthakur0-0 wants to merge 3 commits into
vllm-project:mainfrom
pranavthakur0-0:fix/fp8-ds-mla-global-mutation
Closed

[Bugfix] Fix fp8_ds_mla config leak in DSpark speculative decoding#48415
pranavthakur0-0 wants to merge 3 commits into
vllm-project:mainfrom
pranavthakur0-0:fix/fp8-ds-mla-global-mutation

Conversation

@pranavthakur0-0

Copy link
Copy Markdown
Contributor

Description

When an MLA model (like GLM-5.2 or DeepSeek) is initialized with --kvcache-dtype fp8, the MLAAttention and DeepseekV4Attention constructors secretly mutated the globally shared cache_config instance to force cache_config.cache_dtype = "fp8_ds_mla".

When running speculative decoding (like dspark) alongside it, the draft model uses standard attention layers that read this poisoned global config. Standard backends (FlashAttention, Triton) do not support the DeepSeek-specific MLA memory layout, causing the AttentionSelector to immediately crash with an unhandled ValueError.

Changes

  • Single files modified: vllm/model_executor/layers/attention/mla_attention.py and vllm/models/deepseek_v4/attention.py
  • Removed the global cache_config.cache_dtype = "fp8_ds_mla" assignments.
  • MLAAttention now isolates "fp8_ds_mla" strictly to its internal self.kv_cache_dtype state.
  • Leaves the global config untouched so standard draft models can boot up normally.

Related Issue

Closes #48406

Test Results

Legacy (Without patch):

[FAILED] Standard Attention crashed: No valid attention backend found for cuda with AttentionSelectorConfig(... kv_cache_dtype=fp8_ds_mla ...). Reasons: {FLASH_ATTN: [kv_cache_dtype not supported]...

Fixed (With patch):

INFO: Using fp8 data type to store kv cache.
[INFO] After MLA model load, global cache_config.cache_dtype = fp8
[SUCCESS] Standard Attention initialized successfully using TRITON_ATTN.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mergify mergify Bot added the bug Something isn't working label Jul 12, 2026
@pranavthakur0-0 pranavthakur0-0 closed this by deleting the head repository Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: glm-5.2-fp8 with --kvcache-dtype fp8 with h20-3e with dspark error

1 participant