[Feature]: Performance Optimization for Deepseek V4

[yewentao256]

🚀 The feature, motivation and pitch

Performance Optimization for Deepseek V4

Tasks:

• [Perf] Optimize DSv4 prefill chunk planning, 4.0% E2E Throughput Improvement #45061
• [DSV4 Perf] Optimize dsv4 cudagraph by reducing eager_break_during_capture, 26.8% ~ 27.9% E2E TTFT improvement #45309 *(Current buggy, but worth more exploration)
• [DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement #45863
• [DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 #44577 @tlrmchlsmth
• [Perf] Optimize fused_topk_bias for DSv4, 1.5~2x kernel performance improvement #47463
• [Perf] Cache token_to_req_indices for dsv4, 5x~6x kernel performance improvement #47474
• [Perf] Remove redundant repeat and copy for dsv4, 1.8% E2E TPOT improvement. #48137
• [DSv4] Remove sparse-MLA q-head padding for FlashInfer >=0.6.14 #48047 @majunze2001
• [Perf] Optimize dsv4 routing using specialized kernel, 2.94% E2E TPOT improvement #48660
• prefill chunk size

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[vanmilleru]

Deepseek V4 doesn't seems to work on single 80G GPU memory even if cpu offload is enabled.

Can it work by enabling that?

[yewentao256]

@vanmilleru Please take a look at https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro

[vanmilleru]

@yewentao256 I used --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --attention_config.use_fp4_indexer_cache=True --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --cpu-offload-gb 100".

It still runs out of GPU memory even with eager mode enabled. I wonder if the offload is actually working or not.

[yewentao256]

@vanmilleru I meant usually we use at least 8 cards even for H200 (141GB memory). Even if using cpu-offloading, it may not work for your 80GB GPU.

[vanmilleru]

Can i run DeepSeek V4-Flash with 2*80G using one card per node?

[yewentao256]

I don't think it works as well, you might want to get more gpu resources first.

[vanmilleru]

I thought V4-Flash in fp4 takes ~142GB GPU mem in total. and 2*80 gives 160GB. it will certainly need some activation and KV cache but can most of the MLP be off loaded to use cpu and cpu mem?

[yewentao256]

Thanks for the clarification.

What you describe is not the primary deployment scenarios we optimize or validate in our recipes. In practice, it may work with offloading, or even with other non-standard workarounds such as trimming parts of the model, but those are not the recommended and can easily run into various stability, compatibility, and performance issues.

For that reason, the most practical recommendation is still to use more GPU resources rather than trying to push this configuration further.