[RFC]: FlashInfer NVFP4 KV serving on pre-SM100 GPUs

[lesj0610]

Motivation.

This RFC tracks the work needed to make --kv-cache-dtype nvfp4 usable with FlashInfer on pre-SM100 GPUs, mainly Ampere/Hopper systems.

The practical target is serving large-context Qwen3.6 and Gemma 4 models with FlashInfer-backed NVFP4 KV cache without relying on the Blackwell-only path. This started from trying to serve those models on SM8x hardware, but I have tried to split the work into generic vLLM/FlashInfer pieces instead of model-specific patches.

vLLM already has Triton-based quantized KV cache paths such as TurboQuant 4bit and int4 per-head. The goal here is different: enable the FlashInfer CUDA path to use NVFP4 KV cache on pre-SM100 systems, so users can combine FlashInfer attention/dequantization support with the roughly 3x KV cache capacity benefit of NVFP4.

Proposed Change.

The work is split across several PRs:

PR	Scope	Status
#46963	Core pre-SM100 NVFP4 KV cache update path through FlashInfer slot-mapping append	Ready for review
#46883	Consume FlashInfer workspace sizing in vLLM	Ready for review
#47547	Allow Gemma4 to choose FlashInfer when FA4 is unavailable	Ready for review
#46558	FlashInfer MM-prefix attention support	Draft / included for dependency context
#47637	Qwen3-VL shape-specialized pos-embed JIT fix	Draft / separate follow-up

Some PRs are still draft because I am trying not to add more ready-for-review PRs before the core pieces are triaged. They are included here for context and dependency tracking, not as a request to review everything at once.

The FlashInfer side has been split separately:

• Add NVFP4 slot-mapping paged KV append flashinfer-ai/flashinfer#3740 — NVFP4 paged KV append with slot mapping
• Add paged NVFP4 KV dequantization helper flashinfer-ai/flashinfer#3748 — paged NVFP4 KV dequantization helper
• Optimize FP4 V-scale dequant for h512 FA2 prefill flashinfer-ai/flashinfer#3822 — SM8x-gated h512 NVFP4 FA2 prefill optimization

My current understanding of the review order is:

1. [Core] Use FlashInfer workspace sizing helper #46883, because it wires FlashInfer workspace sizing into vLLM.
2. [Core] Use FlashInfer for pre-SM100 NVFP4 KV cache updates #46963, because it is the core pre-SM100 NVFP4 KV cache update path and depends mainly on the FlashInfer slot-mapping append API.
3. [Model] Allow Gemma4 to use FlashInfer when FA4 is unavailable #47547, so Gemma4 can actually select FlashInfer when FA4 is unavailable.
4. [Bugfix][Attention] Enable FlashInfer mm-prefix attention #46558 and [Bugfix] Avoid shape-specialized Qwen3-VL pos-embed JIT #47637 as follow-up/context PRs, not blockers for the core path.

I would like feedback on whether this split and review order makes sense, and whether any scope or test coverage should be adjusted before deeper review.

Feedback Period.

At least one week, or until the relevant maintainers have had a chance to comment on the preferred review order and ownership.

CC List.

No response

Any Other Things.

The main local validation target is RTX 3090 / SM86. I have been testing the path with Qwen3.6 and Gemma 4 style workloads using --kv-cache-dtype nvfp4. The detailed capacity, MRCR, and serving benchmark tables are in #46963.

I do not have H100/Hopper hardware locally, so the Hopper side is not hardware-verified by me yet. The intent is still generic pre-SM100 support, not an SM86-only path.

This is also not meant to replace the existing Triton-based quantized KV cache options. The changes are intended to keep the generic vLLM behavior intact while allowing FlashInfer-backed NVFP4 KV cache updates on pre-SM100 systems.

If the design, scope, or tests should be changed before review, I can update the PRs.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[waynehacking8]

SM120 datapoint (RTX PRO 6000, current main fa4321d): --kv-cache-dtype nvfp4 currently has no valid attention backend at all here - the selector rejects every candidate with 'kv_cache_dtype not supported', FLASHINFER included. #46963 groups SM120 with SM100 as native (is_device_capability_family(100) or (120) in use_native_nvfp4_kv_cache_update), so as written the new slot-writer path bypasses the one Blackwell family that has no working native path today. If SM12x is allowed to take the pre-SM100 path (or the native gate keys on actual trtllm-gen availability instead of family), I'm happy to validate the stack on this box.

[lesj0610]

@waynehacking8 You were right — the native NVFP4 KV update gate shouldn't key on the SM family number. Updated in cd06010: the native path is now used only when can_use_trtllm_attention returns true for both prefill and decode. Otherwise (including current SM120) it falls back to the slot-mapping writer.

SM100 trtllm-gen path is unchanged. If SM120 gets trtllm-gen support later, it will pick up the native path automatically.

I don't have SM120 hardware, so runtime validation there would help.

[waynehacking8]

Validated on the RTX PRO 6000 (SM120): #46963 at cd06010 with flashinfer main (c53229ef, repo JIT), Qwen3-0.6B, FLASHINFER backend, enforce_eager. --kv-cache-dtype nvfp4 boots and generates coherently (greedy outputs differ from a bf16-KV baseline only at quantization level), and a ~2K-token needle-recall prompt returns the needle exactly through the quantized pages, so the paged read path is doing its job. I also ran the negative control to prove which branch runs: the same config against a wheel without the helpers fails init with your 'FlashInfer NVFP4 slot-mapping KV cache update is required' error, confirming SM120 takes the slot-writer path rather than native.

One packaging note: the slot-writer/dequant helpers aren't in any released wheel yet - I content-checked flashinfer-python 0.6.12/0.6.13/0.6.14 and all lack nvfp4_quantize_append_paged_kv_cache_with_slot_mapping - so with vllm's current 0.6.13 pin this path only works against flashinfer main until the next release cut.

[lesj0610]

Small update on the packaging side: no source checkout needed if using the FlashInfer nightly.

I checked nightly-v0.6.14-20260707, and the flashinfer_python-0.6.14.dev20260707 wheel includes both nvfp4_quantize_append_paged_kv_cache_with_slot_mapping and nvfp4_kv_dequantize_paged.

So this works with the nightly FlashInfer wheels plus FLASHINFER_DISABLE_VERSION_CHECK=1, until the helpers land in a stable release.

[lesj0610]

https://github.com/flashinfer-ai/flashinfer/releases/tag/nightly-v0.6.14-20260707