Environment

• H100 SXM (SM90), driver 550.163.01 (host CUDA toolkit 12.4 at /usr/local/cuda)
• torch 2.11.0+cu129, Python 3.12; vLLM built from source at 34ad158 (the [Bugfix] Preserve Marlin runtime tensor storage across weight reload #48438 branch — its diff doesn't touch FlashInfer or MoE backend selection, and the code cited below is identical on current main)
• flashinfer-python==0.6.13 and flashinfer-cubin==0.6.13 (both from requirements/cuda.txt); flashinfer-jit-cache NOT installed (it is only installed in docker/Dockerfile, not pinned in requirements)

The instance is torn down so I can't attach collect_env.py output, but the relevant coordinates are above and the failure is fully explained by source below.

Repro

from vllm import LLM
llm = LLM(
    model="openai/gpt-oss-20b",
    max_model_len=2048,
    kernel_config={"moe_backend": "flashinfer_cutlass"},
)

FlashInfer JIT-compiles fused_moe_90 for several minutes (successfully), the model loads, then engine init fails:

RuntimeError: Check failed: (false) is false: Could not construct fused moe op with the
requested input combination Activation: bfloat16, Weight: uint8, Output: bfloat16

There is no workaround via dtype: gpt_oss_mxfp4 rejects anything but bfloat16 at config validation.

Root cause

1. vLLM selects the backend: FlashInferExperts._supports_quant_scheme allows (kMxfp4Static, None) on is_device_capability(90) (fused_moe/experts/flashinfer_cutlass_moe.py), and the availability check is has_flashinfer_cutlass_fused_moe(), an import check.
2. vLLM's call contract is correct: packed fp4 weights as torch.uint8 with use_w4_group_scaling=True, which matches flashinfer's isWFP4A16Quant() predicate (use_w4_group_scaling && weight == uint8 && !use_packed_weights).
3. flashinfer gates fp4 out of the build on old toolkits: flashinfer/jit/fused_moe.py:120 — "-DENABLE_FP4" if is_cuda_version_at_least("12.8") else "". With system nvcc 12.4, the JIT build succeeds but the entire isWFP4A16Quant() construction branch in flashinfer_cutlass_fused_moe_binding.cu (which instantiates CutlassMoeFCRunner<..., __nv_fp4_e2m1>, a CUDA 12.8+ type) is compiled out, so runner construction falls through to the ICHECK above.

So "supported" is a property of the flashinfer build, and vLLM's gates only look at the GPU arch and importability. Confirmed empirically: installing flashinfer-jit-cache==0.6.13+cu129 (prebuilt with a modern toolkit, exactly what the Dockerfile does) and rerunning the identical command works; I used the resulting engine for reload validation on #48312-related work the same day.

Who hits this

Anyone installing vLLM from source or pip (not the Docker image) on a host whose CUDA toolkit is older than 12.8 while using any fp4-dependent FlashInfer path (gpt-oss MXFP4, NVFP4 CUTLASS MoE). Docker users are immune because the image installs flashinfer-jit-cache, which is also why CI doesn't see it. Old toolkits on new-enough drivers are common on rented GPUs.

Proposed fix

Either or both, in order of preference:

1. A build-capability probe in vllm/utils/flashinfer.py (prebuilt modules present, or flashinfer's own is_cuda_version_at_least("12.8") predicate passes), threaded into the fp4-dependent scheme gates so selection falls back to Marlin/Triton with a log line instead of crashing after minutes of JIT compilation.
2. Pin flashinfer-jit-cache in requirements/cuda.txt alongside flashinfer-python/flashinfer-cubin so non-Docker installs get toolkit-independent kernels. (It needs the CUDA-specific index URL, which may be why it's Docker-only today.)

The cleanest long-term shape is a capability query exposed by flashinfer itself rather than vLLM probing its jit internals; I've filed flashinfer-ai/flashinfer#3951 asking for that, and either way I'm happy to implement the vLLM side.

AI assistance was used to root-cause and validate this; I reviewed the analysis and can defend it end to end.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.