Your current environment
collect_env output (head node, worker node is identical hardware/software)
Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 24.04.4 LTS (x86_64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version : Could not collect
CMake version : version 3.28.3
Libc version : glibc-2.39
==============================
PyTorch Info
==============================
PyTorch version : 2.13.0+cu130
Is debug build : False
CUDA used to build PyTorch : 13.0
ROCM used to build PyTorch : N/A
XPU used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.14 (main, Aug 25 2026, 14:00:49) [Clang 22.1.3 ] (64-bit runtime)
Python platform : Linux-6.8.0-138-generic-x86_64-with-glibc2.39
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : Could not collect
CUDA_MODULE_LOADING set to :
GPU models and configuration :
GPU 0: NVIDIA GeForce RTX 5090
GPU 1: NVIDIA GeForce RTX 5090
GPU 2: NVIDIA GeForce RTX 5090
GPU 3: NVIDIA GeForce RTX 5090
GPU 4: NVIDIA GeForce RTX 5090
GPU 5: NVIDIA GeForce RTX 5090
GPU 6: NVIDIA GeForce RTX 5090
GPU 7: NVIDIA GeForce RTX 5090
Nvidia driver version : 580.105.08
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : False
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Vendor ID: AuthenticAMD
BIOS Vendor ID: Advanced Micro Devices, Inc.
Model name: AMD EPYC 9654 96-Core Processor
BIOS Model name: AMD EPYC 9654 96-Core Processor Unknown CPU @ 2.4GHz
BIOS CPU family: 107
CPU family: 25
Model: 17
Thread(s) per core: 2
Core(s) per socket: 96
Socket(s): 2
Stepping: 1
Frequency boost: enabled
CPU(s) scaling MHz: 42%
CPU max MHz: 3709.3569
CPU min MHz: 1500.0000
BogoMIPS: 4793.16
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap ibpb_exit_to_user
Virtualization: AMD-V
L1d cache: 6 MiB (192 instances)
L1i cache: 6 MiB (192 instances)
L2 cache: 192 MiB (192 instances)
L3 cache: 768 MiB (24 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-95,192-287
NUMA node1 CPU(s): 96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Mitigation; Clear CPU buffers
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB before exit to userspace
==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.18
[pip3] nccl4py==0.4.1
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.1.3
[pip3] nvidia-cuda-cccl==13.3.3.4.1
[pip3] nvidia-cuda-crt==13.3.73
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.3.73
[pip3] nvidia-cuda-nvdisasm==13.3.73
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.20.0.48
[pip3] nvidia-cudnn-frontend==1.27.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.1
[pip3] nvidia-cutlass-dsl==4.6.2
[pip3] nvidia-cutlass-dsl-libs-base==4.6.2
[pip3] nvidia-cutlass-dsl-libs-core==4.6.2
[pip3] nvidia-cutlass-dsl-libs-cu12==4.6.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.6.2
[pip3] nvidia-ml-py==13.610.43
[pip3] nvidia-nccl-cu13==2.29.7
[pip3] nvidia-nvjitlink==13.3.33
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.3.73
[pip3] pyzmq==27.2.0
[pip3] tokenspeed-triton==3.8.10.post20260721
[pip3] torch==2.13.0
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchcodec==0.16.0
[pip3] torchvision==0.28.0
[pip3] transformers==5.16.1
[pip3] triton==3.7.1
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
vLLM Version : 0.28.0
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
�[4mGPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID�[0m
GPU0 X PIX PXB PXB SYS SYS SYS SYS PXB PXB SYS SYS 0-95,192-287 0 N/A
GPU1 PIX X PXB PXB SYS SYS SYS SYS PXB PXB SYS SYS 0-95,192-287 0 N/A
GPU2 PXB PXB X PXB SYS SYS SYS SYS PXB PXB SYS SYS 0-95,192-287 0 N/A
GPU3 PXB PXB PXB X SYS SYS SYS SYS PIX PIX SYS SYS 0-95,192-287 0 N/A
GPU4 SYS SYS SYS SYS X PXB PXB PXB SYS SYS PIX PIX 96-191,288-383 1 N/A
GPU5 SYS SYS SYS SYS PXB X PXB PXB SYS SYS PXB PXB 96-191,288-383 1 N/A
GPU6 SYS SYS SYS SYS PXB PXB X PIX SYS SYS PXB PXB 96-191,288-383 1 N/A
GPU7 SYS SYS SYS SYS PXB PXB PIX X SYS SYS PXB PXB 96-191,288-383 1 N/A
NIC0 PXB PXB PXB PIX SYS SYS SYS SYS X PIX SYS SYS
NIC1 PXB PXB PXB PIX SYS SYS SYS SYS PIX X SYS SYS
NIC2 SYS SYS SYS SYS PIX PXB PXB PXB SYS SYS X PIX
NIC3 SYS SYS SYS SYS PIX PXB PXB PXB SYS SYS PIX X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
==============================
Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
- vLLM: 0.28.0 (reproduced also on nightly 0.28.1rc1.dev396)
- flashinfer-python 0.6.18 + flashinfer-cubin 0.6.18
- 2 nodes x 8x RTX 5090 (SM120), ray distributed executor backend, NCCL over RoCE (GDR), driver 580.105.08
🐛 Describe the bug
Serving deepseek-ai/DeepSeek-V4-Flash-0731 with --tensor-parallel-size 16 (2 nodes, --distributed-executor-backend ray) fails during model loading on every rank with:
RuntimeError: FLASHINFER_MLA_SPARSE_DSV4 on SM120 requires a FlashInfer DSV4 sparse MLA decode
specialization for (num_q_heads=8, top_k=128). Install a FlashInfer build containing
flashinfer-ai/flashinfer#4380.
What makes this look like a bug rather than a missing-kernel issue: the check contradicts the installed FlashInfer:
- On both nodes, running the exact check the code uses returns True:
$ /root/vllm/bin/python -c "from vllm.utils.flashinfer import has_flashinfer_sparse_mla_sm120_config as f; print(f(8,128))"
True
- The installed FlashInfer dispatch table contains the required entry:
$ python -c "from flashinfer.mla import _sparse_mla_sm120 as m; print((8,128) in m._DECODE_DSV4_DISPATCH)"
True
python -c "import flashinfer; ..." resolves to the same venv on both nodes, and a ray remote task probe confirms ray workers also use that venv (sys.executable and flashinfer.__file__ both point into it).
Yet the same (8,128) check raises inside DeepseekV4FlashInferSM120Attention.__init__ when run under the ray worker during load_model.
Additional data point: TP=8 (16 q-heads per rank) on a single node passes this check and loads fine — only TP=16 (8 heads per rank) triggers the error. Both vLLM 0.28.0 and nightly dev396 behave identically.
To Reproduce
# both nodes: identical venv path /root/vllm, vllm 0.28.0, flashinfer 0.6.18 (+cubin), ray 2.58
ray start --head --port=6379 --num-cpus 64 # node 1
ray start --address=<node1>:6379 --num-cpus 64 # node 2
vllm serve deepseek-ai/DeepSeek-V4-Flash-0731 \
--tensor-parallel-size 16 \
--distributed-executor-backend ray \
--kv-cache-dtype fp8 \
--block-size 256 \
--attention_config.use_fp4_indexer_cache False \
--tokenizer-mode deepseek_v4 \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
--max-model-len 262144 --max-num-seqs 64 \
--gpu-memory-utilization 0.9 --max-num-batched-tokens 32768Traceback (from worker, via RayTaskError):
File ".../vllm/v1/worker/gpu_worker.py", line 457, in load_model
self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
File ".../vllm/v1/worker/gpu/model_runner.py", line 360, in load_model
self.model = model_loader.load_model(...)
...
File ".../vllm/models/deepseek_v4/nvidia/flashinfer_sparse.py", in __init__
required_topk = _required_sm120_sparse_topk(vllm_config, self.window_size)
if not has_flashinfer_sparse_mla_sm120_config(self.padded_heads, required_topk):
raise RuntimeError(
"FLASHINFER_MLA_SPARSE_DSV4 on SM120 requires a FlashInfer "
f"DSV4 sparse MLA decode specialization for "
f"(num_q_heads={self.padded_heads}, top_k={required_topk}). "
"Install a FlashInfer build containing flashinfer-ai/flashinfer#4380.")
Expected behavior
Either the model loads (since the installed FlashInfer dispatch table does contain (num_q_heads=8, top_k=128) and the standalone check returns True), or the capability check is made consistent between the standalone path and the ray-worker path.
Suspected area
vllm/utils/flashinfer.py::has_flashinfer_sparse_mla_sm120_config returning a different result inside the ray worker context (lazy submodule import / dispatch-table visibility under the worker's CUDA-initialized runtime?), or flashinfer.mla._sparse_mla_sm120 failing to expose _DECODE_DSV4_DISPATCH in that context.
Thanks!
Your current environment
collect_env output (head node, worker node is identical hardware/software)
🐛 Describe the bug
Serving
deepseek-ai/DeepSeek-V4-Flash-0731with--tensor-parallel-size 16(2 nodes,--distributed-executor-backend ray) fails during model loading on every rank with:What makes this look like a bug rather than a missing-kernel issue: the check contradicts the installed FlashInfer:
python -c "import flashinfer; ..."resolves to the same venv on both nodes, and a ray remote task probe confirms ray workers also use that venv (sys.executableandflashinfer.__file__both point into it).Yet the same
(8,128)check raises insideDeepseekV4FlashInferSM120Attention.__init__when run under the ray worker duringload_model.Additional data point: TP=8 (16 q-heads per rank) on a single node passes this check and loads fine — only TP=16 (8 heads per rank) triggers the error. Both vLLM 0.28.0 and nightly dev396 behave identically.
To Reproduce
Traceback (from worker, via RayTaskError):
Expected behavior
Either the model loads (since the installed FlashInfer dispatch table does contain
(num_q_heads=8, top_k=128)and the standalone check returns True), or the capability check is made consistent between the standalone path and the ray-worker path.Suspected area
vllm/utils/flashinfer.py::has_flashinfer_sparse_mla_sm120_configreturning a different result inside the ray worker context (lazy submodule import / dispatch-table visibility under the worker's CUDA-initialized runtime?), orflashinfer.mla._sparse_mla_sm120failing to expose_DECODE_DSV4_DISPATCHin that context.Thanks!