Your current environment

Your output of `python collect_env.py` here

🐛 Describe the bug

Describe the bug

Serving DeepSeek V4 with tensor + expert parallelism on Hopper GPUs crashes during weight loading with a CUDA illegal memory access.

The error surfaces asynchronously (at the next torch.cuda.empty_cache() call inside DeviceMemoryProfiler), meaning the traceback line is not the true fault site. The actual out-of-bounds access occurs earlier during the Marlin weight/scale repacking for the MoE layer (ops.gptq_marlin_repack or the scale permutation in prepare_moe_mxfp4_layer_for_marlin).

Traceback / Logs

(Worker_TP7_EP7 pid=3823) ERROR ... torch.AcceleratorError: CUDA error: an illegal memory access was encountered
...
File ".../vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 617, in prepare_moe_mxfp4_layer_for_marlin
File ".../vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 611, in permute_scales
    marlin_scales = mxfp4_marlin_process_scales(...)
File ".../vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py", line 120, in mxfp4_marlin_process_scales
    marlin_scales = marlin_scales.view(-1, 4)[:, [0, 2, 1, 3]].view(...)
torch.AcceleratorError: CUDA error: an illegal memory access was encountered

Root cause

1. Fallback Path: Because Hopper GPUs lack native MXFP4 tensor-core support, _get_priority_backends() in vllm/model_executor/layers/fused_moe/oracle/mxfp4.py falls through DeepGEMM/FlashInfer (which require SM100+) down to the MARLIN weight-only fallback for MXFP4 MoE.
2. Alignment Requirement: The Marlin GEMM/repack kernels require N/K dimensions to be tile-aligned (multiples of 64 or 128).
3. Missing Padding: Under expert parallelism (EP), the per-rank intermediate size (intermediate_size_per_partition) is not guaranteed to satisfy this alignment. However, prepare_moe_mxfp4_layer_for_marlin performs no padding before calling ops.gptq_marlin_repack / marlin_permute_scales—unlike its NVFP4 sibling.

A recent fix, #45295 ("[Kernel] Consolidate Marlin thread-tile padding across all dense Marlin paths"), successfully added this zero-padding to:

• prepare_fp4_layer_for_marlin (dense/linear NVFP4)
• prepare_nvfp4_moe_layer_for_marlin (NVFP4 MoE)

However, it did not update prepare_moe_mxfp4_layer_for_marlin (the MXFP4 MoE variant). The code still assumes the incoming weight shape is already tile-aligned per its own comments:

"Derive dimensions from actual weight shapes to handle rounded/padded sizes correctly... e.g. Mxfp4MoEMethod rounds up hidden_dim"

For DeepSeek V4 under EP, that assumption does not hold, and the unaligned repack triggers the illegal memory access. This belongs to the same bug class as #38022 ("Marlin MoE kernel fails with MXFP4-quantized GPT-OSS 20B — Invalid thread config for non-aligned dimensions").

Expected behavior

prepare_moe_mxfp4_layer_for_marlin should zero-pad the per-rank intermediate size to the nearest Marlin-tile-aligned boundary before repacking—exactly how prepare_nvfp4_moe_layer_for_marlin handles it. This will prevent MXFP4 MoE + Marlin fallback + EP setups from crashing on non-tile-aligned shard sizes.

Reproduction

Not easily reproducible with a minimal script as it requires an 8x Hopper GPU cluster and the DeepSeek V4 checkpoint. I am happy to help test and verify a fix against the original environment if a PR is put together.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.