Skip to content

Commit e1ae98c

Browse files
sw072claude
andcommitted
[Bugfix] Reject CUTLASS block-scaled FP8 when N is not a multiple of 128
Online block-wise FP8 (--quantization fp8_per_block) selected CutlassFp8BlockScaledMMKernel for layers whose output dim N is not a multiple of the weight block size (128). The CUTLASS c3x block-scaled FP8 GEMM (sm90/sm100/sm120) tiles weight scales at a fixed (128,128) granularity and its dispatch only special-cases small/unaligned M (swap_ab); it has no kernel for a partial N tile, so gemm_op.can_implement() rejects the problem and vLLM aborts at runtime with "cutlass_gemm_caller ... Invalid status". Since CUTLASS_BLOCK_FP8_SUPPORTED is True and can_implement() did not check the shape, auto-selection picked CUTLASS and crashed instead of falling back. Add the N/K block-alignment check to can_implement() so selection falls back to a supporting kernel (e.g. Triton). Selection is per-layer, so aligned layers keep the CUTLASS path. Repro: Qwen3.5 GDN linear_attn.in_proj_a/b weights are [16, 1024] (N=16) on sm_120. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent c9a788e commit e1ae98c

1 file changed

Lines changed: 25 additions & 0 deletions

File tree

  • vllm/model_executor/kernels/linear/scaled_mm

vllm/model_executor/kernels/linear/scaled_mm/cutlass.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,31 @@ def can_implement(cls, config: FP8ScaledMMLinearLayerConfig):
307307
"Supports only dynamic per token group activation "
308308
"quantization with group_shape=(1,128).",
309309
)
310+
311+
# The CUTLASS block-scaled FP8 GEMM (sm90/sm100/sm120) tiles the weight
312+
# scale factors with a fixed (128, 128) granularity along (N, K) and
313+
# provides no kernel for a partial N tile (the swap_ab dispatch only
314+
# handles small/unaligned M). If N is not a multiple of the weight
315+
# block size, CUTLASS' gemm_op.can_implement() rejects the problem at
316+
# runtime and raises "Invalid status". Report this here so kernel
317+
# selection can fall back to a kernel that supports it (e.g. Triton)
318+
# instead of crashing. See e.g. Qwen3.5 GDN in_proj layers (N=16).
319+
weight_group_shape = config.weight_quant_key.scale.group_shape
320+
n, k = config.weight_shape
321+
if weight_group_shape.row > 0 and n % weight_group_shape.row != 0:
322+
return (
323+
False,
324+
f"CUTLASS block-scaled FP8 requires the output dim N ({n}) to "
325+
f"be a multiple of the weight block size "
326+
f"({weight_group_shape.row}).",
327+
)
328+
if weight_group_shape.col > 0 and k % weight_group_shape.col != 0:
329+
return (
330+
False,
331+
f"CUTLASS block-scaled FP8 requires the input dim K ({k}) to "
332+
f"be a multiple of the weight block size "
333+
f"({weight_group_shape.col}).",
334+
)
310335
return True, None
311336

312337
def apply_block_scaled_mm(

0 commit comments

Comments
 (0)