Commit e1ae98c
[Bugfix] Reject CUTLASS block-scaled FP8 when N is not a multiple of 128
Online block-wise FP8 (--quantization fp8_per_block) selected
CutlassFp8BlockScaledMMKernel for layers whose output dim N is not a
multiple of the weight block size (128). The CUTLASS c3x block-scaled FP8
GEMM (sm90/sm100/sm120) tiles weight scales at a fixed (128,128)
granularity and its dispatch only special-cases small/unaligned M
(swap_ab); it has no kernel for a partial N tile, so
gemm_op.can_implement() rejects the problem and vLLM aborts at runtime
with "cutlass_gemm_caller ... Invalid status". Since
CUTLASS_BLOCK_FP8_SUPPORTED is True and can_implement() did not check the
shape, auto-selection picked CUTLASS and crashed instead of falling back.
Add the N/K block-alignment check to can_implement() so selection falls
back to a supporting kernel (e.g. Triton). Selection is per-layer, so
aligned layers keep the CUTLASS path. Repro: Qwen3.5 GDN
linear_attn.in_proj_a/b weights are [16, 1024] (N=16) on sm_120.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>1 parent c9a788e commit e1ae98c
1 file changed
Lines changed: 25 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
310 | 335 | | |
311 | 336 | | |
312 | 337 | | |
| |||
0 commit comments