transformers backend: per_token_group_quant_8bit not implemented for Float8_e4m3fn (FP8 MoE models)

[fattchris]

Description

When using --model-impl transformers with FP8-quantized MoE models (e.g., GLM-5.2-FP8), the engine crashes during initialization with:

RuntimeError: Worker failed with error 'operator(), csrc/libtorch_stable/quantization/w8a8/fp8/per_token_group_quant.cu:274, "per_token_group_quant_8bit" not implemented for 'Float8_e4m3fn'

Root Cause

At line 274 of per_token_group_quant.cu, VLLM_STABLE_DISPATCH_FLOATING_TYPES only handles float, half, bfloat16 — not Float8_e4m3fn. When the transformers backend passes FP8-quantized tensors through vLLM's quantization fusion ops (+quant_fp8), the dispatch has no code path for FP8 input.

The kernel template per_token_group_quant_8bit_kernel<T, DST_DTYPE, ...> already handles arbitrary types — ComputeGroupScale does static_cast<float>(src) which works for Float8_e4m3fn. Only the dispatch macro is missing FP8 cases.

Fix

Add VLLM_STABLE_DISPATCH_FLOATING_AND_FP8_TYPES macro to dispatch_utils.h that dispatches FP8 types with scalar_t alias (not fp8_t), and use it at line 274 of per_token_group_quant.cu.

Important: Do NOT change the packed register kernel at line 598 — it has static_assert(GROUP_SIZE==128) that only works for 2-byte types (half/bfloat16). FP8 (1 byte) changes the vectorization width and breaks the constraint.

Reproduction

vllm serve <FP8 MoE model> --model-impl transformers --tensor-parallel-size 8 --enforce-eager --trust-remote-code

Tested with GLM-5.2-FP8 (78-layer MoE, 754B params) on 8×B200.

Environment

• vLLM: 0.25.1 (built from source)
• torch: 2.11.0+cu130
• transformers: 5.13.1
• Model: GLM-5.2-FP8 (finegrained FP8 MoE)

Impact

Blocks --model-impl transformers for all FP8-quantized MoE models. The transformers backend is needed for mid-layer hooks (abliteration, DMEAN, activation steering) that aren't possible with the native vLLM implementation.