[Skill] quant-chain-validate: Validate quantization chain correctness end-to-end

[sunway513]

Skill

quant-chain-validate

Priority: P0 (Critical — scale layout bugs cause silent garbage output)

Motivation

The most insidious bugs in AITER/ATOM are scale layout mismatches in the quantization chain. The quant→GEMM→dequant pipeline must agree on scale layout (row-major vs column-major), shuffle flags, and group size. A mismatch produces output that looks reasonable (correct magnitude) but points in the wrong direction (cosine similarity << 1.0). These bugs are hard to catch because individual components test fine in isolation — the mismatch only manifests in the full chain. See: CK-free Docker garbage output bug (cosine_sim=0.70 due to Triton fallback writing row-major scales while GEMM expected column-major).

What This Skill Should Do

1. Trace the quant chain: For a given model config, trace the full path: input → quantization (per-token/per-group) → scale layout → GEMM kernel → output
2. Verify scale layouts: Check that shuffle_scale, transpose_scale, and group_size flags are consistent between quant and GEMM stages
3. Detect fallback paths: Identify when JIT failure causes fallback to Triton quant, and verify the fallback implements all flags (especially shuffle_scale=True)
4. Cosine similarity test: Run the full chain with known inputs and compare output direction (not just magnitude) against a reference implementation using cosine similarity
5. Report mismatches: If cosine_sim < 0.999, flag the exact point where layout diverges and suggest the fix

Acceptance Criteria

• Can validate FP8 per-1x128 quantization chain (the most common failure mode)
• Can validate FP4 (MXFP4) quantization chain with block scaling
• Detects row-major vs column-major scale layout mismatches
• Detects when Triton fallback ignores shuffle_scale/transpose_scale flags
• Uses cosine similarity (not just norm comparison) as the correctness metric
• Produces a chain trace showing scale layout at each stage