Follow-up from #303 at the maintainer's request: gpt-oss measures badly on any quantized K cache, turbo or otherwise, and the effect is a property of the model class rather than an encoder or kernel defect. Filing so the first heavy user of the V head_dim padding path doesn't read as a turbo regression.
Measurements
Instrument: llama-perplexity --kl-divergence against the model's own f16-KV logits (wikitext probe, 8192 ctx, gfx1100/HIP, but the effect reproduces CPU-only). Reference envelope from a 128-dim-head model we run in production (median KLD ~0.018, same-top ~91.5% on turbo3 K+V).
gpt-oss-20b (stock unsloth GGUF), same instrument:
| K / V |
median KLD |
same-top |
| f16 / f16 |
0 (base) |
100% |
| q8_0 / q8_0 |
0.055 |
74.9% |
| turbo3 / q8_0 |
0.735 |
32.1% |
| turbo4 / q8_0 |
0.233 |
65.9% |
| turbo3 / turbo3 |
0.666 |
35.4% |
Output stays locally fluent throughout — a short-answer smoke test passes while the distribution is wrecked, which is what makes this dangerous to ship silently.
Why we believe it is the model, not the code
- Decode fidelity is at spec: dequantized K rows measure cos 0.983–0.989 against the exact rotated reference on real activations — turbo3's designed ~1.5% angular error, same as on healthy models.
- The KQ rotations are exact (reference and graph-rotated logits agree to 3 decimals on real activations).
- Rotation scheme doesn't move it: 128-groups vs full-head-width rotation groups vs Hadamard-disabled all land in the same band. Only bit depth moves it, and even 8-bit K is ~3x the healthy envelope.
- Mechanism: gpt-oss K rows carry sink-style bias outliers (values to ~23 vs ~2 typical; row norms ~30 vs Q ~10). In-spec quantization noise multiplied by those norm products lands multi-logit errors pre-softmax, and softmax exponentiates them. Sink-heavy attention appears to be intrinsically hypersensitive to K-cache quantization.
Suggested disposition
A docs note (or a startup warning keyed on models with attention sinks + quantized K) recommending f16-K for this model class. We serve gpt-oss f16-KV and everything else turbo3 on the same build.
Happy to re-run any of these measurements on gfx1100 against the integration branch on request.
🤖 Authored by Claude (AI) operating apollosenvy's machine, as disclosed in #299/#303.
Follow-up from #303 at the maintainer's request: gpt-oss measures badly on any quantized K cache, turbo or otherwise, and the effect is a property of the model class rather than an encoder or kernel defect. Filing so the first heavy user of the V head_dim padding path doesn't read as a turbo regression.
Measurements
Instrument:
llama-perplexity --kl-divergenceagainst the model's own f16-KV logits (wikitext probe, 8192 ctx, gfx1100/HIP, but the effect reproduces CPU-only). Reference envelope from a 128-dim-head model we run in production (median KLD ~0.018, same-top ~91.5% on turbo3 K+V).gpt-oss-20b (stock unsloth GGUF), same instrument:
Output stays locally fluent throughout — a short-answer smoke test passes while the distribution is wrecked, which is what makes this dangerous to ship silently.
Why we believe it is the model, not the code
Suggested disposition
A docs note (or a startup warning keyed on models with attention sinks + quantized K) recommending f16-K for this model class. We serve gpt-oss f16-KV and everything else turbo3 on the same build.
Happy to re-run any of these measurements on gfx1100 against the integration branch on request.
🤖 Authored by Claude (AI) operating apollosenvy's machine, as disclosed in #299/#303.