TurboQuant KV cache quantization — feasibility report and optimization plan for Intel XPU

[bryanvine]

TurboQuant KV cache quantization — feasibility report and optimization plan for Intel XPU

Summary

TurboQuant (vLLM PR #38479) is a data-oblivious KV cache quantization scheme based on Walsh-Hadamard rotation + Lloyd-Max scalar quantization, giving 3–5× KV memory compression.

We ported the upstream Triton kernels to Intel XPU (Arc Pro B70, Xe2 Battlemage) and confirmed they compile and produce correct results without any kernel modifications. Details, reproducible tests, and benchmark data are in this public repo: https://github.com/bryanvine/turboquant-xpu

Filing this issue to:

1. Document that TurboQuant works today on XPU via the upstream Triton path
2. Share benchmark data on what Intel users gain (KV capacity) and pay (throughput)
3. Propose a native SYCL optimization plan to close the throughput gap

Functionality — works on XPU today

Using vLLM 0.19.0 + intel-xpu-backend-for-triton (bundled in the intel/vllm:latest image):

• All 3 TurboQuant Triton kernels (fused store, decode stage1, stage2 reduction) compile cleanly on Intel's SPIRV backend
• tl.float8e4nv works on XPU for FP8 key paths
• tl.reshape + grouped sum works for 3-bit MSE packing
• 2D scatter-gather from byte-addressable cache works
• Online softmax accumulation produces bit-correct results vs reference

Tested presets (all confirmed working):

• turboquant_k8v4 (FP8 keys + 4-bit values)
• turboquant_k3v4_nc (3-bit MSE keys + 4-bit values + norm correction)
• turboquant_3bit_nc (3-bit symmetric)

Benchmark data

Qwen3-30B-A3B-Instruct-2507-gptq-4bit (MoE, head_dim=128)

KV cache capacity at max-model-len=262144:

Config	KV budget	KV tokens	vs FP16
FP16 baseline	~10 GiB	~65K	1.0×
TQ k3v4_nc	11.8 GiB	549,888	~8.5×

Throughput (90s per concurrency level, 16-prompt mix):

C	FP16 baseline (+EAGLE3)	TQ k3v4_nc (no spec)	ratio
1	~24 tok/s	8.7 tok/s	0.36×
8	~192 tok/s	63.1 tok/s	0.33×
16	~296 tok/s	140.0 tok/s	0.47×
20	298.5 tok/s	141.1 tok/s	0.47×

Gemma 4 31B W4A16 GPTQ (dense, head_dim=256/512 heterogeneous)

C	FP16 + suffix	TQ k3v4_nc + suffix	ratio
12	134.3 tok/s	33.8 tok/s	0.25×
16	95.9 tok/s	36.3 tok/s	0.38×

Effective KV capacity: 10,240 → 49,408 tokens (4.83×).

Why the slowdown on XPU

The upstream Triton kernels are tuned for NVIDIA SM80/90. On Xe2 the same kernels compile and produce correct results but miss the hardware because:

1. BLOCK_KV=4 hardcoded — Xe2 has 16/32-wide subgroups and 256 EUs; small tile sizes under-utilize the SIMD width
2. num_warps=1 for decode stage1 — under-subscribes EU count
3. Software dequant per attention step — B70 has DPAS/XMX units that aren't touched by the generic Triton path
4. WHT rotation GEMM is external — uses oneDNN with separate launch overhead; fusing into stage1 would save a kernel launch per decode step
5. No CUDA-graph equivalent — eager dispatch adds per-token Python overhead

Proposed optimization plan

We'd like to propose adding TurboQuant attention as a first-class XPU kernel in this repo, similar to how gdn_attention and other specialized kernels are handled.

Phase A — Triton tuning (quick wins, no new kernels)

• Profile TQ decode on Xe2 and tune BLOCK_KV / num_warps / num_stages autotuning configs
• Validate correctness across all 4 TQ presets on Arc B580, B60, B70
• Expose tunables via env vars matching the pattern in flash_attn (VLLM_XPU_FA_*)

Rough expected gain: 1.5–2× over current Triton path (bringing Qwen3-30B TQ from 0.47× → 0.7–0.8× of FP16 baseline throughput).

Phase B — Native SYCL kernels for decode hotpath

Port the three kernels to SYCL targeting Xe2:

• _tq_fused_store_mse — bucketize + centroid gather + residual norm + pack
• _tq_decode_stage1 — split-KV tiled score + value accumulation (the decode hotpath)
• _tq_full_dequant_kv — bulk dequant for continuation prefill
• Fuse the Q @ PiT rotation GEMM into _tq_decode_stage1 (eliminates one launch per decode step)
• Use DPAS intrinsics for the score × value accumulation
• Stage centroid table and Hadamard signs in SLM

Rough expected gain: 2–3× over tuned Triton, approaching FP16 baseline throughput on MoE models.

Phase C — Ecosystem integration

• Wire SYCL kernels through _xpu_C custom ops the same way attention/FA kernels are exposed
• Add TurboQuant to the upstream Qwen3.5 optimization plan — compressing KV by 8× would dramatically help the GDN hybrid's memory pressure at long contexts
• CI: extend the v0.1.6 release tracker to include a TQ test matrix

Why this matters for Xe2 / Arc Pro

The B70 and B60 have 32 GB VRAM but no hardware fp8 matmul, so KV cache is a memory-bound tradeoff. TurboQuant at 8× compression turns "running out of KV" into "we can serve 200K+ context workloads" on a single Arc card — a strong story for Intel's Arc Pro line against NVIDIA's larger HBM cards.

With current Triton-only performance, TurboQuant on XPU is useful but costs 2–4× throughput. With native SYCL kernels, the throughput gap should close to a few percent, making it the default choice for long-context workloads on Arc.

How I can help

We already have:

• A working Triton port with 8 vLLM files patched for integration (mount-based, against unmodified intel/vllm:latest)
• A benchmark harness (bench_tq.py) and reproducible test matrix
• Unit tests for the quantizer math (27/27 passing) and kernel correctness (6/6 passing on XPU)
• All code Apache-2.0, matching the upstream vLLM license

Happy to contribute the Triton patches back upstream into vllm PR #38479 and collaborate on the SYCL port if there's interest.

Repo: https://github.com/bryanvine/turboquant-xpu
Benchmarks: https://github.com/bryanvine/turboquant-xpu/blob/main/docs/BENCHMARK_RESULTS.md
Qwen3-30B analysis: https://github.com/bryanvine/turboquant-xpu/blob/main/docs/BENCHMARK_QWEN3_30B.md

Hardware tested: Intel Arc Pro B70 (32GB GDDR6X, Xe2 / BMG-G31, PCI 8086:e223), Ubuntu 25.10, driver 1.14.36300+8, PyTorch XPU from intel/vllm:latest.

CC: @wuxun-zhang (Qwen3.5 optimization plan author) — filing partly because TQ is a natural complement to the GDN work.

[yma11]

@bryanvine Thanks for raising this ticket. I have monitoring this feature on XPU and based on my check, the latest code can work properly w/o any change, as the PR owner already do necessary support on XPU, see line. For performance, we will first have an internal evaluation and do have plan for SYCL version implementation.

[bryanvine]

Thanks @yma11, glad SYCL TurboQuant is on the roadmap.

A few profile datapoints from Arc Pro B70 (BMG-G31) that may be useful for your internal eval. Shape: N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192, k8v4 preset; torch-XPU 2.8 + Intel Triton:

• Stage1 GPU time: 1087 µs/call
• LZ submission overhead: ~87 µs × 2 dispatches + ~175 µs Python = 262 µs / Python call
• Memory BW: 23.6 / 608 GB/s (3.9%) — not BW-bound
• Compute: ~6.2% of FP32 peak — per-EU occupancy is the bottleneck, not peak throughput
• Wall for the N_spec=8 loop: 8.92 ms, of which ~24% is LZ submission overhead

Arithmetic intensity 20.9 FLOP/byte sits above the FP32 roofline ridge, so nominally compute-bound — but at only 6% of peak.

Full analysis + workload: https://github.com/bryanvine/turboquant-xpu (docs/TRITON_PROFILE_ANALYSIS.md, scripts/profile_triton_decode.py).

Two Triton-level optimizations fall out of this data, orthogonal to a SYCL port:

1. Fuse N_spec queries into one dispatch. Saves ~14% wall directly from dispatch count; the 8× grid expansion should also lift the 6% compute util. Prototyping now; we'll land the implementation and benchmark numbers in https://github.com/bryanvine/turboquant-xpu once it clears the projected 35–55% wall reduction, for you to pick up or reference.
2. Autotune BLOCK_KV / num_warps / num_stages for Xe2. Upstream defaults are SM80/90-tuned; we saw ~2× from hand-swept configs on k3v4_nc (see docs/QUICK_WINS_RESULTS.md in the same repo).

Neither is a SYCL port, so they're fully complementary to your planned work. Happy to share the benchmark harness if it would save setup time on your side.

— Bryan

[bryanvine]

Update: the fused-N_spec Triton kernel landed cleanly. Same shape (N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192) on the B70:

preset	looped (ms)	fused (ms)	speedup
turboquant_k8v4	8.955	3.305	2.71×
turboquant_k3v4_nc	16.005	3.795	4.22×

Clears the projected 35–55% reduction by a comfortable margin. k3v4_nc benefits more than k8v4 — the MSE centroid-gather + norm load amortizes across queries, which is a heavier per-tile cost than the single FP8 load. Notably, k3v4_nc fused (3.795 ms) is now faster than k8v4 looped (8.955 ms) at this shape, which flips the previous preset recommendation for spec-decode workloads.

Implementation + tests + bench script live at https://github.com/bryanvine/turboquant-xpu, commit 425fc5c:

• kernel: src/turboquant_xpu/kernels/triton_decode.py::_tq_decode_stage1_spec
• stage2 for [N_spec, B, Hq, NUM_KV_SPLITS] reduce: src/turboquant_xpu/kernels/triton_stage2.py::_tq_decode_stage2_spec
• launcher: triton_turboquant_decode_attention_spec_xpu in xpu_decode.py
• tests: tests/test_fused_nspec.py (passes vs looped baseline on both presets, atol=5e-3, rtol=1e-2)
• bench: scripts/bench_fused_nspec.py
• writeup: docs/FUSED_NSPEC_RESULTS.md

Feel free to pick this up, reference it, or use as a baseline for the SYCL port. Happy to answer questions if anything is unclear.

— Bryan

[bryanvine]

Small correction on my earlier numbers: the benchmark I shared measured parallel-completion scoring (all N queries attending to the same prefix seq_len), not causal speculative-decode verification (where query n attends to cached_len + n + 1 tokens). The looped baseline used the same seq_len for every loop iteration, so the comparison was internally consistent — but it's not what vLLM's actual spec-verify path does, which uses synth_seq_lens = arange(cached_len+1, seq_len+1) per call.

Added a CAUSAL mode to _tq_decode_stage1_spec that accepts a cached_len scalar and computes per-query effective seq_len inside the kernel (c0a69a3). New causal-verify numbers at the same PoC shape (N_spec=8, B=4, Hq=32, Hk=4, D=128, seqlen=8192, cached_len=8184):

preset	looped causal (ms)	fused causal (ms)	speedup
turboquant_k8v4	8.989	3.523	2.55×
turboquant_k3v4_nc	14.088	4.851	2.90×

Slightly below the parallel-completion numbers (2.76×/3.52×) — the per-query mask comparison costs a bit inside the hot loop. Still a real improvement on the path vLLM actually uses.

Original parallel-completion numbers remain valid for workloads where all queries share the same prefix seq_len (e.g., scoring N alternative continuations from one prefix). Test matrix covers both modes: tests/test_fused_nspec.py::test_fused_causal_matches_looped and ::test_fused_matches_looped.

— Bryan

[bryanvine]

Final update: landed the backend-layer integration and ran the bench against vLLM's actual dispatch path (_prefill_attention continuation-chunk branch in the TurboQuant attention backend). Real numbers at the integration layer:

preset	looped backend (ms)	fused backend (ms)	speedup
turboquant_k8v4	3.14	2.93	1.07×
turboquant_k3v4_nc	3.99	1.95	2.04×

Lower than the kernel-alone micro-bench (2.75× / 2.99× causal), and the delta is load-bearing: the kernel micro-bench looped baseline made N_spec separate B=4 kernel calls, while vLLM's _prefill_attention continuation-chunk path instead makes one B=N_spec=8 call with synthesized seq_lens. The existing baseline already exploits cross-query batch parallelism, so the fused kernel's remaining win comes only from K-dequant sharing — which helps the MSE-centroid path (k3v4_nc) substantially, and the FP8 path (k8v4) barely.

For deployments using turboquant_k3v4_nc with speculative decoding, enabling the fused dispatch is worth ~2× at the attention layer. For k8v4, roughly neutral — gate it off.

Patch is env-var gated (TQ_USE_FUSED_SPEC=1) in patches/vllm_mounts/backends/turboquant_attn.py. Full integration + correctness test + bench in commit 9974d8e. Server-layer tokens/sec wasn't measured — our vllm-xpu container doesn't have TQ mounted right now.

— Bryan