kvcached support matrix

[RixinLiu]

We need a comprehensive compatibility matrix covering the model architectures kvcached interacts with. This tracks what works, what crashes, and what produces wrong output, so users know what to expect and contributors know where to focus.

Test models

Architecture	Model	Why
Standard GQA	Qwen3-8B	Baseline — most common setup
SWA + dense hybrid	GPT-OSS-20B	Alternating sliding-window / full-attention layers
Hybrid linear	Qwen3.5-9B	3:1 GDN hybrid, some layers have no KV
Cross-layer KV sharing	Gemma-4-E2B-it	Layers share KV with earlier layers
Heterogeneous KV groups	Gemma-4-12B-it	Per-layer KV head count varies

vLLM

Model	contiguous	non-contiguous
Qwen3-8B
GPT-OSS-20B
Qwen3.5-9B
Gemma-4-E2B-it
Gemma-4-12B-it

SGLang

Model	contiguous	non-contiguous
Qwen3-8B
GPT-OSS-20B
Qwen3.5-9B
Gemma-4-E2B-it
Gemma-4-12B-it

Legend: ✅ pass / ❌💥 crash at startup (link issue #) / ❌🗑️ garbled output (link issue #) / ➖ N/A

How to contribute

Pick a blank cell, run the model with kvcached enabled, and report the result. A single --max-model-len 1024 short generation is enough to fill a cell.

[jeff3071]

I tested SGLang with KVCACHED_CONTIGUOUS_LAYOUT=true

Env:
Commit: 666671b
SGLang: 0.5.12
Python: 3.12.3
OS: Linux 6.17.0-29-generic / Ubuntu 24.04.1, x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell

Model	Result
Qwen/Qwen3-8B	PASS
openai/gpt-oss-20b	PASS
Qwen/Qwen3.5-9B	PASS
google/gemma-4-E2B-it	PASS

Note: Qwen/Qwen3.5-9B requires KVCACHED_PAGE_SIZE_MB=4 for this test.

Gemma4-12B-it fail, but it does not appear to be caused by kvcached (Sglang v0.5.12).

[RixinLiu]

hi @jeff3071 , thanks for the efforts to test them! I just merged your integration with newer SGLang version, can you retry Gemma4-12B-it model to see whether kvcached works? Thanks!

[jeff3071]

I test with SGLang v0.5.12 v0.5.13 v0.5.14 v0.5.15 on main(8aa9a04)

Model	v0.5.12	v0.5.13	v0.5.14	v0.5.15
Qwen/Qwen3-8B	PASS	PASS	PASS	PASS
openai/gpt-oss-20b	PASS	PASS	PASS	PASS
Qwen/Qwen3.5-9B	PASS	PASS	Crash	Crash
google/gemma-4-E2B-it	PASS	PASS	PASS	PASS
google/gemma-4-12B-it	Crash	Crash	Crash	PASS

Qwen3.5-9B crash is related to Sglang v0.5.14 code and v0.5.15 pass new param to MambaPool.

[rishabhsinha17]

vLLM results, both layouts.

Env:
Commit: 8aa9a04 (main)
vLLM: 0.20.0 (Gemma rows: 0.23.0 + transformers 5.14.1, see note)
Python: 3.11.10
GPU: NVIDIA A100-SXM4-80GB (driver 565.57.01, cuda-compat-13-0 for vLLM's cu130 torch)

Model	contiguous	non-contiguous
Qwen/Qwen3-8B	PASS	PASS
openai/gpt-oss-20b	PASS	PASS
Qwen/Qwen3.5-9B	PASS	PASS
google/gemma-4-E2B-it	Crash, #417 PASS on 0.24	Crash, #417 PASS on 0.24
google/gemma-4-12B-it	Crash (by design) PASS on 0.24 (#445)	Crash, #438 PASS on 0.24

Notes:

• Qwen/Qwen3.5-9B requires KVCACHED_PAGE_SIZE_MB=4, same as @jeff3071 found on SGLang. The GDN block is 2.0625 MiB (33x64 KiB), so at 4 MB pages one block fits per page: ~48% of the linear-attention pool is padding. 8 MB: 23%. 16 MB: 10%. The startup error already computes the minimum page size; auto-picking it instead of crashing would remove the footnote. Can file separately.
• Gemma rows cannot run on vLLM 0.20: it predates the gemma4_unified model support (registry entry added in 0.23). The only vLLM with the arch before the 0.24 incompatibility (page-size calculation incompatible with vLLM 0.24+ AttentionSpec.indexes_kv_by_block_stride #399) is 0.23.0, tested there. E2B dies in kvcached's layer binding with KeyError: 'language_model.model.layers.15.self_attn.attn', exactly Need support for Gemma model which has cross-layer kv sharing #417 (repro posted there). 12B contiguous hits the documented heterogeneous-groups guard (needs non-contiguous). 12B non-contiguous hits ElasticBlockPool.free_blocks() breaks on newer vLLM: unexpected keyword argument 'prepend' #438.
• Caveat: vanilla vLLM 0.23 + transformers 5.14 produces degenerate output for the Gemma models even without kvcached, so the 0.23 cells are crash-attribution only, not output validation. The Gemma/vLLM rows are sequenced behind newer-vLLM support (page-size calculation incompatible with vLLM 0.24+ AttentionSpec.indexes_kv_by_block_stride #399, Fix vLLM padded KV page allocation and strides #426, ElasticBlockPool.free_blocks() breaks on newer vLLM: unexpected keyword argument 'prepend' #438) and then Need support for Gemma model which has cross-layer kv sharing #417.

PASS cells: greedy 48-token outputs coherent and identical across the two layouts.

Edit 2026-08-13: Gemma rows re-run on vLLM 0.24 + main 094481f after #445, now PASS; the 0.23 degenerate-output caveat below was my harness sending raw completion prompts to -it models, not a vLLM problem. Full corrected matrix in the follow-up comment.