NexusQuant

Near-lossless KV cache quantization. Training-free. One line of code.

---

E8 lattice quantization applied to the KV cache after prefill. No training, no calibration data, no model modifications.

Live demo: huggingface.co/spaces/jmarquex/nexusquant-demo

Headline: K3V2 pb=0 achieves +0.276% PPL at 5.83x compression on Mistral-7B-v0.1 (wikitext-2, n=60 chunks, KIVI-iso protocol). NIAH recall preserved through 32K context on Mistral-Inst-v0.3 (A100, 24-25/25 per cell). Validated on 9 model families. Head-to-head on Llama-3.1-8B-Instruct 4K: NQ K2V2 30/30 NIAH vs TurboQuant 2-bit 0/30 (n=30, McNemar b=29/c=0).

The PPL deltas above were measured with the 0.5.x quantizer. nexusquant-kv 0.6.0 corrects the E8 tie-break (Conway-Sloane largest-error re-rounding instead of smallest-error), lowering mean lattice distortion by roughly 8% on Gaussian test inputs (not yet measured on real KV), so the published deltas are conservative for current installs.

Install

pip install nexusquant-kv
pip install "nexusquant-kv[hf]"  # with HuggingFace transformers

compress_kv_cache (quant-only API) requires transformers >= 4.46 (oldest version tested). nexusquant_evict requires transformers >= 5.0 and torch >= 2.4 (eviction hooks use DynamicLayer/DynamicSlidingWindowLayer, which are 5.0+ only).

Quickstart

Quant-only (near-lossless, recommended)

Prefill all but the last token, compress the cache, then let generate consume the final token. Copy and run it.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, DynamicCache
from nexusquant import compress_kv_cache

model_name = "mistralai/Mistral-7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float16, device_map="auto"
)
model.eval()

text = "The KV cache stores key and value tensors for each attention layer."
inputs = tokenizer(text, return_tensors="pt").to(model.device)
input_ids = inputs["input_ids"]

cache = DynamicCache()
with torch.no_grad():
    out = model(input_ids[:, :-1], use_cache=True, past_key_values=cache)
past_key_values = out.past_key_values

rope_base = getattr(model.config, "rope_theta", 10000.0)
compressed_kv = compress_kv_cache(
    past_key_values,
    mode="quant_only",
    rope_base=rope_base,
)

with torch.no_grad():
    gen_out = model.generate(
        input_ids,
        past_key_values=compressed_kv,
        max_new_tokens=200,
        do_sample=False,
    )

print(tokenizer.decode(gen_out[0], skip_special_tokens=True))

At 128K context on Mistral-7B (32 layers, 8 KV heads, head_dim=128) this drops the KV cache from ~17.2 GB at FP16 to ~2.95 GB at K3V2 pb=0 (5.83x). The flow above is verified to exit 0 with generated text on CPU using hf-internal-testing/tiny-random-LlamaForCausalLM, nexusquant-kv 0.5.3, torch 2.2.2, transformers 4.46.3, run from /tmp.

With eviction (higher compression)

Requires transformers >= 5.0, torch >= 2.4, and a float16 or bfloat16 model (float32 is not supported for eviction; use compress_kv_cache(mode="quant_only") for float32). nexusquant_evict evicts low-attention tokens and E8-quantizes the survivors in-place during prefill. NIAH recall degrades past 35% eviction (see Limitations).

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced") as nq:
    output = model.generate(input_ids, max_new_tokens=512)

After the context exits, nq.last_mask holds the (batch, seq) eviction attention mask; pass it to follow-up generate() calls for multi-turn.

HuggingFace ecosystem

Two on-ramps wire E8 lattice KV quantization (training-free) into the HuggingFace stack.

The standalone cache is a DynamicCache subclass you pass to model.generate. It needs no change to transformers core. from_model reads rope_theta and rope_scaling from the config, so Llama-3.1-family scaling is handled.

from nexusquant.integrations.quantized_cache import E8QuantizedCache

cache = E8QuantizedCache.from_model(model)
output = model.generate(**inputs, past_key_values=cache, max_new_tokens=64)

For the kvpress ecosystem, E8Press is a kvpress.BasePress subclass that runs the E8 quant pass inside kvpress's prefill compress() hook. Install with pip install nexusquant-kv[kvpress].

from nexusquant.integrations.kvpress import E8Press

press = E8Press(key_bits=3, value_bits=2)
with press(model):
    output = model.generate(**inputs, max_new_tokens=64)

Behavior in both: the prefill KV is E8-quantized (K3V2 default). Recent single-token decode writes are kept as an fp16 residual (KIVI-style), so the long-context prefill carries the recall signal while recent tokens stay full precision. That residual is small at long context but not free; count it in any compression-ratio number.

API reference

compress_kv_cache

from nexusquant import compress_kv_cache
compressed_kv = compress_kv_cache(past_key_values, mode="quant_only", **kwargs)

Parameter	Type	Default	Description
past_key_values	DynamicCache	required	From model(..., use_cache=True).past_key_values
mode	str	"simple"	"quant_only" (near-lossless), "simple" (5.3x), "fast" (no RoPE rm), "max" (PCA, needs calibration)
head_dim	int	128	Attention head dimension; auto-detected if you use the HF nexusquant context manager
rope_base	float	10000.0	RoPE theta from model.config.rope_theta
rope_scaling	dict or None	None	Pass model.config.rope_scaling for Llama-3.1 and similar
bits	int	3	Quantization bits (2, 3, or 4)
merge_pct	float	0.0	Token merge fraction (0 = disabled); only for "simple" mode

Returns the same past_key_values object, modified in-place with quantized KV tensors.

nexusquant_evict

from nexusquant import nexusquant_evict

with nexusquant_evict(model, quality="balanced") as nq:
    output = model.generate(input_ids, max_new_tokens=200)

Parameter	Type	Default	Description
model	HF causal LM	required	Any model using DynamicCache
quality	str or None	"balanced"	Preset: "high" (K3V2, 35% evict, ~9x), "balanced" (K2V2, 60% evict, ~17x), "max" (K2V2, 80% evict, ~33x), "asym" (K3V2, 60% evict, ~14x). Set None to use raw args.
eviction_rate	float or "auto"	0.6	Fraction of prefix tokens to evict. Ignored when a named quality preset is given.
sliding_window	int	32	Recent tokens always kept, never evicted
obs_window	int	32	Query positions used to score token importance (key-key scorer)
bits	int	2	E8 quantization bits for surviving tokens
key_bits	int or None	None	Override bits for keys only (e.g. key_bits=3, bits=2 = K3V2)
value_bits	int or None	None	Override bits for values only
scorer	str	"key-key"	"key-key" (fast, no extra pass) or "real" (softmax weights, needs attn_implementation="eager")
input_ids	Tensor or None	None	Required when scorer="real". The (batch, seq) prefix token ids for the importance-scoring forward pass.
protect_boundary	int or "auto"	0	Keep first+last N layers at FP16. Use 2 for Qwen-family models.
protected_layers	set or None	None	Set of layer indices (0-based) to skip compression entirely (kept at FP16).
min_context_for_compression	int	0	Skip compression when the prefill sequence is shorter than this value.
protected_positions	Tensor or None	None	Token indices that are never evicted (e.g., VLM image spans).
truncate	bool	False	Physically remove evicted tokens; saves real GPU memory but needs position_ids in generate()
soft_eviction	bool	False	1-bit instead of zero for evicted tokens
adaptive_context	bool	False	Scale eviction rate down for short prefixes
compress_layers	str	"all"	"global_only" skips sliding-window attention layers (Gemma4, Phi)
layer_bit_profile	str	"uniform"	"graduated" gives boundary layers more bits
verbose	bool	True	Print compression stats on exit

Yields a compressor object. After the first generate() inside the context, nq.last_mask is a (batch, seq) float attention mask for the evicted cache; pass it as attention_mask to follow-up generate() calls (multi-turn).

Results

PPL (quant-only, no eviction)

Mistral-7B-v0.1, wikitext-2, 1K-token prefix, n=60 chunks, KIVI-iso protocol, paired positions, relative (aggregate) PPL delta vs FP16.

Config	bpe (honest)	PPL delta vs FP16
FP16	16.000	0.000%
K4V2 pb=0	3.125	+0.15%
K3V2 pb=0	2.625	+0.276%
K2V2 pb=0	2.125	+0.96%

bpe = (key_bits + scale_bpe + value_bits + scale_bpe) / 2; scale_bpe = 16/128 = 0.125.

Deltas above use the published 4-decimal PPL chain (e.g. 4.8356/4.8223 for K3V2, 4.8687/4.8223 for K2V2); the shipped sidecar's full-precision values are 0.274%/0.961% for the same two cells, agreeing within the rounding chain (experiments/kaggle/results/rank116_kivi_iso_bpe.json).

The K4V2 pb=0 row (+0.15%) is not in that sidecar, which holds only the FP16, K3V2 pb=0, and K2V2 pb=0 cells. It traces to the original Kaggle run (kernel rankLL_kivi_h2h_llama_inst) whose result JSON was not persisted and ships as run-log provenance, matching the paper's Table 6 note.

NIAH recall (quant-only, chat-template, Mistral-Inst-v0.3, A100-80GB)

Context	FP16	K3V2 pb=0	K2V2 pb=0
12K	25/25	25/25	25/25
32K	25/25	25/25	24/25

32K K2V2 pb=0: one miss at depth=0.5. At 16K (not shown), FP16 and K2V2 pb=0 score 25/25 and K3V2 pb=0 scores 24/25 (one miss at depth=0.5).

AQUA-KV comparison (Llama-3.1-8B-base, both under AQUA-iso: wikitext-2, 6 x 8192, prefix=1024)

Both columns run the AQUA-iso protocol (arXiv:2501.19392, Table 2). The NexusQuant rows are an iso-replication of AQUA-KV's harness; NexusQuant matches or improves on AQUA-KV's reported PPL deltas at lower or comparable raw bits-per-element. The pipeline is calibration-free; AQUA-KV requires per-architecture calibration.

Method	Raw bpe	PPL delta vs FP16	Calibration
NexusQuant K3V2 pb=0	2.625	+0.487%	none
AQUA-KV (reported, Table 2)	3.06	+0.53%	required
NexusQuant K2V2 pb=0	2.125	+1.649%	none
AQUA-KV (reported, Table 2)	2.09	+1.96%	required

Competitive head-to-head (2-bit KV, Llama-3.1-8B-Instruct, 4K, n=30)

Same harness, same 30 NIAH needles, T4 x2. The TurboQuant comparison is iso-bpe (same harness, same needles). The Block-GTQ comparison is NON-ISO (Block-GTQ is K-only + calibrated; NexusQuant is K+V calibration-free).

Config	bpe	NIAH hits	Notes
FP16	16.0	29/30	1 natural model failure
TurboQuant bw2	2.125	0/30	Complete retrieval collapse
Block-GTQ K2V2	2.0	29/30	Calibrated, pre-RoPE K-alloc
NexusQuant K2V2 pb=0	2.0	30/30	Calibration-free, post-RoPE E8

TurboQuant vs NexusQuant: McNemar b=29, c=0. Block-GTQ vs NexusQuant: 1 discordant pair, non-discriminative. Results: experiments/kaggle/results/nq_tq_h2h_llamainst.json, experiments/kaggle/results/nq_blockgtq_h2h_4k.json.

References:

• Paper PDF: paper/nexusquant.pdf
• Repro scripts (neither reproduces the n=60 +0.276% wikitext headline, which is Kaggle-kernel provenance):
  • scripts/repro_headline.py: the n=161 paired noise-floor gate (checks the +0.31% mean per-chunk delta vs FP16).
  • scripts/kivi_iso_repro.py: the KIVI Table-3 task-accuracy gate (CoQA/TruthfulQA/GSM8K), not a PPL reproducer.
  • To verify the headline value itself, read experiments/kaggle/results/rank116_kivi_iso_bpe.json.
• HF model card: scripts/HF_MODEL_CARD.md

Two-tier compression

Tier	What it does	Compression	PPL impact	NIAH recall	Use case
Quant-only	E8 lattice VQ, no eviction	~6x	~0% (near-lossless)	preserved on all tested models	Quality-critical apps
Light eviction	E8 VQ + light eviction (real scorer)	~5-7x	model- and scorer-dependent	model- and scorer-dependent	Balanced quality + compression
Aggressive eviction	E8 VQ + 35-80% eviction	8-33x	+0.1-2.1% (context-dependent)	degrades	Memory-critical ("fits vs doesn't fit")

The NIAH cliff position is model- and scorer-dependent: on Gemma-2-2b-it the proxy scorer fails below 15% eviction at 4K while the real scorer holds to about 25% at 2K. Validate the cliff on your model before deploying eviction.

Supported architectures

Quality correlates strongly with KV head count. Models with 4+ KV heads are safe at K3V2 (exception: Qwen family needs boundary protection). Models with fewer than 4 heads degrade.

Model family	Status	Caveats
Mistral-7B-v0.1	Validated	Main PPL benchmark; K3V2 pb=0 +0.276% PPL (wikitext-2, n=60 chunks, KIVI-iso protocol)
Mistral-Inst-v0.3	Validated	K3V2 pb=0 +0.33% PPL (wikitext-2, 1K-prefix); NIAH 40/40 at 4K and 8K, 24-25/25 per cell through 32K on A100
Llama-3 / Llama-3.1	Validated	Pass rope_scaling=model.config.rope_scaling; K4V2 recommended (rope_scaling makes 3-bit keys harder: K3V2 +0.64% vs K4V2 +0.29%)
Yi-6B / Yi-6B-Chat	Validated	K3V2 pb=0 +0.35% PPL; chat-template NIAH 5/5 at 4K
Qwen2.5-7B	Validated	Requires protect_boundary=2; boundary-off is catastrophic (+539x PPL)
Qwen3-8B	Validated	K3V2 pb=0 +1.130% PPL (iso-protocol, strict FP16-weights baseline; NF4-weights sensitivity run gives +0.38%, not comparable to the FP16-weights rows); boundary protection recommended
Mixtral-8x7B-Inst-v0.1	Validated	K3V2 pb=0 +0.55% PPL (NF4-weights baseline, not true FP16, n=40; 47B model cannot fit FP16 weights on the validation hardware; delta is vs the NF4 baseline, not comparable to the FP16-weights rows above); NIAH 15/15 through 12K
Gemma-2-2b	Validated	K3V2 pb=0 +0.21% PPL global-only (+0.57% all-layers); bf16 weights required
Phi-3 / Phi family	Validated	Low PPL delta (~0.4% K3V2); set compress_layers="global_only" for SWA layers
Gemma4 (SWA models)	Supported	Use compress_layers="global_only"; 1 KV head = NIAH not reliable
Qwen2.5-1.5B	Not recommended	2 KV heads: +5.04% PPL at K3V2. Below the safe threshold.
GPT-NeoX / GPT-J	Not supported	Interleaved RoPE not yet implemented
Encoder-decoder (T5, BART, Whisper)	Not supported	KV cache structure differs
Vision-language models	Not tested	Untested; may work for the language decoder

How it works

1. Importance scoring - rank tokens by attention weight
2. Token eviction (optional) - drop lowest-scoring tokens; always keep BOS and a recent sliding window
3. RoPE removal - undo rotary embeddings on keys so they share a common subspace
4. Hadamard rotation - spread energy uniformly across dimensions
5. E8 lattice quantization - quantize 8-float groups onto the E8 root lattice. Asymmetric: 3-bit keys + 2-bit values
6. Boundary protection - optionally keep first/last N layers at FP16 (mandatory for Qwen-family)
7. Entropy coding (optional) - zstd over the E8 index stream picks up residual codeword redundancy, a small further lossless gain (up to ~10% at K2V2, less at K3V2)

Why

Without NexusQuant	With NexusQuant
128K context on 70B = ~42 GB KV cache (GQA)	Same context = ~7 GB KV cache (6x quant-only)
KV cache competes with model weights for VRAM	KV cache fits comfortably alongside weights
Long context needs multi-GPU or offloading	Single GPU, single machine
Deploy a fine-tuned retrieval model	One with block, no code changes

Quality presets

Measured on Mistral-7B (A100 or A10G as noted per row). Compression ratios include all overhead.

Preset	Compression	PPL degradation	Config
high	~9x	+0.35%	K3V2 + 35% evict (A100, 3544-tok)
asym	~14x	estimated <1%	K3V2 + 60% evict (not fully validated)
balanced	~17x	+0.82%	K2V2 + 60% evict (A10G, 1664-tok)
max	~33x	+2.13%	K2V2 + 80% evict (A10G, 1664-tok)

(ablation-derived from the measured eviction ablations; no standalone sidecar)

PPL alone does not tell the full quality story. Eviction modes show degraded NIAH recall despite small PPL increases. Use quant-only when factual accuracy matters. See Limitations below.

Cross-architecture PPL results

Quant-only K3V2 pb=0 is near-lossless on all models with 4+ KV heads.

Model	KV Heads	K3V2 pb=0 PPL delta	Notes
Gemma-2-2b	4	+0.21%	global-only (SWA layers at fp16); +0.57% all-layers
Mistral-7B	8	+0.276%	Main benchmark
Yi-6B	4	+0.35%	2K prefix (+0.41% at 1K)
Mistral-Inst-v0.3	8	+0.33%	1K-prefix
Llama-3.1-Inst	8	+0.64%	P1 protocol; needs rope_scaling propagation
Mixtral-8x7B-Inst	8 (MoE)	+0.55%	NF4-weights baseline, not true FP16, n=40; not comparable to the FP16-weights rows above (no strict FP16 rerun exists; 47B model can't fit FP16 on the validation hardware)
Qwen3-8B	8	+1.130%	iso-protocol, strict FP16-weights baseline; NF4-weights sensitivity run +0.38% (not comparable to FP16-weights rows)
Qwen2.5-1.5B	2	+5.04%	2 KV heads = danger zone

Advanced options

Graduated layer bit profile - gives boundary layers (first/last 15%) higher precision (3-bit K+V) while middle layers use standard asymmetric (K3V2).

with nexusquant_evict(model, quality="high", layer_bit_profile="graduated"):
    output = model.generate(input_ids, max_new_tokens=200)

Hybrid model compression - for models like Gemma4 with sliding-window + global attention layers, only compress the global layers.

with nexusquant_evict(model, compress_layers="global_only"):
    output = model.generate(input_ids, max_new_tokens=200)

Compared to

Method	Compression	Quality (as reported)	Training required	Notes
NexusQuant K3V2 pb=0	6x	+0.28% wikitext-2 PPL	No	Quant-only, no eviction
NexusQuant K2V2 pb=0	7.5x	+0.96% wikitext-2 PPL	No	Quant-only, no eviction
TurboQuant (Google)	4.6x (3.5 bits) / 6.4x (2.5 bits)	quality-neutral at 3.5 bits, marginal loss at 2.5 (their eval); 0/30 NIAH at 2.125 bpe in our iso-bpe H2H	No	Quant-only, no eviction
Block-GTQ (arXiv:2606.24033)	K-only, calibrated	29/30 NIAH at 2.0 bpe in our NON-ISO H2H	Yes (1 fwd pass calibration)	K-only allocation, pre-RoPE
KVarN (arXiv:2606.03458)	sub-8-bit	calibration-free, Hadamard normalized	No	vLLM backend (K4V2 preset)
HyperQuant (arXiv:2606.23406)	2-4 bps	Hadamard + E8 lattice + Rice entropy coding	No	bps = bits per scalar = bpe
KVTC (NVIDIA)	up to 20x	accuracy maintained (their benchmarks; no wikitext-2 PPL published)	Yes (calibration)
CommVQ	8x at 2-bit	minimal accuracy loss (their benchmarks)	Yes (EM-trained codebook)

Competitor numbers from their papers or our head-to-head experiments; H2H rows are measured on our hardware. Quality columns use each paper's own metric unless marked H2H, so rows are not directly comparable.

Troubleshooting

Import resolves to the repo directory instead of the installed package

If you run python from inside a cloned copy of this repo, Python's import machinery finds the local nexusquant/ directory before the installed package. Run your script from a different directory, or use the full venv path:

cd /tmp
/path/to/your/venv/bin/python your_script.py

ImportError: cannot import name 'DynamicLayer' from 'transformers.cache_utils'

This means your transformers version is below 5.0. The eviction API (nexusquant_evict) requires transformers >= 5.0 and torch >= 2.4. The quant-only API (compress_kv_cache) works from transformers 4.46.

pip install "transformers>=5.0" "torch>=2.4"

torch version ceiling on Apple Intel (x86_64)

Apple Intel Macs are limited to torch <= 2.2.x. Torch 2.3+ requires Apple Silicon or Linux. On Intel Macs the eviction API (nexusquant_evict) is not available because it needs torch >= 2.4. The quant-only API works on torch 2.2.

transformers and torch version matrix

API	transformers	torch	Notes
compress_kv_cache	>= 4.46	>= 2.0	Tested on 4.46.3
nexusquant context manager	>= 5.0	>= 2.4	Uses DynamicLayer hooks
nexusquant_evict context manager	>= 5.0	>= 2.4	Uses DynamicLayer hooks

Limitations

• Quality is text-dependent. Creative/narrative text degrades more than structured/technical text.
• Short prefixes hurt. Prefixes under 500 tokens see more degradation.
• Architecture-dependent boundary protection. Qwen-family models catastrophically fail without protect_boundary=2. Always test your specific model.
• E8 quantization is CPU-bound. Triton GPU kernel is written but not yet benchmarked for latency.
• Eviction hurts factual recall. NIAH benchmark shows degradation at 35%+ eviction. PPL hides this damage.
• PPL is not a sufficient quality metric. Always validate with NIAH or downstream accuracy.
• Results on 7B-class models primarily. 70B validation pending at scale.
• Batch size > 1 is limited. NexusQuantMax and NexusQuantAsymmetric do not support batch size > 1. NexusQuantSimple and the quant-only pipelines compress every batch index.
• vLLM runtime integration is not functional. nexusquant/integrations/vllm_backend.py provides experimental page-compression utilities only; register_nexusquant_backend raises NotImplementedError. Use the HF integration.
• Multi-turn chat is not supported. The hook compresses on every incoming prefill.
• Speculative decoding is not supported.
• KV cache offloading is not supported.
• Encoder-decoder models (T5, BART, Whisper) are not supported.
• Vision-language models are untested.
• GGUF models are not supported.

Reproducibility: paper claims

Standalone runners under reproducibility/ reproduce the headline tables in the paper.

Script	Paper table
reproducibility/niah_kv_canonical.py	tab:niah_kv_canonical
reproducibility/commvq_head_to_head.py	tab:commvq_head
reproducibility/ruler_13task.py	tab:ruler_13task
reproducibility/entropy_coding_live.py	tab:entropy_live
reproducibility/turboquant_validated.py	TurboQuant H2H
reproducibility/niah_long_context.py	tab:nq_long_a100 (12K/16K/32K)
reproducibility/kivi_upstream_gate.py	tab:kivi_h2h (upstream gate cells)

See reproducibility/README.md for setup, GPU requirements, and reproduction commands.

Citation

@software{nexusquant2026,
  author  = {Marques, Jo\~{a}o},
  title   = {{NexusQuant}: Training-Free {KV} Cache Compression via {E8} Lattice Quantization and Attention-Aware Token Eviction},
  year    = {2026},
  url     = {https://github.com/jagmarques/nexusquant},
  license = {Apache-2.0},
}

License

Apache 2.0. See LICENSE.