[BlivionIaG]

Summary

LeapQuant stores each Gated DeltaNet (GDN) recurrent state as a small INT8 blob
instead of FP32 (arXiv 2609.38166,
vllm#64725).
It is opt-in (VLLM_RDNA_GDN_LEAPQUANT=1), lossy, and gfx10x only.
The default stays FP32 with 1024-token blocks.

MTP2, MonoKernel, FULL_AND_PIECEWISE and prefix caching stay enabled.
All LeapQuant ops are HIP. No Triton path is added.

Status: ready for review. The matrix is in (see Matrix). Two gates are not met; the owner accepted the 1k c=1 loss.

This description makes no speed or accuracy claim yet. The speed and trade-off
conclusions will be written from the full matrix. The table below is earlier
qualification data, kept as measured numbers only.

How a step works

flowchart LR
    P[Prefill chunk] -->|materialize blob to FP32,<br/>run HIP prefill, store back| B[(Blob in<br/>home block)]
    B --> N[Normalize: commit accepted records,<br/>fold when 32 records exist]
    N --> D[Decode or verify:<br/>read blob and replay records]
    D -->|append records,<br/>set pending| B

Loading

• A verify step appends one record per draft token. It commits nothing yet.
• The next step commits only the accepted records. Rejected drafts are never committed.
• Normalize runs once per KV-cache group, never across groups.
  (vLLM puts the 36 GDN layers in 3 groups. Each group has its own block table.)

Page layout (MTP2, 512-token block, per rank)

Part	Size
12 head blobs x 37,024 B (window 32, rank 4)	433.875 KiB
Convolution state	25 KiB
Spare in the 512 KiB page	53.125 KiB

The FP32 state is 64 KiB per head. The blob is 1.77x smaller.
KV capacity goes from 266,945 to 273,456 tokens (+2.44%). This is not a 2x gain.

Changes

• HIP kernels: normalize, decode/verify, materialize, store. MonoKernel support.
• Accepted-record commit and rollback. V2 align-mode prefix copies.
• 512-token block mode. Bounded admission: the scheduler reserves output growth
  and every in-flight verify batch. Two unresolved prefills can retain five
  pages. Decoders go first (FCFS). No speculative slot or checkpoint is removed.
• Adaptive prefill fitting repair: learn from the smallest chunk, expire cold
  buckets, refit the slope when the intercept is negative.
  No sleeps, no watermark increase, no artificial decode-only cadence.
• Decode-KL fix: include the first divergent prediction (its input context is
  still shared). Boundary regressions and CPU CI wiring added.
• New recipe flashnext-leapquant-mtp2 (see Use).

Qualification runs

Two baseline and two candidate boots, bracketed (A1, C1, C2, A2), with one
rejected diagnostic (B1) between A1 and C1. This is not uninterrupted ABBA.
These runs came before the full matrix. They cover only the cells listed.
Setup for all runs: Flash-Next AWQ, MTP2, MonoKernel, FULL_AND_PIECEWISE, TP4.

• Baseline: FP32 state, 1024-token blocks, mixed-step budget off.
• Candidate: LeapQuant window 32, 512-token blocks, adaptive 450 ms
  mixed-step budget, decode share 0, max overhead 1.
• Sampling: T1, top-p 0.95, top-k 20, min-p 0, presence 0, repetition 1.
  The explicit T0 cell is greedy. Coding requests use the real thinking chat
  template and API seed 12345.
• Requests: 16 per boot (coding c8: 32, max 1024 output, EOS allowed).

Cell / mode	PP/req	PP wave	TG/req	TG aggregate	TTFT mean s	ITL p50 / p99 ms	MTP length
1k/512 c1 T1 / FP32 1024	2517.9	2466.7	88.01	82.38	0.407	23.68 / 24.21	2.077
1k/512 c1 T1 / LQ 512	2054.1	2013.7	91.32	84.01	0.499	23.60 / 24.23	2.158
32k/2k c8 T1 / FP32 1024	407.2	1799.3	20.21	82.32	80.564	44.58 / 434.91	2.281
32k/2k c8 T1 / LQ 512	536.4	1909.9	23.41	100.46	61.119	45.69 / 436.19	2.277
1k/512 c1 T0 / FP32 1024	2532.1	2466.0	85.94	80.62	0.404	23.48 / 24.05	2.021
1k/512 c1 T0 / LQ 512	2053.2	2015.4	88.93	81.98	0.499	23.42 / 24.07	2.084
coding c1 T1 / FP32 1024	2347.1	2216.4	82.20	79.55	0.425	23.89 / 24.38	1.965
coding c1 T1 / LQ 512	1926.1	1865.5	81.86	78.62	0.518	23.81 / 24.38	1.950
coding c8 T1 / FP32 1024	990.2	2795.3	39.13	288.97	1.008	47.46 / 368.37	1.993
coding c8 T1 / LQ 512	867.5	2469.0	39.22	288.86	1.151	46.39 / 273.60	1.995

Rates are tok/s. Rate and latency columns are means of two boot summaries.
Acceptance pools 1 + accepted/draft_steps. PP/req = input tokens / mean TTFT.
PP wave = effective concurrent wave / p99 TTFT (not a separately timed GPU prefill).
TG/req = 1 / mean TPOT.

Matrix (four boots: A1 FP32, C1 LeapQuant, C2 LeapQuant, A2 FP32)

Same build and protocol for all four boots: TP4 on V620, MTP2, MonoKernel, FULL_AND_PIECEWISE,
16 requests per cell, random prompts, ignore EOS, T1 / top-p 0.95 / top-k 20. FP32 uses block 1024
(the default flash-next recipe); LeapQuant uses flashnext-leapquant-mtp2 (block 512). One boot per
arm and cell, so each range is two single boots. c=4 is not in this matrix.

Cell	c	TG aggregate FP32 (2 boots)	TG aggregate LQ (2 boots)	Δ	TTFT mean s FP32 / LQ	ITL p50 ms FP32 / LQ	MTP length FP32 / LQ
1k/512	1	83.15-84.07	76.08-80.11	-6.6%	0.41 / 0.50	23.7 / 23.6	2.120 / 1.992
1k/512	8	247.07-280.67	268.35-268.99	+1.8%	1.31 / 1.59	48.9 / 47.1	2.143 / 2.159
16k/1024	1	60.39-60.96	60.14-60.83	-0.3%	5.97 / 6.05	24.2 / 24.1	2.266 / 2.264
16k/1024	8	112.81-113.23	104.35-112.61	-4.0%	19.78 / 20.86	50.3 / 48.7	2.305 / 2.164
32k/2048	1	59.07-63.28	62.22-63.28	+2.6%	12.20 / 12.29	24.3 / 24.3	2.333 / 2.437
32k/2048	8	85.23-89.93	105.01-105.14	+20.0%	78.73 / 64.62	44.1 / 45.6	2.373 / 2.449

PP/req and the full per-boot table are in bench_results/2026-11-14_leapquant-matrix/.

What the numbers show:

• 32k/2k c=8: +20.0% aggregate. The FP32 boots preempted (A1 10, A2 6); both LeapQuant boots preempted 0. This run changes block size and admission together, so it does not separate the two. There is no FP32 + new-scheduler control yet.
• 1k/512 c=1: -6.6% aggregate. Both LeapQuant boots are below both FP32 boots (76.08 and 80.11 versus 83.15 and 84.07). ITL p50 is equal (23.6 versus 23.7 ms); MTP length is lower (1.937 and 2.050 versus 2.135 and 2.105) and TTFT is higher.
• TTFT at 1k is +23% at c=1 and +21% at c=8 (about 0.09 s and 0.28 s). At 16k and 32k c=1 it is +1.3% and +0.7%.
• 16k/1k c=8: -4.0%. One LeapQuant boot (112.61) is 0.2 below the FP32 range (112.81-113.23); the other (104.35) is 7.5 below. Its MTP length is 2.09 versus 2.28 and 2.34 for FP32.
• Everything else is within the FP32 boot-to-boot range or a few percent either way. Boot-to-boot spread is large (for example 1k c=8 FP32: 247.07 and 280.67).
• Cause of the 1k c=1 acceptance drop: [INFERENCE] the lossy state lowers draft agreement at short context. Not tested.

Verification

• 22 GPU lifecycle tests pass: both windows, padded pages, rejected drafts,
  group normalization, prefix precopy, graph replay, MonoKernel.
• 40 CPU controller and scheduler tests pass (1 skipped). Three new adaptive-fitting
  regressions fail before the repair and pass after it.
• 384 qualification requests complete. 192 are real-coding requests with zero
  corruption-detector flags. Coding output is not scored for correctness.
• Decode-KL, 128 x 512-token probes per boot, 2,963 shared positions
  (97.8% probability mass): head 0.01398, floor 0.01262, ratio 1.107,
  95% bootstrap interval [0.836, 1.317], below the 1.5x threshold.
  This applies to these four boots only. It is not a full-trajectory or
  universal accuracy guarantee.
• Candidate preemptions 0/0. Baseline 14/10 (counters are disclosed; the
  zero-preemption gate applies to candidates only).
• No KFD eviction, GPU fault, thermal, build-overlap or quiet-window invalidation.
• Prefix, PPL, concurrency and after-load probes pass on both candidate boots.
• Rejected run B1: the first latency-cap trial locked at 64-token chunks.
  The repaired C1 run recovers 64 -> 2048 tokens without waits.
• Three baseline CPU fixture failures reproduced before this work are not
  claimed fixed. The whole compatibility suite is not claimed green.

Merge gates

• No candidate preemption, KFD eviction, GPU fault or thermal abort (peak junction 83-84 C). Loop flags on random outputs: FP32 17 and 17, LeapQuant 16 and 9 of 96 requests per boot, so no excess.
• MTP acceptance within the baseline spread: not met at 1k c=1 (both LeapQuant boots below both FP32 boots) and in one 16k c=8 boot.
• PPL 4.81 and 4.85 versus FP32 4.83 and 4.81; decode-KL from the qualification run above.
• No cell regresses beyond the baseline spread without a stated reason: not met at 1k c=1 (-6.6%); cause not established.
• The image build after the push to rdna_extras is green

The two unmet gates are accepted by the owner (2026-11-14): the 1k c=1 loss is the cost of the recipe.

Use

bash tools/rdna/serve_rdna.sh RECIPE=flashnext-leapquant-mtp2

The recipe sets LeapQuant on, 512-token blocks, MonoKernel, MTP2,
FULL_AND_PIECEWISE and the 450 ms budget. flashnext-mtp2 is unchanged
(LeapQuant off, 1024-token blocks).

Record: docs/rdna2/leapquant-gdn-2026-11-13.md.
Machine-readable data: bench_results/2026-11-14_leapquant-acceptance/.
Raw traces stay on the build box in ~/w4a8_runs/leapquant/.
Packing two LeapQuant layers per page is a separate second PR.

Credits

• Algorithm: LeapQuant (arXiv 2609.38166),
  Yi Pan, Haocheng Xi, Kan Zhu, Xingyang Li, Yibo Wu, Mayank Mishra,
  Hongtao Zhang, William X. Zheng, Baris Kasikci, Song Han, Kurt Keutzer,
  Rishabh Iyer, Ion Stoica.
• Record ring, cursor and per-window flush: ReplaySSM by Dao AI Lab and NVIDIA
  (vllm#48018,
  vllm#62004 by Johnny-Liou).
  The GDN decode kernel of [Kernel] ReplaySSM: cache SSM inputs for faster Gated DeltaNet standard decode vllm-project/vllm#62004 is not in this tree.
• Proposal: vllm#64725
  by lihaoyang-amd. It has no source code, and none is used here.
• The HIP kernels, layout header, FP32 oracle, scheduler changes and tests are
  original work for this fork. Existing files keep their upstream authors in git history.

[BlivionIaG]

ITL optimization priorities (recommendations, not implemented or GPU-qualified):

• c=1: fixed-512 serving has not been measured. In the saved 1024-block, 1k/512, MonoKernel-on ABBA, LeapQuant changes median ITL about 23.5 -> 23.3 ms, but mean acceptance about 2.152 -> 1.698 and output 85.42 -> 68.95 tok/s. Preserve/recover acceptance before optimizing mixed-batch scheduling, which cannot help true solo decode. Trace the first divergent draft/target decision; numerical causality is not isolated yet.
• Accuracy/performance candidates: the current state plus conv is 362.125 KiB in a 512 KiB page. Exact byte-layout calculations give 458.875 KiB for a 32-token residual window at rank 4, or 386.125 KiB for rank 8 at window 16. Both fit on paper, not proof of accuracy or speed. Longer windows reduce fold frequency but increase record replay. Higher rank needs HIP/MonoKernel implementation work: current kernels require rank 4.
• c>=2 with concurrent prefills: bound actual mixed-step prefill work using the existing MixedStepController, with decode_time_share=0 (no debt/cadence mechanism), while retaining full prefill-only batches and MTP2. The existing recipe leaves this controller opt-in; an older setting improved worst gaps but cost 13% output at 16k c8. Reuse and validate it, not another governor or a fixed global chunk-size reduction.
• Profile GDN folding/replay and materialize/store only after separating decode-only and mixed-step costs. Do not assume smaller pages or a larger accuracy window improve end-to-end speed.

No new GPU tests were started. Full acceptance/quality and 512 ABBA gates remain paused.

[BlivionIaG]

Two fixes are now pushed:

• 6d6fc19484: decode-KL includes the first divergent prediction, which is still conditioned on the same preceding tokens. Historical decode-KL figures are marked accordingly; three boundary regressions and a real CLI smoke pass.
• db40b66ae6: the adaptive prefill controller no longer ignores its own minimum-sized chunks or retains unvisited cold buckets forever; negative-intercept fits also refit the slope. Three failing-before regressions now pass. Controller + LeapQuant scheduler suite: 40 passed, 1 skipped.

The window-32/512-page candidate passes 22 GPU lifecycle tests. Its first full live run confirms recovery from a 64-token cap to 2048, zero preemptions/evictions/faults, and unchanged MTP2/FULL_AND_PIECEWISE. The earlier latency-cap trial is rejected (self-locked fitter, plus an overlong quiet window), not counted as a qualified result.

Still draft: repeating the repaired candidate and baseline before accepting performance/acceptance and the corrected decode-quality comparison. The reference incurred 14 preemptions; they are retained as reference behavior, while the candidate is still required to have zero.