Investigate rewriting the #544 spec conv-state path (causal_conv1d spec decode) without its performance cost

[CySpiegel]

Summary

Upstream vllm-project#544 ("Fix token-indexed conv-state layout in causal_conv1d spec-decode kernel") introduced the sliding-window conv-state convention for the spec-decode causal_conv1d path. Its epilogue rebuilds the state line in place after the main loop (aliased same-pointer shift, runtime-bounded loop with a per-iteration branch, K redundant global re-reads of mixed_qkvz), once per GDN layer per decode step under MTP serving. This issue tracks its cost so it is not lost, and what is still open.

What was measured (2x Arc Pro B70, TP=2, Qwen3.8-27B INT4 + MTP k=1; bench ledger A16/A17)

level	pre-vllm-project#544	vllm-project#544	vllm-project#544 + register writeback (vllm-project#551)
end-to-end greedy TPOT, same session	14.47 ms	14.46–14.48 ms	14.46 ms
per-step decode (median ITL), all eras	24.4–24.9 ms	24.4–24.9 ms	23.3–23.4 ms today
batched conc-32 out tok/s	211.0 (08-25)	—	208.6 / 209.1 (08-27)
kernel microbench (benchmark_causal_conv1d, spec workloads)	baseline	—	-1..-4% real shapes, -11% synthetic

End-to-end the epilogue is neutral (µs scale); the 13.76 -> 14.5 ms "decay" initially attributed to vllm-project#544 was environmental (torch-compile cache regeneration; ledger A16 addendum 3, A17). The kernel-level overhead is real and is removed by upstream PR vllm-project#551 (bit-identical state, 1303/0/128 matrix).

Open items

• This fork currently carries its own port of [GDN] Fix token-indexed conv-state layout in causal_conv1d spec-decode kernel vllm-project/vllm-xpu-kernels#544 (1f30181) plus mixed-batch support (06e0d7f). The pending kernels rebase must drop the port in favor of upstream's 5802a41 and reconcile 06e0d7f with upstream [GDN][MTP] fix split PR bug  vllm-project/vllm-xpu-kernels#537 — then re-measure per-step ITL (greedy, --save-detailed) against today's 23.3 ms / 209 t/s to confirm upstream's version carries no additional cost.
• If [GDN] Write spec conv-state from registers instead of epilogue roll vllm-project/vllm-xpu-kernels#551 is not accepted upstream, keep the register-writeback epilogue on the fork and re-benchmark after each kernels sync.
• Measurement rule for anything MTP-related: compare per-step ITL with --temperature 0; never bisect on the T=1.0 TPOT (acceptance noise +-2 ms).

[CySpiegel]

Investigation scope (why this issue exists)

Goal: determine whether the vllm-project#544 conv-state convention can be reimplemented without its cost, not merely tracked.

What is already done

• [GDN] Write spec conv-state from registers instead of epilogue roll vllm-project/vllm-xpu-kernels#551 (upstream PR, this fork) rewrites [GDN] Fix token-indexed conv-state layout in causal_conv1d spec-decode kernel vllm-project/vllm-xpu-kernels#544's epilogue: history rows and draft inputs are written from registers at priming/load time, epilogue removed, state bit-identical. That covers the epilogue's aliased shift / branchy loop / K redundant mixed_qkvz reads. Kernel-level -1..-4% on real shapes.

What is still open — the layout itself
vllm-project#544 makes the spec conv state a sliding window of width - 1 + (K - 1) rows with the accepted window starting at row num_accepted - 1. Costs to quantify that vllm-project#551 does not touch:

1. The roll/re-read of the window on the next decode step (state consumer side).
2. The token-indexed interface path: [GDN] Fix XE2 delta epilogue OOB write with non-contiguous token_indx vllm-project/vllm-xpu-kernels#552 stages the XE2 epilogue output and scatters per row via index_copy_ (measured zero cost at engine level, but it is an extra pass).
3. Whether a fused conv + state-update kernel for the spec path can skip the window materialisation entirely (write only the rows the next step reads).

Sizing the prize (from ledger A16; re-measure on the rebased kernels)

• Spec causal_conv1d kernel: ~64-72 us/call recorded. x48 GDN layers = ~3.1-3.5 ms of a 23.3 ms decode step (~14%).
• [GDN] Fix token-indexed conv-state layout in causal_conv1d spec-decode kernel vllm-project/vllm-xpu-kernels#544's epilogue is a few percent of that call; end-to-end neutral (see table above). A rewrite only pays off if it attacks the whole spec conv path, not just the epilogue.

Method

• benchmark_causal_conv1d spec workloads on BMG for per-call numbers; vllm bench serve greedy --save-detailed per-step ITL for end-to-end (T=0 only; MTP acceptance noise otherwise).
• Correctness bar: tests/gdn_attn full matrix + the mixed-batch bitwise differential (12/12) — the state must stay bit-identical to the reference chaining.
• Do this on the rebased kernels tree (upstream 5802a41 [GDN] Fix token-indexed conv-state layout in causal_conv1d spec-decode kernel vllm-project/vllm-xpu-kernels#544 + [GDN][MTP] fix split PR bug  vllm-project/vllm-xpu-kernels#537), so the baseline is upstream's code, not the fork's port 1f30181.