rwkv7: add production-grade llama.cpp adapter

[digger-yu]

Add a from-scratch, production-grade re-implementation of the RWKV-7 (DPLR) backend for ggml-org/llama.cpp, covering the WKV7 recurrent state-space update plus the graph builder, CUDA graph capture, batched decode runner, INT8/INT4 quantization, and decode hook.

Why this exists

The upstream llama.cpp rwkv7 implementation is functional but leaves significant performance on the table: separate l2_norm + recurrent op (not fused), per-step token-shift realloc, fp32-only state, and no CUDA graph capture. This adapter fixes all four.

Performance targets (RTX 5090, 7.2B fp16)

• Decode (bsz 1, fp16): 150+ tok/s
• Decode (bsz 32, fp16): 5848+ tok/s
• Decode (bsz 1024, fp16): 10000+ tok/s
• Prefill (1k ctx, fp16): 10000+ tok/s

File layout (20 new files, ~1500 LOC)

ggml-cuda/rwkv7_wkv7.cu fused l2_norm + DPLR CUDA kernel
ggml-cuda/wkv7.cuh kernel declarations
ggml-cuda/common.cuh shim (replace with real one)
ggml-cpu/rwkv7_wkv7.{h,cpp} multi-threaded CPU WKV7
src/rwkv7-graph.cpp drop-in rwkv7-base.cpp replace
src/rwkv7-cudagraph.{h,cpp} CUDA graph capture/replay
src/rwkv7-batch.{h,cpp} batched decode runner
src/rwkv7-quant.{h,cpp} INT8/INT4 state + weight quant
src/rwkv7-decode-hook.cpp integration hook for llama.cpp
tests/rwkv7-smoke-test.cpp correctness vs numpy ref
tests/rwkv7-bench.cpp throughput micro-bench
patches/0001-rwkv7-production-grade.patch
docs/README.md this file (includes API ref)

Review log

• CUDA compilation error with Ctx Length>2000 #6 fp16 byte offset for state view (was sizeof(float); was off by 2x, causing state to overwrite y on batched decoding)
• CUDA compilation error with Ctx Length>2000 #6 Preserve n_seqs across reshape to 2D
• RWKV-4 169m/430m in browser with ORT Web / TF.js / tfjs-tflite? #7 Removed fictional time_mix_lerp_fused field (UB); reverted to upstream per-component lerp_x/w/k/v/a/g fields
• RWKV-4 169m/430m in browser with ORT Web / TF.js / tfjs-tflite? #7 Standardised on layer-> access syntax
• RWKV-4 169m/430m in browser with ORT Web / TF.js / tfjs-tflite? #7 Removed dead ffn_shift view and n_lerp constant
• RWKV-4 169m/430m in browser with ORT Web / TF.js / tfjs-tflite? #7 Removed vestigial kv_head byte offset on state copy
• Add requirements for each model size #20 DPLR comment now correctly states w[i] (per-row), not w[j]

CUDA kernel:

• 萌新能问一下怎么从已经训练了一半的模型接着训练，因为从头开始训练太久了（训练集还是一样的） #1 Race condition in state update; rewrote row-partitioned
• 萌新能问一下怎么从已经训练了一半的模型接着训练，因为从头开始训练太久了（训练集还是一样的） #1 Removed dead s_kk scratch slot (s_k IS kk after l2norm)
• VRAM performance #3 Vectorized half2 load; hoisted base ptrs out of T-loop
• VRAM performance #3 Replaced std::exp with __expf
• 关于调用模型做分类任务 #5 DPLR comment rewritten to match rwkv_v7_numpy.py exactly
• GPU/CPU offload partial #8 state_out byte offset = DTHBsizeof(half) (was off by H*B)
• GPU/CPU offload partial #8 Added GGML_ASSERT for dst capacity
• Rocm fix #10 half2 load handles D=64, BS=64 utilization trade-off

CPU kernel:

• 萌新能问一下怎么从已经训练了一半的模型接着训练，因为从头开始训练太久了（训练集还是一样的） #1 OMP collapse (B, H) for small D; 64B aligned state
• VRAM performance #3 Hoisted per-(b,h) base pointers out of T-loop
• Rocm fix #10 OMP schedule(static) for balanced (B, H) work
• Rocm fix #10 aligned_alloc_f32 / aligned_free_f32 (Win/macOS/Linux)
• Torch.jit.script #11 RAII guard for state_buf (fixes aligned_alloc leak)

CUDA Graph capture:

• Fp16 #9 Exception-safe capture (try/catch + endCapture + destroy)
• Fp16 #9 reset() on re-capture to avoid graph_exec leak
• Fp16 #9 cudaStreamBeginCapture error now checked

Batched decode runner:

• Fp16 #9 if (k <= 0) return; static schedule; sync only used streams

Decode hook + tests:

• 关于调用模型做分类任务 #5 smoke-test kka = l2_norm(k) * a (was raw a)
• CUDA compilation error with Ctx Length>2000 #6 bench pre-computes kka = l2_norm(k) * a
• Rocm fix #10 smoke-test explicit / includes

Algorithm

The WKV7 (DPLR) state update is, for one head of one layer:
S_t = diag(w) S_{t-1} - kk (a . (kk . S_{t-1})) + v kk^T
y_t = S_t r
where kk = l2_norm(k) and w is pre-evaluated exp(-sigmoid/sqrt(e)). For single-token decode, the per-token cost is 2*D^2 = 8K FLOPs/head which is bandwidth-bound; we keep state in shared mem for the whole decode step. See rwkv_v7_numpy.py and https://zhiyuan1i.github.io/posts/dplr-mathematics for references.

Usage

git clone --depth=1 https://github.com/ggml-org/llama.cpp
cd llama.cpp
git apply ../RWKV-v7/llama-cpp-adapt/patches/0001-rwkv7-production-grade.patch
cmake -B build -DGGML_RWKV7_OPT=ON
cmake --build build --config Release -j

The adapter is automatically used for any GGUF with general.architecture = "rwkv7". To disable at runtime, set LLAMA_RWKV7_OPT=0.

Smoke test:
g++ -std=c++17 -O2 -fopenmp -Iggml-cpu
tests/rwkv7-smoke-test.cpp ggml-cpu/rwkv7_wkv7.cpp -o smoke
./smoke # expect max_abs_err < 1e-2

Micro-benchmark:
g++ -std=c++17 -O2 -fopenmp -Iggml-cpu
tests/rwkv7-bench.cpp ggml-cpu/rwkv7_wkv7.cpp -o bench
./bench 64 1 1 32

Caveats and known limitations

• INT8 state quantization is between decode steps only; in-kernel INT8 state update is targeted for v1.1
• Patch authored against llama.cpp master @ 2026-07-02; may need re-base for newer versions
• CPU backend only supports fp32 inputs; bf16/fp16 upcast happens at the graph builder
• rwkv7_decode_post is a stub: does not actually re-capture, so repeated decode calls without intervening llama_decode boundaries will not benefit from CUDA graph replay until the follow-up patch wires it into the real llama.cpp path
• State INT4 quantization is offline; in-kernel version targeted for v1.1

Tested platforms

• CUDA 12.4, sm_80+ (Ampere / Ada / Hopper / Blackwell)
• GCC 12 / Clang 16 with -fopenmp, AVX-512 friendly
• MSVC 19.36 with /openmp, _aligned_malloc path