iOS: on-device compiled pipelined decode bundle produces corrupt output when the KV cache state is bound at seq >= 2048 (<=1024 correct; macOS correct at all sizes)

[john-rocky]

Summary

On iPhone 17 Pro (A19 Pro, iOS 27 beta, build 24A5380h), a GPU-pipelined S=1 decode bundle with a dynamically-sized KV cache generates corrupt output from the first token whenever the bound KV state's sequence dimension reaches 2048 or 4096. The same bundle and binary with KV capacity ≤1024 is token-exact against its fp32 oracle. The same shapes on macOS (M4 Max, macOS 27.0) are correct.

The corruption is accompanied by physically impossible throughput (~10× the weight-bandwidth floor) and a collapsed TTFT, which suggests the miscompiled specialization elides or mis-addresses the state reads, so the pipeline never stalls on them.

Reproduction model (public, pinned)

• Bundle: ukint-vs/Nanbeige4.2-3B-CoreAI @ 5864ec7, path gpu-pipelined/nanbeige4_2_3b_decode_int8hu_block32_sym_s1
• int8 LanguageBundle produced by coreai-core 1.0.0b2; static-S=1 input_ids [1,1]; dynamically-sized KV state [44, 1, 8, ctx, 128] (max_context 4096)
• Runtime: the CoreAILanguageModels GPU-pipelined engine (community 0.2.0-zoo runtime, but the failure reproduces beneath the Swift layer — see the cache-evict row below). COREAI_CHUNK_THRESHOLD=1.
• The engine pre-grows the KV cache to prompt + maxTokens before prefill, so maxTokens selects the bound state shape; the growing cache doubles 256 → 512 → 1024 → 2048 → 4096.

Result matrix (all: same device, same bundle, same 48-token prompt, temperature 0.7 unless noted)

Condition	Bound KV seq	Output	Decode tok/s	TTFT
maxTokens 150 / 200 (greedy) / 512	≤1024	coherent, correct	6.2–7.6	5.2–5.6 s
maxTokens 1024	2048	corrupt from token 1 (multilingual token soup)	55.5	0.65 s
maxTokens 2048	4096	corrupt from token 1	65.5	0.62 s
.fixedSize KV (4096 at engine creation), maxTokens 2048	4096	corrupt	58.2	0.71 s
Full Library/Caches evict, then fresh 24.8 s on-device compile, fixedSize 4096	4096	still corrupt	59.4	2.9 s
macOS M4 Max, maxTokens 1024 (greedy)	2048	token-exact vs fp32 oracle	55.5	—

Small-capacity correctness on the device is well established: the same bundle passes a 24/24 greedy token-exact gate vs the Mac engine reference (which is itself token-exact vs the fp32 eager oracle), reproduced across two full runs.

Sample corrupt output

Джерела随著 Джерела Джерела Джерела ... 参差不 vegg 佛罗伦 çà KDW 要知道 ...

(random valid vocab pieces at full pipeline speed, from step 1)

Why this matters

Any chat host that passes a generous maxTokens (e.g. a 2048-token response budget — a common default) silently crosses the shape cliff on iOS and produces garbage, while every "benchmark-sized" run (g=256) stays in the clean regime and looks perfect. We hit this in a shipping chat app and bisected it to the shape.

Ask

Is this a known AICode/MPSGraph specialization limit for large mutable-state bindings on device? Happy to provide raw logs, the exact repro commands, or run instrumented builds — the repro is fully public and deterministic.

[stikves]

Hello,

A few questions to help narrow down the issue

1 - Does this happen with the sequential engine as well (coreai-sequential, which can be requested in the CoreAILanguageModel.init(..., variant: "coreai-sequential")

2 - Are the versions of macOS, iOS and Xcode are aligned (all of them Beta 4, or similar)?

3 - Is this error encountered only for this specific model / quantization combination?

Once again thank you for the bug report.