13-prefix-cache

13 — System-Prompt Prefix Caching

Status: ✅ Already working — no action needed. llama.cpp's built-in prompt cache caches the ~600-token system prompt across requests automatically.

Measurement (2026-06-22)

measure_cache.py against running prod (IQ3_XXS, -c 65536 --cache-ram 5120 --parallel 16):

Scenario	Wall time	Prompt eval
Cold (first request with this SP)	454ms	350ms
Warm (SP prefix cached)	126ms	38ms
Best-case (identical payload)	130ms	39ms
Control (no system prompt)	117ms	29ms

The system prompt's 538 tokens cost 350ms cold but ~38ms warm — a 9× speedup on prompt eval. Warm latency (126ms) is within 9ms of the no-SP baseline (117ms). The cache means the bot effectively only pays for the ~10 new user tokens per turn, not the full SP reprocessing.

Why it works

• llama.cpp's prompt cache (--cache-ram <MiB>) stores processed prompt KV state in RAM, keyed by longest-common-prefix similarity.
• Server logs show f_keep = 0.92 (92% of a prior prompt reused) and sim_best = 0.95 on typical bot requests — the SP is the common prefix.
• --cache-ram 5120 (set 2026-06-21) gives 5 GB of cache room — far more than needed (SP is ~13 MB of KV).

Config

Already in scripts/start-llama.sh:

-c 65536 --cache-ram 5120 --parallel 16

No further tuning warranted. The cache hit rate is ~92%+ on real bot traffic.

Reproduce

python3 research/13-prefix-cache/measure_cache.py