Name and Version

version: 10068 (571d0d540)
built with AppleClang 21.0.0.21000101 for Darwin arm64

Verified still present on master (178a6c44): tools/server/server-context.cpp
has no commits between 571d0d540 and 178a6c44.

Operating systems

Mac (macOS 26.3, Apple M5 Pro, Metal)

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server -m Qwen3.6-35B-A3B-MXFP4_MOE.gguf \
  --host 127.0.0.1 --port 8080 -c 0 -ngl 99 --jinja \
  --slot-save-path ./kv

Problem description

POST /slots/{id}?action=restore reports complete success — n_restored equals
the full token count and the slot subsequently reports those tokens in
/slots — but the restored state is never reused. The next request with an
identical prefix reprocesses every token: cache_n = 0.

The endpoint therefore appears to work while delivering none of its benefit, and
says nothing about it at default verbosity.

On a hybrid/recurrent model this is not a partial regression, it is total: the
restore is worth exactly zero. Measured here on a 14,906-token prefix — restore
completes in 0.12s versus 75.9s to recompute, and then all 14,914 tokens are
recomputed anyway.

This is a different bug from the known in-memory checkpoint issues; see
"Relationship to existing issues" below. In-memory reuse on this setup is
healthy (97.9% hit rate) — only the on-disk path is broken.

Steps to reproduce

#!/usr/bin/env bash
# Requires: a hybrid/recurrent model (e.g. Qwen3.5/Qwen3.6, DeltaNet, Mamba)
# and llama-server started with --slot-save-path
H='Content-Type: application/json'
U=http://127.0.0.1:8080

# a prompt long enough to be worth caching
P=$(python3 -c "print('### POLICY\n' + '\n'.join(f'Rule {i}: verify precondition {(i*7)%89} then record outcome {(i*13)%83}.' for i in range(300)))")

req() { jq -n --arg p "$1" '{prompt:$p, n_predict:1, cache_prompt:true}'; }

# 1. cold — populates a slot
curl -s $U/completion -H "$H" -d "$(req "$P

Q: a?
A:")" | jq -c '.timings | {cache_n, prompt_n}'
#    → {"cache_n":0,"prompt_n":5529}

# 2. find the slot that actually holds it, and save
ID=$(curl -s $U/slots | jq -r 'map(select(.n_prompt_tokens > 0))[0].id')
curl -s -X POST "$U/slots/$ID?action=save" -H "$H" -d '{"filename":"t.bin"}' | jq -c '{n_saved}'
#    → {"n_saved":5529}

# 3. wipe every slot
for i in $(curl -s $U/slots | jq -r '.[].id'); do curl -s -X POST "$U/slots/$i?action=erase" >/dev/null; done

# 4. restore — reports full success
curl -s -X POST "$U/slots/$ID?action=restore" -H "$H" -d '{"filename":"t.bin"}' | jq -c '{n_restored}'
#    → {"n_restored":5529}

# 5. same prefix again, pinned to that slot
curl -s $U/completion -H "$H" -d "$(req "$P

Q: b?
A:" | jq --argjson s "$ID" '. + {id_slot:$s}')" | jq -c '.timings | {cache_n, prompt_n}'
#    → {"cache_n":0,"prompt_n":5529}   ← BUG: expected a large cache_n

Step 5 should reuse the prefix. It reprocesses all of it.

What is actually happening

The prefix match itself is correct. server-context.cpp:3149

n_past = slot.prompt.tokens.get_common_prefix(input_tokens);

returns the full 5,529 — restore does repopulate slot.prompt.tokens. The
result is then discarded a few lines later, at server-context.cpp:3284-3318:

if (pos_min >= pos_min_thold) {
    const auto it = std::find_if(
        slot.prompt.checkpoints.rbegin(),
        slot.prompt.checkpoints.rend(), ...);

    bool do_reset = it == slot.prompt.checkpoints.rend();     // :3300
    ...
    if (do_reset) {
        SLT_TRC(slot, "forcing full prompt re-processing due to lack of cache data ...");
        pos_next = 0;
        n_past   = 0;                                          // :3317
    }
}

A hybrid/recurrent model cannot partially rewind its recurrent state, so a
context checkpoint is the only way to roll back to a prefix — a requirement
extended from SWA-only to hybrid/recurrent in #16382. After a restore,
slot.prompt.checkpoints is empty, so do_reset is always true.

It is empty for two compounding reasons:

1. The save file never contained checkpoints. server-context.cpp:2529-2531
   serializes only the token list and the sequence's KV cells:

   const llama_tokens tokens = slot->prompt.tokens.get_text_tokens();
   const size_t nwrite = llama_state_seq_save_file(ctx_tgt, filepath.c_str(),
                                                   slot->id, tokens.data(), token_count);

2. Restore then clears whatever was there. server-context.cpp:2576-2577

   slot->prompt.clear();              // clears tokens AND checkpoints
   slot->prompt.tokens.insert(tokens);

So the on-disk path can never satisfy the checkpoint requirement introduced by
#16382. git log shows the save/restore payload has not been touched
substantively since 1d36b3670 (2025-05-02); #16382 landed 2025-10-03.

Notably the in-memory prompt cache added in #16391 got this right — it
stores a whole server_prompt, checkpoints included
(server-task.cpp: prompt = std::move(it_best->prompt);), and even accounts
for checkpoint bytes in server_prompt_cache_state::size(). That is precisely
why RAM reuse works while disk reuse does not.

Why this is invisible

The reset is logged with SLT_TRC → LOG_TRC → LOG_LEVEL_TRACE (= 4).
At default verbosity, and even at -lv 3, nothing is printed. Users see a
successful restore, a correct n_restored, correct /slots output, and silent
full reprocessing. -lv 4 is required:

slot   operator(): id  3 | task 10 | forcing full prompt re-processing due to
lack of cache data (likely due to SWA or hybrid/recurrent memory, see
https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)

Control experiment

To confirm checkpoints are the load-bearing mechanism rather than something
specific to restore — with no save/restore involved at all, only
--ctx-checkpoints 0:

1st (cold)  cache_n=0  prompt_n=5529
2nd (warm)  cache_n=0  prompt_n=5529   ← ordinary reuse also broken

Same log line, once. Two consecutive identical-prefix requests fail to reuse.
That is the same end state a restore leaves the slot in, reached by a different
route, and it confirms the causal chain.

Ruled out

(Incidentally: --no-kv-unified appears to be ignored while the slot count is
auto — kv_unified stays 'true' until --parallel is given an explicit
value. Possibly worth its own issue.)

Relationship to existing issues

• Misc. bug: Context checkpoints always invalidated on hybrid/recurrent models  #24055 (context checkpoints always invalidated on hybrid/recurrent) — open,
  in-memory only, no mention of /slots save/restore. Not this bug: our
  in-memory checkpoints work correctly (97.9% hit rate on the same model and
  build). This report is about checkpoints not surviving the disk round-trip.
• server: fix context checkpoint restore for hybrid/recurrent models (DeltaNet/Mamba) #22384 (closed), Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn #20225, Eval bug: [Bug] Qwen3-Coder-Next (Hybrid) Prompt Cache forces full re-processing due to 'invalidated context checkpoint' despite --swa-full #19794 — all in-memory checkpoint
  matching/creation on hybrid models. Same subsystem, different failure.
• Server forces full prompt re-processing on subsequent requests (SWA/recurrent memory error) #21831 — reports the same trace line, but for the in-memory path.
• feat: use llama.cpp slot state save/restore instead of text-based compression to eliminate re-prefill QwenLM/qwen-code#5760 proposes adopting /slots save/restore to eliminate
  re-prefill. Open and untriaged; it would not work today on these models.

Possible fixes

1. Sidecar file (no libllama format change): in the save handler, write
   filepath + ".ckpt" containing each common_prompt_checkpoint's
   n_tokens, id_task, pos_min, pos_max and the length-prefixed
   data_tgt / data_dft / data_spec blobs, behind a magic + version. In
   restore, rebuild slot->prompt.checkpoints after the prompt.clear() at
   :2576. A missing or version-mismatched sidecar degrades to today's behaviour.
2. Unify the two paths: serialize a whole server_prompt_cache_state so the
   on-disk format is a superset of the in-memory cache entry. Removes the
   divergence that produced this bug, at the cost of a larger change.

Either way, ctx_dft state is also not saved, so --model-draft setups will
desync after a restore.

Caveat for implementers: synthesizing a checkpoint at pos_min = 0 in the
restore handler is not a valid shortcut. The matcher would take the
cur.pos_min == 0 branch at :3298 and decode on top of a recurrent state that
has already consumed that token — silently corrupt output rather than merely
slow.

Reuse after a fix will be bounded by checkpoint granularity
(--ctx-checkpoints, --checkpoint-min-step), i.e. it should match in-memory
behaviour, not necessarily produce a full-prefix hit.