GLM: opt-in RAM/disk conversation slots for interleaved agents

.github/workflows/tests.yml

@@ -45,5 +45,7 @@ jobs:
           "$RUNNER_TEMP/ds4_cpu_prepare_stub_test"
       - name: Server tests
         run: '"$RUNNER_TEMP/venv/bin/python" -m unittest discover -s serve -p "test_*.py" -t .'
+      - name: GLM conversation-slot policy (CPU only)
+        run: '"$RUNNER_TEMP/venv/bin/python" tools/test_glm_slot_spill.py --receipt "$RUNNER_TEMP/glm-slot-spill.json"'
       - name: Installer and tools tests
         run: '"$RUNNER_TEMP/venv/bin/python" -m unittest discover -s tools -p "test_*.py"'

docs/GLM_SLOT_SPILL.md

@@ -0,0 +1,67 @@
+# Opt-in RAM and disk conversation slots
+
+Concurrent agents often interleave more conversations than the engine has active contexts. An agent may pause for a tool or another task and then return to the same token prefix. With the legacy count cap, that conversation can already have been discarded even when host memory and disk have room. This setting lets the conversation remain in RAM, then spill to disk under separate byte budgets, reducing repeated prefill when it returns.
+
+This is a capacity and reuse feature. It does not add decoding workers, change the scheduler, share arbitrary agent memory, or make unrelated prefixes reusable. It does not change weights, routing, sampling, or the token-prefix/image compatibility check.
+
+## Configuration
+
+`MAYA_GLM_SLOT_SPILL` is **off by default**. Set it to `1` to opt in; leave it unset or set it to `0` for the existing count-capped behavior.
+
+| Setting | Behavior when spill is enabled |
+|---|---|
+| `MAYA_GLM_SLOT_RAM_GB` | RAM state budget in GiB; 8 when unset |
+| `MAYA_GLM_SLOT_GB` | Existing disk state budget in GiB; 16 when unset |
+| `MAYA_GLM_SLOT_DIR` | Existing slot directory; choose a private, writable directory appropriate to the deployment |
+| `MAYA_GLM_SLOTS=0` | Disables slots, including spill |
+
+The deployment chooses both budgets; no machine's RAM capacity or deployment-specific cache allocation is embedded in the implementation. These are retained-state budgets, not preallocated memory or a limit on total process RSS. Serialization can temporarily need another state buffer; token metadata and other engine allocations also need headroom. The existing free-disk guard remains in place. The separate DSA prompt-cache path still takes precedence where it disables conversation slots.
+
+RAM pressure spills the least recently used RAM state. Disk pressure evicts the least recently used disk state, including a state just spilled from RAM. Moving a state is not a use and preserves its last-use timestamp across restart. A state larger than the disk cap, including a zero cap, is rejected before writing or evicting another file. If a state cannot be retained, its next request reads that prefix again.
+
+Spill mode retains the folder across runs and flushes RAM states on normal CLI/serve exit. A crash or forced kill does not guarantee persistence. When shutdown has more state than the disk budget, older states can be dropped. This is a bounded cache, not durable archival storage. Miss logs identify no candidate, divergent tokens, an insufficient prompt length, or changed images without printing prompt text.
+
+## Current standalone branch: host verification
+
+Base: upstream `7e68a30f2f208056195352129188661fb6b94445`, without the separate development branch's NUMA/shared-pool and dynamic-DFlash overlays.
+
+On 2026-11-15, Windows/MSVC 14.50 completed **17/17 host scenarios** and compiled the complete modified `generate.cpp` translation unit. The latter is a host object build, not a CUDA build or final engine link. The source and harness hashes, each scenario's actual output/assertions, and exit codes are in [the host receipt](testing/glm-slot-spill-host.json); only temporary filesystem paths were normalized.
+
+The harness extracts the actual slot structure and management lambdas, plus the actual CLI success tail, from `generate.cpp`. Only the model save/load surface uses deterministic byte buffers. Coverage includes:
+
+- RAM pressure and disk LRU, including cold spill versus a recently used file;
+- oversized/zero budgets, a discarded snapshot's retained flag, and failed writes;
+- normal-return flushing, CLI exit, restart indexing and exact saved-byte recovery;
+- mixed RAM/disk last-use order across restart;
+- the runtime miss reasons and refusal to load invalid prefixes;
+- legacy disk/RAM-only count caps, an unset opt-in, and configurable fallback budgets.
+
+Run it with a C++20 compiler, with no model or GPU:
+
+```sh
+python tools/test_glm_slot_spill.py --receipt glm-slot-spill.json
+```
+
+On Windows, run from an MSVC developer shell. On POSIX, the default compiler is `c++`. `--cxx` or `CXX` selects an executable; no local Visual Studio path is hardcoded. The normal GitHub CPU-only workflow also invokes the harness. Fixture cleanup is confined to its fresh temporary directory.
+
+## Historical evidence reused, not rerun
+
+The following measurements belong to the earlier v1.1.63 development/deployment build, not this standalone PR head. They used a GLM-5.3-Flash Maya-L pack, four RTX 4090s at 255W each, about 251.3GiB host RAM, 144 CPU lanes, four active contexts, a 1M context limit, 5120-token prefill chunks and layer split `12,24,35`. That branch included separate NUMA/shared-pool and dynamic-DFlash changes. [The historical receipt](testing/glm-slot-spill-prior.json) records exact source/binary identities, explicit test budgets, outputs, and version differences.
+
+| Window | Evidence | Boundary |
+|---|---|---|
+| Pre-fix host regression | 6/15 passed; nine expected failures | Negative control for the original spill prototype; not upstream `main` |
+| Repaired host regression | 15/15 passed | Deterministic host state bytes, not real GPU/model state |
+| Repaired deployment | Linux/CUDA incremental link succeeded; runtime executable hash matched | Earlier deployment binary, not a build of this standalone PR |
+| Repaired deployment smoke | Seven requests all returned `OK`; returning to the first 1470-token prompt restored 1469 tokens from RAM; editing its beginning restored 0 | Five different short conversations, return, then negative control; natural production traffic interleaved |
+| Earlier capacity observation | After nine different 9236-token conversations, returning to the first gave 4.480s / 0 cached tokens with legacy slots, versus 0.202s / 9235 cached tokens with RAM spill | Pre-fix prototype, a single synthetic return; not an isolated benchmark or a production average |
+
+The five-conversation version of the historical capacity check did not exhaust four active contexts plus four legacy slots. It therefore did not answer the eviction question. Large client delays in other historical rows included production queuing and are not prefill comparisons. These observations motivate retaining more conversations; they do not establish an average speedup for a real multi-agent workload.
+
+The standalone branch deliberately reuses the existing 16GiB disk fallback instead of the prototype's 32GiB fallback. Explicit environment budgets still override it. The historical smoke supplied explicit budgets, so its allocation is not a PR default.
+
+## What remains unverified
+
+No connection to the model host, model load, inference request, benchmark, or production change was performed to prepare this PR. There is no new `tools/agent_bench.py` comparison, decode/prefill throughput claim, full quality suite, exact PR-head token-parity measurement, or PR-head CUDA link. No local WSL was started.
+
+The historical model smoke hit RAM and did not force disk overflow, restart and real-model disk restoration. Full-budget memory pressure, crash recovery, and production-wide multi-agent hit rate/latency remain unmeasured. Host byte tests cover policy and persistence ordering, not those model-level boundaries. The existing deployment was left running while the standalone PR was prepared.

docs/SETTINGS.md

@@ -108,6 +108,7 @@ These settings change what the engine chooses (put them in the config with `--en
 | `MAYA_GLM_MULTI_CKPTS` | `0` (off) | `--parallel` above 1: `1` keeps the conversation checkpoints above for each conversation at once (#711) - in its own context, with its sequence state, so a request that leaves its conversation before the snapshot's position (an edited or regenerated last message, a cancelled answer sent again without it, a client that rewrote or trimmed an older turn) restores the longest checkpoint it starts with and reads only the rest, where it read from the system prompt's slot (`MAYA_GLM_PREFIX_CUT`) or from token 0. A request goes to the idle context whose snapshot or checkpoint it extends furthest, else the least recently used one, as before. Each context keeps up to `MAYA_GLM_CKPTS` (fewer when the contexts are many: 32 in all, ~150 MB of RAM each for GLM-5.3-Flash - 2.4 GB for four conversations at the default 4); the periodic ones (`MAYA_GLM_CKPT_EVERY`) stay off here unless that is set, since each one ends a prompt-path call while the others wait to decode. The system prompt's slot is still kept (another context restores it). Text prompts only |
 | `MAYA_GLM_SLOT_MIN`, `MAYA_GLM_SLOT_GB`, `MAYA_GLM_SLOT_DIR` | 1024, 16, `<pack>/slots` | the shortest conversation kept aside (tokens, down to 1), their total size on disk (GB), and the folder |
 | `MAYA_GLM_SLOT_KEEP`, `MAYA_GLM_SLOT_DAYS` | off, 7 | `1`: the kept conversations outlive the engine - the folder is not emptied at start (a slot is kept when its `.meta` names this model, engine version and pack, else removed), the conversation in the model is kept when the engine ends, and a slot unused for `MAYA_GLM_SLOT_DAYS` days is removed. With a supervisor that stops the model when it idles (llama-swap), the next request after a restart reads only its new tokens (a 4K-token conversation back in 0.2 s instead of re-read) |
+| `MAYA_GLM_SLOT_SPILL` | off | `1`: hot conversation states stay in RAM (`MAYA_GLM_SLOT_RAM_GB`, 8 GiB when unset) and the least recently used is written to `MAYA_GLM_SLOT_DIR`. Retained state files use the existing `MAYA_GLM_SLOT_GB` cap (16 GiB when unset); both budgets are configurable, and a single state larger than the disk cap is not written. Disk eviction uses last use, including the state just spilled from RAM; moving a state to disk preserves its last-use timestamp for restart and age limits. The folder is kept across runs, and RAM states are written out on normal exits in both CLI and serve modes (not guaranteed on a crash or forced kill). Miss notes distinguish no candidate, token divergence, a prompt that is not longer than the saved prefix, and changed images, without the prompt text; evictions also log that the state is no longer retained. The count cap (`MAYA_GLM_SLOTS`) does not delete while this is on; `MAYA_GLM_SLOTS=0` still disables slots. Unset keeps the previous count cap. `MAYA_GLM_SLOT_RAM_GB` without this still parks every slot in RAM and drops past the cap. See [validation and limits](GLM_SLOT_SPILL.md) |
 | `MAYA_GLM_SEQS`, `MAYA_GLM_SEQ_CTX` | 1, the context | the first is `--parallel` (above): n conversations at once, n sequence states - the first for the whole context, the others for `MAYA_GLM_SEQ_CTX` tokens. On a split a token of each goes through the GPUs one behind the other, so every GPU works on a different conversation (4x RTX 4090, Maya-S24: 110 tok/s alone, 132 in all for two, 225 for four); each extra state takes its size from the expert pool (three at 64K: 3.5 GB) |
 | `MAYA_GLM_SPLIT_ROWS` | `0` (off) | Multi-GPU decode only: `1`–`8` batches up to this many concurrent requests on each part. Each group shares dense weight reads while sequence-specific attention and routed experts remain per-request; groups pipeline across the split. Each part allocates row activations before sizing its expert pool (about 8 MiB per GPU). Unset or `0` preserves the existing pipeline. On four RTX 4000 Ada GPUs with Maya-M and eight requests, `2` measured 94.5 tok/s aggregate (+27% over the 74.42 tok/s legacy median); larger groups were slower. Single-request decode and prefill stayed within ±5% in the matched check. |
 | `MAYA_GLM_EVEN_GROUPS` | `0` (off) | With `MAYA_GLM_SPLIT_ROWS` = k: `1` cuts the running requests into max(2, ceil(n / k)) groups of near-equal size (k at most) instead of filling groups of k in turn, so k or fewer requests no longer make one group that leaves every GPU but one idle (with k = 4: two groups of 2 at four requests, two of 4 at eight). The groups are sized from all running requests, so a group that comes back from the split is not cut again. The multi bench's split-rows path uses the same groups |