feat: keep models in memory, offload them when idle, and give whisper.cpp a resident worker

[Mr-Sunglasses]

Summary

Three related changes to when a model is in memory.

Model memory lifecycle — Settings → Speech engine gains Offload model when idle (off by default): a resident model is released after 10, 15, 30, 60, or 120 minutes without a transcription and loads again on the next request. Engines that own a loaded model share a MemoryResidentEngine protocol, so the idle monitor unloads faster-whisper, sherpa-onnx, Moonshine, MLX Audio, WhisperKit, and VocaMac's persistent path through one interface. Batch and streaming requests take an engine lease for their duration, so the monitor never unloads a model that is in use. The Overview model-cache card and a Models badge report the offloaded state, and the choice is persisted in the runtime config and carried by /v1/admin/config.

Leases now protect an engine swap as well. A settings change used to close the previous engine immediately, which could unload the model out from under a request still transcribing with it — a lease keeps the engine object it was handed. A replaced engine is retired and closed on the release of its last lease; one that nothing holds is closed at once, as before. And configure() compares the settings that actually decide which engine gets built — engine, device, precision, thread count, and each model selection — so choosing an offload interval no longer discards the very model the setting is about.

A resident whisper.cpp worker — whisper-cli reloads the model and rebuilds the accelerator context on every run, which dominates a short dictation clip. When the build ships whisper-server beside whisper-cli, the engine starts it on an ephemeral 127.0.0.1 port, keeps the parsed model and the GPU context loaded between requests, and transcribes over /inference. Nothing extra is published and no host port is added. Warmup starts the worker, so Load brings the model into memory rather than only warming the page cache, and idle offload now reaches whisper.cpp too.

Measured on an M1 Pro with ggml-tiny.en and a real whisper-server, a warm request drops from ~300 ms to ~70 ms; the saving grows with the model size and with a CUDA or Vulkan context, where building the context is the expensive part.

Nothing regresses on a host without the binary: a missing whisper-server, a build that cannot serve, or a worker that dies mid-request falls back to one whisper-cli run per transcription, exactly as before. A port lost between reserving it and binding it is retried on a fresh one so a race does not retire the fast path, and a read timeout is reported as a slow transcription rather than a dead worker (which would otherwise re-run the same audio and double the wait).

The worker holds up under a request that goes wrong — the client is written on asyncio streams rather than urllib in a thread, because a blocking client cannot be cancelled: the socket stayed open until the thread returned, so a timed-out or cancelled clip left the worker decoding audio nobody wanted and the next recording queued behind it. Closing the socket is exactly what the pinned server watches — it passes is_connection_closed() to whisper.cpp's abort callback — so the decode now stops with the request. A worker that does not answer again within a short grace period is killed and reloaded before the next clip rather than serving it late; it is killed rather than asked politely because whisper-server catches SIGTERM and shuts down gracefully, which means waiting out the decode it is running.

A start records the process the moment it is spawned, so cancelling a load can no longer leave a process reading a whole model into memory with nothing left to stop it, and readiness polling is asynchronous for the same reason. One deadline covers the whole request: the worker start, the inference, and any CLI fallback share it instead of each spending a full timeout while the caller waits.

The worker's stderr is captured instead of discarded — that is where whisper.cpp names the ggml backend it selected and the thread count it got — and its tail is reported when a build cannot serve the model. Transcripts never reach it: the segment printer writes to stdout, and only under --print-realtime, which the gateway does not pass.

Decoding parity and a knob for measuring it — the server path had lost the CLI's -nt. whisper-server defaults to no_timestamps=false and, with token_timestamps unset, derives it as !no_timestamps, so response_format=text only hid the timestamps: the worker still decoded them and still ran the token-level pass over the signal. Both fields now travel with every request, and -nt is passed at launch so a default a request somehow misses is already the gateway's.

VOCAGATEWAY_WHISPER_DECODER_PRESET adds what a CPU decoding comparison needs: fast decodes greedily (whisper.cpp picks the greedy sampler whenever the beam size is not above one), quality keeps the narrowed beam search and stays the default until a word error rate comparison on the target machine says otherwise. Temperature fallback stays on in both.

Behavior note: transcripts are now collapsed to a single line on both whisper.cpp paths — the server prefixes each segment line with a space, and the CLI already emitted one line per segment.

Verification

• just test equivalents: ruff check, ruff format --check, flake8 --select=WPS,E999 app, mypy strict, 571 tests, docker compose config, uv sync --locked --check and uv pip check
• Container build, if Dockerfiles or lockfile changed — not run locally; the Dockerfile change adds whisper-server to the binaries copied out of the existing whisper.cpp build, and whisper-server is a build target of the pinned v1.9.1. The Container workflow builds all three accelerators on this PR.
• Docs updated for setup, network, or configuration changes — README (env table, readiness, idle offload, lease-protected swaps), docs/configuration.md, docs/troubleshooting.md, .env.example, compose.yaml, and the Settings hint

Checked against the pinned whisper.cpp v1.9.1 source: -bs/-bo, --host/--port and -nt exist; the per-request language, response_format, no_timestamps and token_timestamps fields are read; token_timestamps defaults to !no_timestamps; auto is accepted; inference is serialized behind the server's own mutex, which the handler takes before it looks at the request; and abort_callback is req.is_connection_closed().

Measured end to end against a real whisper-server (1.9.2, ggml-tiny.en), not the test stand-in:

	measured
Timestamps decoded vs. disabled, 10 s clip	69.8 → 67.9 ms median (~3%; decoder-side, so worth more on a CPU-only host and a larger model)
Beam 2/best-of 2 vs. greedy, same clip	75.3 → 67.9 ms, equivalent transcript
A 3.0 s decode cancelled at 0.4 s, then a short clip	69 ms — the abort is honoured
Liveness probe, idle worker vs. mid-decode	1 ms vs. 2270 ms — the probe really does wait on the model mutex
stop(), idle worker vs. mid-decode	18 ms vs. 2760 ms — which is why a wedged worker is killed, not asked
Cold start, flat 100 ms readiness poll vs. 5 ms with backoff	205 → 185 ms

Also verified live that omitting the language field decodes as English whatever the audio is, which is why it is sent on every request.

Privacy and security

• No secrets, recordings, transcripts, or private hostnames added — the captured worker stderr carries backend and model-load lines only; whisper.cpp writes transcripts to stdout, and only under a flag the gateway never passes
• No weakening of bearer auth, upload limits, retention, or default binding without discussion — the worker binds an ephemeral loopback port inside the gateway's own process namespace, is never published, and carries no additional auth surface

[netlify]

✅ Deploy Preview for voca-gateway canceled.

Name	Link
🔨 Latest commit	73dfe55
🔍 Latest deploy log	https://app.netlify.com/projects/voca-gateway/deploys/6a9ba0a1b298e500086d2556

[greptile-apps]

Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.

[greptile-apps]

Mr-Sunglasses has reached the 50-credit limit for trial accounts. To continue receiving code reviews, upgrade your plan.