lobes never advertises a capability it cannot serve (#92 · #91 · #95 · #97 · #96 · #74 · #69) (#102)

Author: OriNachum

.devague/current

@@ -1 +1 @@
-lobes-preserves-qwen-thinking-traces-across-multi
+lobes-never-advertises-a-capability-it-cannot-serv

.devague/current_plan

@@ -1 +1 @@
-lobes-preserves-qwen-thinking-traces-across-multi
+lobes-never-advertises-a-capability-it-cannot-serv

.devague/frames/lobes-never-advertises-a-capability-it-cannot-serv.json

@@ -4,6 +4,42 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.40.0] - 2026-07-09
+
+### Security
+
+- The `Host` request header is validated against a strict host-authority allowlist before it is echoed as an advertised origin in `GET /capabilities`. Previously the c29 origin-resolution change reflected an unsanitized, client-controlled `Host` header into every role's `endpoint` (e.g. `http://evil.example/../<script>` or a `user@host` authority), so a client scraping the contract could be handed an attacker's origin to dial. An invalid host is now treated as "no origin supplied" (empty endpoint), exactly as an absent header is. The trusted operator override `GATEWAY_PUBLIC_URL` is unaffected. (SonarCloud `pythonsecurity:S5131`.)
+
+### Added
+
+- `scripts/live-check.sh` + `tests/test_live_capabilities.py` — a LOCAL, single-trigger, unattended pre-PR gate that dials every advertised role endpoint+path, every id in `/v1/models`, checks CLI/gateway agreement, reproduces Colleague's role-discovery path, and fails on deployed-gateway version skew. It FAILS rather than skips when armed. A 429 (pressure shed) and a 503 carrying `Retry-After` count as reachable; only a 404, a connection failure, a `Retry-After`-less 503, or a bare 5xx are faults. that dials every advertised role endpoint+path, every id in `/v1/models`, checks CLI/gateway agreement, reproduces Colleague's role-discovery path, and fails on deployed-gateway version skew. It FAILS rather than skips when armed. A 429 (pressure shed) and a 503 carrying `Retry-After` count as reachable; only a 404, a connection failure, a `Retry-After`-less 503, or a bare 5xx are faults.
+- `lobes/gateway/_readiness.py` — a bounded background probe of each backend's `/health`, mirroring `PressureCache`. Tri-state (`True` healthy / `False` reached-but-unhealthy / `None` unreachable), daemon thread, socket-free `.current()`, and a probe that degrades to `None` on `OSError`, `http.client.HTTPException` and `ValueError`.
+- `GET /health` now reports `{"version": ...}`, and `lobes doctor` gains a `gateway_version_match` check that fails on skew between the deployed gateway and the CLI wheel (issue #99).
+- `lobes/gateway/_routing.py::is_unknown_model` — a pure predicate separating "unknown model id" from "unspecified model".
+- Ground-truth perception probes for the `senses` role: a stdlib-generated solid-colour PNG whose colour the model must name, and a Chatterbox-synthesized word the model must transcribe. Both carry negative controls; the old placeholder-media tests are relabelled as wire checks.
+
+### Changed
+
+- **No cross-backend failover.** `order_backends` now returns at most one backend. A request naming the cortex model can never be answered by the Gemma backend, which protects the `final_authority` role contract from #81.
+- `GET /v1/models` and `GET /capabilities.ready` are backed by the live readiness cache rather than by configuration. A wired-but-dead backend is no longer advertised.
+- `RoleInfo.ready` is no longer an alias of `loaded` for `cortex`/`senses`/`embedder`/`reranker`. `build_role_registry` self-enforces the invariant: a supplied `backend_ready` map is authoritative, and a present `None`, a present `False`, and a missing key all mean not-ready.
+- `lobes capabilities` / `lobes endpoint` now render the running gateway's `GET /capabilities` when it is reachable, falling back to an offline `.env`-derived view with `ready=false` on every role. The CLI and the gateway can no longer disagree, because there is now one derivation instead of two. The `--json` payload is keyed strictly by role name in both modes (byte-identical to the gateway's payload in live mode); the live-vs-offline signal travels on stderr (a `source:` notice) rather than as an extra top-level key, so a strict `set(keys) == ROLES` consumer never trips (Qodo).
+- A backend is wired only when its `*_BASE_URL` is set; the `or *_SERVED_NAME` clause is gone (issue #97).
+- `senses` is documented as vision-only intake. The checkpoint declares audio support but vLLM's `gemma4_unified` path does not serve it (issue #101); `stt` remains the supported speech path.
+- `docs/gemma4-mtp-draft.md` carries a superseded banner: DSpark does not load on vLLM 0.23 (#75). Issue #69's disabled-experiment-entry criterion is closed answered-negative.
+- README's quickstart no longer claims `lobes init` scaffolds the single-model deployment; the fleet duo has been the default since #69.
+
+### Fixed
+
+- **#91** — a dead cortex backend no longer surfaces as a terminal `404 model does not exist`. `handle_post` rewrote the model id once, before the failover loop, then retried the same body against a backend serving a different model, which correctly 404'd; the `4xx = client error, no failover` rule relayed that as terminal and killed multi-step agent loops. A dead, unreachable or warming owner now yields **503 + `Retry-After`** with `type: backend_unavailable`.
+- **#92 / #95** — the gateway never advertises an origin built from its internal listen port. Precedence is `GATEWAY_PUBLIC_URL` (an operator override for a tunnel or Host-rewriting proxy) > the request `Host` header > an empty endpoint. `GATEWAY_PUBLIC_URL` is deliberately NOT defaulted: a defaulted `public_url` outranks `Host` and would advertise loopback to every remote client.
+- **#96** — `AUDIO_URL` now reaches the gateway from the base fleet compose, so `stt`/`tts` stop advertising `ready=true` on a path that 404s when the audio overlay is not composed in.
+- **#97** — `GET /v1/models` no longer advertises phantom backends wired from a `*_SERVED_NAME` alone against a `default_url` naming a container that need not exist.
+- `ReadinessCache` seeds its initial per-backend snapshot with `dict.fromkeys` instead of a dict comprehension (SonarCloud `python:S7519`).
+- `ReadinessCache.stop()` no longer clears its thread reference while the refresh thread is still alive, so a subsequent `start()` cannot spawn a second concurrent refresh loop; the join bound now covers a full sequential refresh pass and `start()` cleanly replaces a genuinely-dead thread (Qodo reliability).
+- An unknown model id returns `404 model_not_found` instead of being silently served by the default backend under a different model's weights. Unknown-ness is decided against the routing table, never against the readiness-filtered `/v1/models` list, so a wired-but-dead backend still yields 503 rather than 404.
+- `test_live_main_text_returns_nonempty_content` no longer fails on a thinking model: `max_tokens=16` was consumed entirely by the reasoning trace, leaving `content=None` and `finish_reason=length` (issue #93).
+
 ## [0.39.0] - 2026-07-07
 
 ### Added

.devague/plans/lobes-never-advertises-a-capability-it-cannot-serv.json

@@ -58,8 +58,11 @@ output quality is expected and opt-in, not guaranteed by the diagnostic. See
 Beyond `cortex`, the **fleet** exposes SIX first-class, Colleague-facing
 **roles** (issue #81) — the primary contract callers should address, not raw
 model ids: `cortex` (the 27B primary — reasoning/deciding/final authority),
-`senses` (the Gemma 4 12B multimodal gear — vision+audio intake/perception;
-never decides or takes repo actions), `embedder` (`Qwen/Qwen3-Embedding-0.6B` →
+`senses` (the Gemma 4 12B multimodal gear — vision intake/perception; never
+decides or takes repo actions; the checkpoint declares audio support but it is
+**not currently served** on this vLLM path — issue #101 — so `senses` is
+vision-only in practice, and the purpose-built `stt` role, below, is the
+supported path for speech), `embedder` (`Qwen/Qwen3-Embedding-0.6B` →
 `POST /v1/embeddings`), `reranker` (`Qwen/Qwen3-Reranker-0.6B` → `POST
 /v1/rerank` + `/v1/score`), and the opt-in audio overlay's `stt`/`tts`. Roles
 are routed by **task family** (`generate` / `embed` / `score` / `rerank`) and

.eidetic/memory/lobes__public.jsonl

@@ -50,17 +50,24 @@ CLIs in loops, so safe-by-default is mandatory.
 ## Running the model locally (vLLM)
 
 `lobes init` scaffolds a deployment directory (default `$HOME/.lobes`) from the
-packaged templates: a `docker-compose.yml` that stands up the vLLM model as an
-OpenAI-compatible server on `:8000`, plus a `.env`. Tuned for DGX Spark (GB10
-Grace Blackwell, 128 GB unified memory) per
+packaged templates. **Since issue #69, bare `lobes init` (no flags) scaffolds
+the fleet duo by default** — see ["Running the model behind a gateway
+(fleet)"](#running-the-model-behind-a-gateway-fleet) below; that is almost
+certainly what you want. This section instead walks through the **legacy
+single-model** scaffold: one vLLM server, no gateway, opted into with
+`--single` (alias `--legacy`) — a `docker-compose.yml` that stands up the
+vLLM model directly as an OpenAI-compatible server on `:8000` (the
+*container's own* port; there is no gateway in front of it in this mode —
+see the port note in the fleet section below), plus a `.env`. Tuned for DGX
+Spark (GB10 Grace Blackwell, 128 GB unified memory) per
 [build.nvidia.com/spark/vllm](https://build.nvidia.com/spark/vllm).
 
 Prerequisites: the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html),
 and `docker login nvcr.io` with an [NGC API key](https://org.ngc.nvidia.com/setup/api-key)
 to pull the `nvcr.io/nvidia/vllm` image.
 
 ```bash
-lobes init --apply          # writes $HOME/.lobes/{docker-compose.yml,.env}
+lobes init --single --apply # writes $HOME/.lobes/{docker-compose.yml,.env} (legacy single-model)
 # edit $HOME/.lobes/.env to set HF_TOKEN if the model repo is gated
 lobes serve --apply         # first run downloads ~28 GB of weights (the 27B primary)
 lobes status                # waits/reports until /health is up
@@ -178,19 +185,37 @@ tunnel` for the full flow.
 
 ## Running the model behind a gateway (fleet)
 
-`lobes init --fleet` scaffolds a **multi-container** deployment instead of one: the
-always-warm Qwen generate primary, two tiny co-resident **embedding** and
-**reranker** gears, and a single stdlib **gateway** that fronts them on the host
-port the acp `vllm-local` provider already expects. The gateway routes each
-request by its `model` field — to the primary, the embedder, or the reranker by
-task family (generate / embed / score / rerank) — and defaults an unknown/missing
-name to the primary, so existing single-model clients keep working unchanged. The
-same front fans `/v1/audio/*` out to the `--audio` overlay, and a warm *generate*
-fallback can be wired in later (the gateway adds it, with failover, only when one
-is configured).
+**This is the default scaffold since issue #69** — plain `lobes init --apply`
+(no flags) already gives you this; `--fleet` is a back-compat no-op kept only
+so old scripts that passed it keep working, and `--single` (alias `--legacy`)
+is what opts you *out*, back to the one-container deployment the previous
+section describes. `lobes init` scaffolds a **multi-container** deployment:
+the always-warm Qwen generate primary, the Gemma 4 12B multimodal gear, two
+tiny co-resident **embedding** and **reranker** gears, and a single stdlib
+**gateway** that fronts them all on the host port the acp `vllm-local`
+provider already expects — `VLLM_PORT` (packaged default `:8000`; `:8001` on
+the reference DGX Spark deployment, deliberately set apart from the
+single-model story's default). The gateway routes each request by its
+`model` field — to the primary, the multimodal gear, the embedder, or the
+reranker by task family (generate / embed / score / rerank) — and defaults
+an unknown/missing name to the primary, so existing single-model clients
+keep working unchanged. The same front fans `/v1/audio/*` out to the
+`--audio` overlay, and a warm *generate* fallback can be wired in later (the
+gateway adds it, with failover, only when one is configured).
+
+> **`:8000` means two different things depending on scaffold — this
+> ambiguity is what issue #92 was about.** In the legacy single-model
+> section above, `:8000` is the vLLM **container's own** port, published
+> straight to the host with no gateway in front of it. Here, `:8000` (or
+> whatever `VLLM_PORT` is set to) is the **gateway's** published port,
+> fronting several backends by `model` field. They happen to share the same
+> default number, but a client must dial whichever origin its deployment
+> actually publishes — never assume a bare port number implies a particular
+> topology. `lobes doctor` and `GET /capabilities` are the source of truth
+> for what a given deployment actually serves and where.
 
 ```bash
-lobes init --fleet --apply        # $HOME/.lobes/{docker-compose.yml,.env,Dockerfile.gateway}
+lobes init --apply                # $HOME/.lobes/{docker-compose.yml,.env,Dockerfile.gateway} — the default
 docker login nvcr.io              # NGC API key for the vLLM image
 lobes fleet up --apply            # builds the gateway image + starts the backend
 lobes fleet status                # container states + gateway /health + /v1/models
@@ -204,7 +229,7 @@ curl -s http://localhost:8000/v1/chat/completions -d '{"model":"sakamakismile/Qw
 
 The fleet runs a **default-on `cortex` + `senses` duo** (the `main` + `multimodal`
 backends) — the 27B Qwen text generate primary served at **128K** (`cortex`, util
-`0.30`) and the Gemma 4 12B vision+audio gear served at **32K** (`senses`, util
+`0.30`) and the Gemma 4 12B vision gear served at **32K** (`senses`, util
 `0.14` — provisional pending live validation; `coolthor/gemma-4-12B-it-NVFP4A16`,
 native MTP default-on — the coder fine-tune, `sakamakismile/gemma-4-12B-coder-…`,
 is kept as an opt-in `multimodal-coder` gear; see

CHANGELOG.md

@@ -27,12 +27,23 @@ below.
 | Role | Backend / service | Endpoint path | What it's for |
 |---|---|---|---|
 | `cortex` | `primary` (generate) | `POST /v1/chat/completions` | Reasoning, deciding, planning, tool use, repo actions — the final authority. |
-| `senses` | `multimodal` (generate) | `POST /v1/chat/completions` | Intake/perception (text+image+audio) and speaking back to the user. Does **not** decide or act. |
+| `senses` | `multimodal` (generate) | `POST /v1/chat/completions` | Intake/perception (text+image) and speaking back to the user. Does **not** decide or act. |
 | `embedder` | `embed` (pooling) | `POST /v1/embeddings` | Dense text embeddings for memory/retrieval. |
 | `reranker` | `rerank` (pooling) | `POST /v1/rerank` (+ `/v1/score`) | Reordering/scoring retrieved candidates. |
 | `stt` | Parakeet (audio overlay, opt-in) | `POST /v1/audio/transcriptions` | Speech-to-text. |
 | `tts` | Chatterbox (audio overlay, opt-in) | `POST /v1/audio/speech` | Text-to-speech. |
 
+> **`senses` is vision-only intake — audio is not currently served (issue
+> #101).** The `coolthor/gemma-4-12B-it-NVFP4A16` checkpoint behind `senses`
+> declares an `audio_config` in its own model config, but on this vLLM serving
+> path (`gemma4_unified`) an `input_audio` content part is silently **dropped**
+> rather than rejected: a caller gets `200 OK` and a fluent answer that ignored
+> the audio. Live evidence and the tracking issue are in
+> [`docs/gemma-4-12b-nvfp4.md`](gemma-4-12b-nvfp4.md#live-validation-status-71).
+> For speech, use the purpose-built **`stt`** role (Parakeet, `POST
+> /v1/audio/transcriptions`) instead — it remains first-class and is
+> unaffected by this gap.
+
 `cortex`, `senses`, `embedder`, and `reranker` are always enumerated (present
 with `loaded=false` if their gear isn't wired in this deployment); `stt`/`tts`
 require `lobes init --fleet --audio`. **`brain` is not a valid role name** —
@@ -313,7 +324,7 @@ scaffold did solo, in exchange for co-residency:
 | **Fleet duo, pre-rebalance** | 64K, util 0.30 | 128K (`MULTIMODAL_MAX_MODEL_LEN=131072`, util 0.22) |
 | **Fleet duo, current (this doc)** | **128K** (`PRIMARY_MAX_MODEL_LEN=131072`, util 0.30 — util-bound, not context-bound) | **32K** (`MULTIMODAL_MAX_MODEL_LEN=32768`, util 0.14) |
 
-The pre-rebalance duo gave the vision/audio gear its full native 128K at the
+The pre-rebalance duo gave the vision gear its full native 128K at the
 cost of trimming `cortex` to 64K; the current default flips that trade-off —
 `cortex` (the final-authority reasoning role) now gets its full native 128K,
 and `senses` (intake/perception) is trimmed to 32K, which is ample for the

CLAUDE.md

@@ -347,6 +347,17 @@ is required (pinning keeps the image reproducible); from-source/dev boxes that r
 ahead of a PyPI release point `MODEL_GEAR_VERSION` at a published TestPyPI `.devN`
 build.
 
+**That pin is written once and nothing re-pins it (issue #99).** `lobes init`
+sets `MODEL_GEAR_VERSION` at scaffold time; no other verb touches it
+afterwards, so a long-lived deployment can silently keep running an old
+gateway image long after the host's own `lobes` binary — and PyPI — have
+moved on. `GET /health` reports the deployed gateway's own `lobes-cli`
+`version` (additive field, issue #99) precisely so this is detectable without
+docker: `lobes doctor`'s `gateway_version_match` check compares it against the
+CLI's own version and fails the run on a real mismatch (remediation: bump
+`MODEL_GEAR_VERSION` in `.env` and `docker compose up -d --build gateway`) —
+see [Verbs](#verbs) below.
+
 ### Auth (known limitation)
 
 The gateway is a **pass-through** and is **not auth-aware** — it does not inspect