lobes never advertises a capability it cannot serve (#92 · #91 · #95 · #97 · #96 · #74 · #69) - #102
Merged
Merged
Conversation
…#74 · #69) Unifies five open items into one invariant — advertised implies reachable — after live investigation on the DGX Spark rig proved the issues describe the symptoms correctly but the causes wrongly: * #92 is not a regression of #87. `reachable_origin` shipped in 0.38.0 (PR #90); the rig runs a gateway image built 2026-07-03 carrying lobes 0.36.0, so the fix was never deployed and nothing detects the skew. Host :8000 is not dead either — it is an unrelated uvicorn service (reachy-mini-dae). * #91 is not a backend-reload window. `handle_post` rewrites the model id once, before the failover loop, while `order_backends` offers every same-task generate backend as a failover target. Cortex 5xx (EngineDeadError, 04:08:34) -> the same body is retried against Gemma -> Gemma correctly 404s -> the "4xx = client error, no failover" rule relays it as terminal. The primary container's own logs show the symmetric case at 04:34:01 and 04:39:01. * Two unfiled bugs of the same shape: AUDIO_URL never reaches the gateway container (stt/tts advertise ready=true on a path that 404s), and /v1/models advertises phantom backends (6 models, 4 containers). Decisions recorded in the frame: no cross-backend failover at all; readiness becomes a background cached probe; phantom backends stopped by both a config gate and the readiness filter; #69's DSpark criterion closed answered-negative. Evidence: coolthor/gemma-4-12B-it-NVFP4A16 genuinely perceives images (red -> "Red", blue -> "Blue", ground-truth checked). Audio perception is blocked by the AUDIO_URL bug and is gated behind its fix. Frame: .devague/frames/lobes-never-advertises-a-capability-it-cannot-serv.json Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…#92 · #91 · #74 · #69) Forward leg from the converged frame. Tasks are decomposed by FILE so each wave is operationally parallel, not merely formally so: wave 0 t1 _config.py (backend wiring gate) t2 _routing.py (no cross-backend failover) t3 _readiness.py (new, background probe cache) t4 fleet templates (GATEWAY_PUBLIC_URL + AUDIO_URL to the gateway) wave 1 t5 roles.py (ready decoupled from loaded; never advertise the internal port) wave 2 t6 server.py (503 + Retry-After, readiness wiring, reachable_origin) wave 3 t7 CLI (capabilities/gateway agreement, doctor version-skew) t8 senses perception probe (ground-truth image + audio) wave 4 t9 the local, single-trigger, unattended pre-PR live gate wave 5 t10 documentary repairs (gemma docs, DSpark banner, README quickstart) Risks recorded: *_BASE_URL gate may break a hand-edited .env (r1); background probe-thread lifecycle inside ThreadingHTTPServer (r2); Chatterbox's poisoned-CUDA history can flake the audio probe (r3); the live gate cannot run in CI, so nothing structurally forces it to run — exactly how #87's fix shipped in 0.38.0 while the rig kept running 0.36.0 (r4, follow-up). New issues filed from this investigation: #96 (AUDIO_URL never reaches the gateway), #97 (phantom backends in /v1/models), #98 (cortex EngineDeadError). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…e gateway (#96) Two env vars never reached the gateway container from the base fleet compose, both causing it to advertise things it cannot serve: - GATEWAY_PUBLIC_URL defaulted to empty, so the advertised /capabilities origin fell back to Host-header inference — and, absent a Host header, fabricated a URL from the gateway's internal listen port rather than the published host port. It now defaults to http://localhost:${VLLM_PORT:-8000}, the same port the gateway is actually published on, while staying operator-overridable for a tunnel / Host-rewriting reverse proxy. - AUDIO_URL only reached the gateway via the --audio overlay (docker-compose.audio.yml). A base-only deployment left ServerConfig.audio_url unset in code but 'lobes capabilities' (reading the merged .env) still reported stt/tts as ready=true, and POST /v1/audio/speech 404'd. AUDIO_URL is now declared on the base template too, defaulted to empty so a base-only deployment resolves it to unset (loaded=false) instead of pointing at a realtime container that was never started; the audio overlay's override still supplies the real value when present. Verified the nested ${GATEWAY_PUBLIC_URL:-http://localhost:${VLLM_PORT:-8000}} interpolation with `docker compose config` against a scratch copy of the template, both with and without the audio overlay layered on top. tests/test_fleet_template_gateway_env.py (new) parses the packaged fleet compose and locks in both keys/defaults. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
_optional_backend previously wired a backend when EITHER its *_BASE_URL
OR its *_SERVED_NAME env var was set, falling back to a hardcoded
default_url naming a compose service that need not exist. On the
reference rig this invented two phantom backends -- multimodal-coder and
middle -- that GET /v1/models advertised while no container served them.
Drop the `or name_key` clause: a served name with no URL describes a
model, not a reachable backend. This matches the contract the fleet
already documents for MINOR_BASE_URL ("empty => minor silently
unwired"). Verified on the live rig this unwires exactly
multimodal-coder and middle, keeping primary/multimodal/embed/rerank.
Repairs the tests that asserted the old permissive behaviour
(test_gateway_routing, test_fleet_minor, test_gateway_tiers, and the
test_gateway_server _cfg fixture, which relied on FALLBACK_SERVED_NAME
alone wiring the fallback backend for its failover tests) and adds
tests/test_gateway_config_wiring.py covering the new contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
New module lobes/gateway/_readiness.py: a bounded, background-cached probe of each fleet backend's /health so the gateway can distinguish "answered recently" from "merely configured" (today RoleInfo.ready aliases loaded). Mirrors PressureCache's shape/naming/threading discipline: a single daemon thread refreshes an in-memory snapshot on an interval; .current() returns a copy without ever probing (O(1), socket-free on the request path). Readiness is tri-state, matching probe_audio_ready: True (reached, 200) / False (reached, non-200 e.g. warming) / None (unreachable/unknown) — False and None are not collapsed. The probe helper catches OSError, HTTPException AND ValueError (the non-numeric-port bug caught on PR #90) and degrades to None. The background thread is a daemon and stop()/close() joins it cleanly. Standalone + fully unit-tested (tests/test_gateway_readiness.py); a later task wires .current() into GET /v1/models and GET /capabilities. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
for remote clients c11 required the fleet compose to inject GATEWAY_PUBLIC_URL derived from the published VLLM_PORT, and for reachable_origin to prefer it over the request Host header. That is wrong: a defaulted public_url is ALWAYS set, so the Host header is never consulted, and a LAN or tunnel client GETting /capabilities is told to dial http://localhost:8001 — which on that client's machine is a different service. The fix for "the advertised endpoint points at a foreign daemon" would have reintroduced exactly that defect one rung out. c29 replaces it: explicit operator override (GATEWAY_PUBLIC_URL, for a tunnel or Host-rewriting proxy) > the origin the client actually dialed (Host header) > nothing (an empty endpoint). Never an absolute URL built from the internal GATEWAY_PORT, and never a defaulted localhost public_url. Each caller receives an origin correct for itself. The AUDIO_URL half of the compose change (issue #96) is unaffected and stands. Plan coverage repointed: c29/h25 -> t4 (compose) + t6 (origin resolver). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
Amends the previous commit's GATEWAY_PUBLIC_URL half. The c11 requirement it implemented (default the origin from the published VLLM_PORT) was defective: a defaulted public_url reintroduces the #92 defect. reachable_origin prefers a set public_url OVER the request Host header, so any localhost/published-port default advertises loopback to every remote client — a LAN/tunnel caller dialing spark.local:8001 is told to use http://localhost:8001, a foreign service on their machine. Amended requirement (c29): origin precedence is explicit operator override > Host header > empty. GATEWAY_PUBLIC_URL exists ONLY as an operator override for a tunnel / Host-rewriting reverse proxy; it must not be defaulted to a localhost URL. So: - docker-compose.yml: GATEWAY_PUBLIC_URL back to ${GATEWAY_PUBLIC_URL:-}, with a load-bearing comment explaining WHY it must stay empty (a defaulted public_url outranks the Host header) so it is not "helpfully" restored later. - env.example: document GATEWAY_PUBLIC_URL as the tunnel / proxy override only; empty makes the gateway echo the origin the client dialed. No published-port default claim. - test_fleet_template_gateway_env.py: invert the ${VLLM_PORT}-default assertions. Now asserts the default is exactly ${GATEWAY_PUBLIC_URL:-} AND the negative regression guard — the default contains neither 'localhost' nor 'VLLM_PORT'. Kept the ports: mapping test (documents the published-vs-internal port distinction that is the root cause). The AUDIO_URL=${AUDIO_URL:-} half (the #96 fix) is unchanged. Re-verified with `docker compose config`: GATEWAY_PUBLIC_URL stays empty even with VLLM_PORT=8001 set, operator overrides win, and the --audio overlay still supplies AUDIO_URL while leaving GATEWAY_PUBLIC_URL empty. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
order_backends() previously walked every same-task generate backend as a failover chain (e.g. cortex -> multimodal). A dead cortex vLLM engine got silently retried against the Gemma backend with a body still naming the Qwen model id, producing a terminal 404 that killed long-running agent loops -- or worse, a real answer from the wrong model, violating the final_authority role contract (#81). order_backends(table, served) now always returns a list of length <= 1: the served name's owning backend, or the default_model's owner for an unknown name. No runtime cross-backend retry, ever. The static tier-alias upward fallback in tier_aliases() is unrelated and preserved -- that resolves an unwired capability tier at table-build time, before order_backends runs. Inverted tests to the new contract (renamed, not deleted, to keep the bug's history legible): - test_gateway_routing.py: test_order_backends_generate_still_failovers_ between_generate_backends -> test_order_backends_generate_never_failovers_ across_models; test_order_backends_owner_first_then_failover -> test_order_backends_owner_only_no_failover; added test_order_backends_always_returns_at_most_one_backend, test_order_backends_known_served_returns_its_own_owner, test_order_backends_unknown_served_returns_default_owner. - test_fleet_minor.py: test_order_backends_minor_is_owner_with_primary_ failover -> test_order_backends_minor_is_owner_with_no_failover; the "primary failover includes minor" test -> test_primary_never_failovers_ to_minor_when_minor_present. - test_gateway_server.py: test_failover_on_connection_refused -> test_no_failover_on_connection_refused; test_failover_on_5xx -> test_no_failover_on_5xx; test_explicit_fallback_routes_to_fallback_first -> test_explicit_fallback_routes_to_fallback_only; updated test_all_backends_down_returns_502 for the single-attempt attempts list. All still assert the existing 502 upstream_unavailable (handle_post's 502->503+Retry-After conversion is t6's scope, not touched here). lobes/gateway/server.py, _config.py, _readiness.py, and roles.py are untouched -- handle_post's existing for-loop over order_backends() already degrades correctly to a single attempt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
… gateway endpoint from internal host:port (t5) Two bugs let GET /capabilities advertise ready: true for a role whose endpoint 404s (advertised-implies-reachable violation): 1. RoleInfo.ready was a bare alias of loaded (a config fact) for cortex/senses/embedder/reranker, unlike stt/tts which already split the two (issue #89/#90). build_role_registry now accepts an optional backend_ready: Mapping[str, bool | None] keyed by ROLE_BACKEND name (the exact shape lobes.gateway._readiness.ReadinessCache.current() returns), mirroring audio_ready's shape/defaulting. ready is structurally clamped to False whenever a role's backend isn't wired or its endpoint is empty, so no caller-supplied signal can fabricate ready=True — generalising the #89/#90 clamp to all six roles. 2. _gateway_base_url(server) fabricated an absolute URL from the gateway's INTERNAL container listen port (GATEWAY_HOST/GATEWAY_PORT), which need not match the published host port a caller can actually dial. It now only ever returns server.public_url (an operator- declared GATEWAY_PUBLIC_URL) or "" — never a value derived from host:port. An empty endpoint is covered by the same ready clamp. Repairs the resulting blast radius: tests/test_roles.py (new backend_ready + empty-endpoint coverage, replacing the now-obsolete GATEWAY_HOST bracket/normalize tests), tests/test_gateway_capabilities.py (explicit gateway_url added where a test needs a dialable endpoint), and tests/test_colleague_contract.py (the fake fleet now passes its own real loopback origin as gateway_url, matching what the production HTTP route does via reachable_origin()). lobes/gateway/server.py is untouched — wiring ReadinessCache into the live gateway route is a follow-up task. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…igin built from the internal port (c29)
…ess-gated /v1/models & /capabilities (#91 #92 #14) Integration task the "advertised implies reachable" plan converges on. Job 1 — status matrix (handle_post, #14/#91). With order_backends now single-element (no cross-backend failover), a present-but-dead owner (refusal / timeout / >=500) becomes a RETRYABLE 503 backend_unavailable + Retry-After, never a terminal 404/502. The 429 server_busy shed is untouched; a 4xx (incl. the owner's own 404 "model does not exist") is a client error relayed verbatim; 502 upstream_unavailable survives only for the degenerate empty order_backends (malformed routing table). Rewrote handle_post's + the module docstring, which documented removed failover. Job 2 — readiness wiring (#92, c15/h14). serve() constructs a ReadinessCache, does one bounded synchronous refresh() BEFORE binding (closes the startup window — the cache seeds to None), then start()s the daemon; _make_handler binds it. GET /v1/models lists only backends whose readiness is True (list_models_payload gains an additive ready filter in _routing.py). GET /capabilities folds the snapshot into role readiness. Bridged a semantic mismatch at the boundary (_ready_iff_true): the cache's None means "unreachable" but roles.py reads None as "no signal → fall back to loaded", so a raw pass-through would advertise a wired-but-dead backend ready=True — collapse None/False→False so a dead backend is advertised nowhere. The POST hot path opens no probe (reads socket-free .current()). Job 3 — origin resolution (c29/h25). Locked reachable_origin precedence (GATEWAY_PUBLIC_URL > Host > empty) with tests, incl. the regression guard that an unset public_url + Host: spark.local:8001 yields the client's own origin, never localhost/GATEWAY_PORT (host :8000 is a foreign daemon on the rig), and that None → "" endpoint end to end. Invariants: race (listed → owner killed → 503, never 404), no-Gemma (dead primary never dials the multimodal backend, asserted on opener call sites), converse (unknown model id routes to default owner — current behaviour locked as a documented choice; recommend a future 404 in _routing.py). Added public ReadinessCache.refresh() (permitted additive change) for the seed-before-bind. Tests: 1151 passed, 6 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…on tests (#74, #96) Layer B's image/audio "confirmed" claims only proved the wire (1x1 PNG / near-silent WAV, asserting non-empty content) -- a model ignoring the media entirely would still pass. Add real perception tests instead: * test_live_multimodal_image_perception_names_colour[red|blue]: a solid-colour PNG generated in-process (stdlib zlib/struct only) must be named correctly. Verified live against http://localhost:8001 (red -> 'Red', blue -> 'Blue'), and verified falsifiable via a negative control (asserting 'red' against a blue image fails). * test_live_multimodal_audio_perception_transcribes_known_word: synthesizes a known word via the rig's own TTS (POST /v1/audio/speech) and asserts the transcription contains it. Fails loudly (not skip) on the current 404 -- AUDIO_URL not yet wired into this running gateway, issue #96, fixed in the base fleet template by t4 but pending redeploy -- and gives TTS a bounded retry so a poisoned-CUDA-context 500 (Chatterbox's known failure mode) reports as "TTS backend unhealthy", distinct from "senses cannot hear". The old placeholder-media tests are kept but renamed to test_live_multimodal_accepts_{image,audio}_content_part_wire_check with docstrings stating they assert transport only, not perception. Offline suite unchanged: 1151 passed (baseline), 9 skipped (+3 new live-gated tests, still gated on LOBES_SMOKE_BASE_URL). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…checks relabelled (#74)
Closes the two gaps t6 left as caller-discipline patches. Gap 1 (#92/h14) — build_role_registry now self-enforces the readiness invariant its own docstring promises. A SUPPLIED backend_ready mapping is authoritative: a present None, a present False, and a missing key all mean NOT ready (ready = get(name) is True), still clamped on loaded + non-empty endpoint. Only an OMITTED mapping falls back to the loaded proxy (back-compat). This removes the trap where the readiness cache's None (UNREACHABLE) was read as roles.py's None (no signal -> fall back to loaded=True), resurrecting the #92 defect. server.py's _ready_iff_true bridge is now redundant and deleted; the /capabilities route passes ReadinessCache.current() straight through, and the coercion is the builder's job for every caller, not one call site. Gap 2 (h23 converse) — an unknown model id is no longer silently served under the default backend's weights. New pure predicate is_unknown_model distinguishes UNKNOWN (a non-empty id that is neither an alias nor any wired backend's served name) from UNSPECIFIED (missing/blank -> default_model, still served). handle_post 404s an unknown id (model_not_found) before routing. Unknown-ness is decided against the ROUTING TABLE, never the readiness-filtered /v1/models list: a wired-but-dead backend is dropped from /v1/models but is still KNOWN, so it routes to its owner and yields the retryable 503, not a 404 (keeps #91 fixed). resolve_model's signature is unchanged (companion-predicate design), so its many callers and the tier tests are unaffected. Inverted tests: test_backend_ready_missing_entry_falls_back_to_loaded -> _is_not_ready; added present-None coverage; replaced test_unknown_model_id_routes_to_default_owner_documented_choice with the h23 converse 404 test plus wired-but-dead 503-not-404 proofs at both the handle_post and route level. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
… ids 404 instead of being silently served (h14, h23)
…ew (#96, #99) lobes capabilities / lobes endpoint no longer re-derive the six-role contract from .env — they GET the gateway's own /capabilities and render it verbatim, falling back to the offline .env-derived registry (every role's ready forced false, source="offline"/"gateway" marks which) only when no gateway answers. This makes the CLI/gateway honesty condition (h3) true by construction instead of by keeping two independent derivations in sync, which had already drifted in both directions (#92 underreported, #96 overreported stt/tts as ready=true on a 404 path). GET /health now reports the gateway's own lobes-cli version (additive), and `lobes doctor` gained a gateway_version_match check comparing it to the CLI's own version: mismatch is a real error (fails the run, names the exact MODEL_GEAR_VERSION fix), an unreachable/pre-#99 gateway degrades to a non-fatal info result rather than a false pass. This targets issue #99, the structural cause of #92 (a gateway image pinned once at `lobes init` time and never re-pinned). tests/conftest.py's autouse fixture now also neutralises the new gateway probes (matching how it already neutralises /health), since the dev rig has a real unrelated daemon on host port 8000 that could otherwise leak into the "offline" test path. Full suite: 1169 passed, 9 skipped (was 1158 passed, 9 skipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
test_live_main_text_returns_nonempty_content assumed a non-thinking model: with max_tokens=16 the cortex model (a Qwen3.6 reasoning model with preserve_thinking, issue #93) can spend the whole budget on its reasoning trace and return content=None with finish_reason="length". Raise the budget to 256 and accept the reasoning trace as evidence of life only when finish_reason explains it as budget exhaustion, so the fallback can't mask a genuine empty-content bug. test_live_multimodal_audio_perception_transcribes_known_word fails for a real, now-tracked reason: the deployed Gemma silently discards input_audio content parts (vLLM gemma4_unified gap, not a checkpoint gap -- issue #101). Mark it xfail(strict=True) citing the mechanism and evidence; the probe body is untouched so it flips to XPASS (and fails the suite) the day audio ingestion starts working. Verified against the live rig (LOBES_SMOKE_BASE_URL=http://localhost:8001): main-text passes, both colour perception tests pass, audio perception xfails, both wire checks pass. Offline: 1169 passed, 9 skipped (unchanged from baseline).
… retire DSpark route Job 1 — docs/gemma-4-12b-nvfp4.md's #71/#73 admission ("not independently re-run against the base checkpoint") is now resolvable: live evidence against coolthor/gemma-4-12B-it-NVFP4A16 via model=multimodal shows image+text VERIFIED (ground truth + negative control) and audio+text NOT SUPPORTED — vLLM's gemma4_unified silently drops the input_audio content part instead of rejecting it (200 OK, audio ignored). Tracked as issue #101. Also corrects the provenance of the coder checkpoint's original "audio+text ✓ (transcribed verbatim)" claim: that check only asserted HTTP 200 + non-empty content against a placeholder clip, never ground truth — and the claim didn't hold when tested properly. lobes/catalog.py's coolthor and coder entry comments carry the same correction. Job 2 (user decision) — a capability lobes cannot serve must not appear in a contract Colleague reads. docs/colleague-stack.md and CLAUDE.md described `senses` as "vision+audio intake/perception"; corrected to vision-only, with a clearly-marked note pointing at issue #101 and the purpose-built `stt` role as the supported path for speech. README.md carried the same overclaim in two spots ("vision+audio gear"); fixed for consistency. lobes/roles.py's ROLE_RESPONSIBILITIES tokens are untouched — they never claimed audio. Job 3 (user decision, closed answered-negative) — docs/gemma4-mtp-draft.md still presented the DSpark route as "the ONE route task t3 should wire next," inconsistent with issue #75's live finding (Gemma4DSparkModel does not load on vLLM 0.23) already recorded in docs/gemma-4-12b-nvfp4.md and docs/vllm-nightly-migration.md. Added a superseded banner citing #75 and pointing at the two docs carrying the current story; the research content stays below it as the record of how the question was answered. Also states that issue #69's last acceptance criterion (a disabled-by-default DSpark entry) is answered-negative: no catalog entry ships for a model that cannot load. Job 4 — README.md's quickstart still documented bare `lobes init --apply` as scaffolding the single-model deployment and framed `--fleet` as the way to opt into the multi-container deployment. Both have been false since issue #69: the fleet duo is the default, `--fleet` is a back-compat no-op, and `--single`/`--legacy` opts out. Reworked both sections accordingly and added a note on the :8000-means-two-different-things ambiguity (container port vs. gateway-published port) that issue #92 was about. uv run pytest -n auto -q: 1169 passed, 9 skipped (matches baseline). markdownlint-cli2 passes on every touched markdown file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
…DME quickstart matches the fleet default (#69)
…runner)
Adds the executable local pre-PR gate the "advertised implies reachable"
plan exists to create. It is NOT a CI job (CI has no GPU/fleet): a developer
runs it against a running deployment before opening a PR.
New files:
* tests/test_live_capabilities.py — five checks, each mapped to its issue:
1. every ready role in GET /capabilities is reachable at endpoint+path
(404 or a bare 5xx-without-Retry-After = fail; #92/#96/#89).
2. every GET /v1/models id is reachable on its own task lane, never a
404 "model does not exist" (#91) — lane resolved from the live
contract so pooling models aren't falsely dialed on a chat route.
3. `lobes capabilities --json` agrees with GET /capabilities on
endpoint/ready/loaded for all six roles, via source==gateway (#95/#92).
4. gateway GET /health version == lobes.__version__; a missing version
field is skew, not a pass (#99).
5. a Colleague can resolve+dial cortex/senses from the contract alone,
no hardcoded model ids (#81/#87).
* scripts/live-check.sh — the single trigger: resolves the port the way the
CLI does (--port -> VLLM_PORT in .env -> 8000), arms the gate via
LOBES_SMOKE_BASE_URL, runs pytest, prints a human summary + pass/fail exit.
Fail-not-skip: the module skips cleanly when LOBES_SMOKE_BASE_URL is unset
(offline suite stays green: 1169 passed, 14 skipped) but FAILS — never
skips — on any fault once armed. 429 (pressure shed #88) and 503+Retry-After
(honest warming/dead-owner) are treated as reachable so a busy box never goes
red. Stdlib only (urllib/json/struct).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
CHANGELOG covers the twelve merged tasks. Also records this session's findings into the in-repo eidetic store: the frozen fleet image pin (#99), the sticky swap-occupancy pressure trigger (#100), and senses' silent audio drop (#101). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
PR Summary by QodoEnforce “advertised ⇒ reachable” across gateway, CLI, and fleet templates
AI Description
Diagram
High-Level Assessment
Files changed (43)
|
Code Review by Qodo
Context used✅ Tickets:
🎫 Gateway returns terminal 404 'model does not exist' for a listed model during a backend reload window 🎫 Make Spark default serve Qwen3.6 + Gemma4 duo✅ Compliance rules (platform):
26 rules✅ Skills:
doc-test-alignment, cicd 1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
reachable_origin echoed the request Host header into every role's endpoint field unsanitised — a client-controlled value (path traversal, script markup, userinfo-style credential injection like 127.0.0.1:8001@attacker.test) reflected straight into the /capabilities JSON body, flagged as a SonarCloud BLOCKER (pythonsecurity:S5131). Add a strict host-authority allowlist regex (hostname/IPv4 or bracketed IPv6, optional :port) and gate the Host-header echo behind it. A Host that fails validation degrades to None, same as no Host header at all, so the endpoint comes back empty rather than reflecting attacker input. GATEWAY_PUBLIC_URL (trusted operator config) is untouched and keeps winning first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
stop() joined the background refresh thread with a bounded timeout and then unconditionally cleared self._thread. Since a single refresh pass probes every target sequentially (worst case len(targets) * timeout), a slow probe / short timeout could outlast the join, leaving the thread alive while self._thread was already nulled. A later start() then saw _thread is None and spawned a second, overlapping refresh thread — extra probe load and inaccurate lifecycle state (Qodo finding on PR #102). - stop() only clears self._thread when the join actually observes the thread exit; otherwise it leaves the reference in place so a live thread is never orphaned. The join bound is also widened to len(targets) * timeout + 1.0 so a clean shutdown normally completes within the call instead of racing it. - start() now treats an existing-but-dead thread reference as restartable (clears it before spawning a fresh thread) but still refuses to spawn a second thread while one is genuinely alive. - Added tests/test_gateway_readiness.py coverage that reproduces the race with an Event-blocked probe (fails against the old stop()), proves start() after an incomplete stop() never creates a second live thread, proves a stale-but-dead reference is replaced on restart, and proves stop() stays non-hanging and idempotent under the race. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
Qodo action-required finding on PR #102: t7 added a top-level `source` sibling next to the six role keys, so `lobes capabilities --json` no longer matched the gateway's own `GET /capabilities` shape byte-for-byte — a strict `set(payload.keys()) == ROLES` check broke, ironic given t7's whole point was making the CLI and gateway agree. `--json` output is now the bare six-role dict in every mode, verbatim in gateway mode and identically shaped in the offline fallback. The offline/gateway distinction moves out-of-band: the human table keeps its `# source: ...` header, and `--json` mode writes a one-line offline notice to stderr (never into the stdout JSON object) only when it degrades to the `.env` fallback. Updates the CLI-and-gateway-agree live gate (test_live_capabilities.py) to detect the offline fallback via stderr instead of the now-removed `source` key, and adds a dedicated regression test asserting gateway-mode JSON keys equal exactly the six role names.
…role; source moves to stderr (Qodo)
…ar S5131 BLOCKER)
- Security (S5131 BLOCKER): validate the Host header against a strict host-authority allowlist before advertising it as an origin. The c29 change reflected unsanitized client input into every role's /capabilities endpoint; an invalid host now yields an empty endpoint, like an absent header. The operator override GATEWAY_PUBLIC_URL is unaffected. +13 tests. - Qodo (correctness): `lobes capabilities --json` top level is keyed strictly by role; the live/offline signal moved from a top-level `source` key to a stderr notice, so the CLI payload is byte-identical to the gateway's and a set(keys)==ROLES consumer never trips. +1 test. - Qodo (reliability): ReadinessCache.stop() only clears its thread reference once the thread has actually exited, and start() refuses to spawn a second live thread — no overlapping refresh loops. +4 tests. Pre-existing realtime/* code smells Sonar surfaced are out of scope (not in this PR's diff; last touched 2026-05-31 / 06-21) and are left untouched. 1187 passed, 14 skipped. black/isort/flake8/bandit clean; afi doctor 26/26. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
Contributor
Author
Review round — SonarCloud + Qodo triage (
|
ReadinessCache seeds every backend to None via a constant-value dict comprehension; SonarCloud flags this as S7519 (prefer dict.fromkeys). None is immutable so there's no shared-mutable-default hazard here.
S8714 (test try/except in tests/test_live_capabilities.py) accepted in SonarCloud with rationale: it is a unit-test rule applied to an operator-facing live gate that consumes an arbitrary base URL, where the caught body preview is the diagnostic that reveals an operator dialing the wrong endpoint (200 + HTML). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MrghbUdRuVa2Y98EV8RHQg
|
Contributor
Author
SonarCloud triage — 2 remaining issues (
|
Contributor
Author
Diverse review — second model, clean passRan an independent review of Verdict: no bugs. It independently confirmed each control:
One observation worth recording (pre-existing, out of scope):
|
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
lobes never advertises a capability it cannot serve
Five open items — #92, #91, #95 (dup of #92), #74, #69 — turned out to be one disease: lobes advertises configuration and calls it reachability, and nothing ever dials what it advertises. This PR makes "advertised ⇒ reachable" an executable invariant.
Built through the devague pipeline:
/thinkspec →/spec-to-plan(10 tasks, 6 waves, 39/39 coverage targets) →/assign-to-workforce(12 tasks merged, each TDD-gated: tests green before and after every merge).The issues named the symptoms correctly and the causes wrongly
#92 is not a regression of #87 — the fix was never deployed
Dockerfile.gatewayrunspip install "lobes-cli==${MODEL_GEAR_VERSION}";lobes initwrites that pin once and no verb ever bumps it. The rig ran three lobes versions at once (gateway 0.36.0, realtime 0.34.1, CLI 0.39.0) while 0.37.0/0.38.0/0.39.0 sat published on PyPI. Filed as #99.And
:8000was never a dead port — it isreachy-mini-dae, an unrelated uvicorn service. The advertised endpoint pointed a client at a foreign daemon.#91 is not a backend-reload window — it is cross-model failover
handle_postrewrote the model id once, before the failover loop, whileorder_backendsoffered every same-task generate backend as a retry target —[primary, multimodal, multimodal-coder, middle]for cortex. A cortex 5xx therefore retried the same body, still naming the Qwen model, against the Gemma backend, which correctly answered404 does not exist, and the4xx = client error, no failoverrule relayed that as terminal.Proven from the containers' own logs, in both directions:
That last line is the Qwen container being handed the Gemma id. And
tests/test_gateway_routing.py::test_order_backends_generate_still_failovers_between_generate_backendsasserted this behaviour as intended, which is why it survived review.Resolution (user-decided): no cross-backend failover at all.
order_backendsreturns at most one backend. A dead / unreachable / warming owner yields 503 +Retry-After,type: backend_unavailable. A caller who asks for cortex can never silently receive Gemma — which also protects #81'sfinal_authoritycontract.Three bugs found while chasing it, now filed
AUDIO_URLreaches the gateway only viadocker-compose.audio.yml, sostt/ttsadvertisedready=trueon a path returning404 "audio endpoints are not configured"._optional_backendwired a backend from*_SERVED_NAMEalone against an inventeddefault_url./v1/modelslisted 6 models against 4 running containers.sensesadvertises audio intake and silently drops it.Plus #98 (the
EngineDeadErrorthat triggers #91, ~once/day), #99 (the frozen image pin), #100 (the pressure policy triggers on sticky swap occupancy; PSI showedfull avg10=0.00while the gateway shed 100% of generate traffic).Neither contract surface was authoritative — and they were wrong in opposite directions
For the generate roles the gateway JSON was wrong (its internal
:8000) and the CLI was right. For the audio roles the CLI was wrong (ready=truefrom a string in.env) and the gateway was right.build_role_registryreally was one builder — fed two differentgateway_urlvalues. "One source of truth" held for the shape and failed for the origin.Fixed structurally:
lobes capabilitiesnow renders the running gateway'sGET /capabilities, falling back to an offline view tagged"source": "offline"withready=falseeverywhere. Two derivations became one.#74: one box proven, one box disproven
The unchecked box asked to "confirm image+text and audio+text" on
coolthor/gemma-4-12B-it-NVFP4A16. Token accounting settles both:prompt_tokens"Red"/"Blue"— verified against ground truth""The vision tower is wired and reads pixels (with a negative control: a blue image correctly fails a
"red"assertion). Audio is dropped, not rejected — the caller gets200 OKand a fluent answer that ignored the audio. The checkpoint declaresaudio_config/audio_token_id, so this is a vLLMgemma4_unifiedgap (#101).Why it went unnoticed: the existing "audio+text ✓" check asserted only
HTTP 200 + non-empty contentagainst a 1×1 placeholder. It proved the wire, not the perception.sensesis now documented vision-only;stt(Parakeet) is the supported speech path and works —tts("banana") → stt → "Banana."The gate
Per the requirement "a local test before PRs that live-tests our capabilities", "runs locally and once triggered, runs without intervention":
Five checks, each naming the issue it maps to. It fails rather than skips when armed.
429(pressure shed, #88) and503withRetry-Aftercount as reachable; only a404, a connection failure, aRetry-After-less503, or a bare5xxare faults.It is red on the reference rig right now — via #91 (over-listed candidate models) and #99 (deployed gateway reports no
/healthversion). That is the gate working: this is the check whose absence let a merged fix sit undeployed for five days while #92 was filed against it.Notable
The spec itself was wrong, and testing the requirement caught it.
c11required the compose to defaultGATEWAY_PUBLIC_URLfromVLLM_PORT. A subagent implemented it faithfully. Butpublic_urloutranks theHostheader, so a defaulted value would have told every LAN and tunnel client to dial its own loopback — reintroducing #92 one rung out. Caught by running the real resolver against a simulated remoteHostbefore merging; amended toc29(explicit override > Host > empty, never the internal port) and committed with the reasoning. A negative regression guard (test_default_has_no_localhost_or_vllm_port) now stops anyone "helpfully" restoring the default.1169 tests pass (from 1084).
black/isort/flake8/banditclean;afi cli doctor . --strict26/26.Closes #92 · Closes #91 · Closes #95 · Closes #97 · Closes #96 · Closes #74 · Closes #69
Follow-ups: #98 · #99 · #100 · #101