diff --git a/.devague/current b/.devague/current index c10e082..6fb4fdf 100644 --- a/.devague/current +++ b/.devague/current @@ -1 +1 @@ -lobes-preserves-qwen-thinking-traces-across-multi +lobes-never-advertises-a-capability-it-cannot-serv diff --git a/.devague/current_plan b/.devague/current_plan index c10e082..6fb4fdf 100644 --- a/.devague/current_plan +++ b/.devague/current_plan @@ -1 +1 @@ -lobes-preserves-qwen-thinking-traces-across-multi +lobes-never-advertises-a-capability-it-cannot-serv diff --git a/.devague/frames/lobes-never-advertises-a-capability-it-cannot-serv.json b/.devague/frames/lobes-never-advertises-a-capability-it-cannot-serv.json new file mode 100644 index 0000000..a337d11 --- /dev/null +++ b/.devague/frames/lobes-never-advertises-a-capability-it-cannot-serv.json @@ -0,0 +1,467 @@ +{ + "slug": "lobes-never-advertises-a-capability-it-cannot-serv", + "title": "lobes never advertises a capability it cannot serve: every endpoint, model and role in the machine-readable contract is proven reachable, and a fleet fault degrades to a retryable 503 instead of a terminal 404", + "schema_version": 1, + "status": "exported", + "created": "2026-07-09T05:14:35Z", + "updated": "2026-07-09T07:02:42Z", + "claims": [ + { + "id": "c1", + "kind": "announcement", + "text": "lobes never advertises a capability it cannot serve: every endpoint, model and role in the machine-readable contract is proven reachable, and a fleet fault degrades to a retryable 503 instead of a terminal 404", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h1", + "text": "On the live rig, for EVERY role in the contract, the advertised endpoint+path answers a real request with a non-404 status; and no model listed by GET /v1/models returns a 404 'does not exist' from POST /v1/chat/completions on the same origin.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c2", + "kind": "audience", + "text": "Programmatic consumers of the lobes contract \u2014 Colleague's resolve_role_base_url dialing GET /capabilities, and any long-running agent loop driving /v1/chat/completions through the gateway \u2014 plus the lobes operator who runs the fleet on the DGX Spark.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h2", + "text": "Colleague's resolve_role_base_url, given ONLY GET /capabilities and no COLLEAGUE_*_BASE_URL override, reaches cortex and senses and gets an answer \u2014 the scenario that failed on 2026-07-06 and again on 2026-07-09.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c3", + "kind": "before_state", + "text": "lobes advertises CONFIGURATION and calls it REACHABILITY, and nothing ever dials what it advertises. GET /capabilities returns endpoint=http://localhost:8000 with ready=true for every role while the gateway is published on :8001 and :8000 is an unrelated uvicorn service (reachy-mini-dae) answering {\"detail\":\"Not Found\"}. GET /v1/models lists 6 models while only 4 vLLM containers run. RoleInfo.ready is a config fact (==loaded), not a probe \u2014 its own docstring says so. And the deployed gateway silently runs lobes 0.36.0 while main is 0.39.0, so the #87 fix that shipped in 0.38.0 is not even in the running image.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h3", + "text": "A test asserts that the origin the gateway advertises never equals its own internal listen port when a published port differs, and that GET /capabilities and 'lobes capabilities --json' return byte-identical endpoint values for the same deployment.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c4", + "kind": "before_state", + "text": "A fleet fault is laundered into a terminal client error. handle_post rewrites the model id ONCE before the failover loop; order_backends offers every same-task generate backend as a failover candidate ([primary, multimodal, multimodal-coder, middle] for cortex). When the cortex vLLM returns >=500 (EngineDeadError, observed 2026-07-09 04:08:34) the gateway retries the same body \u2014 still naming the Qwen model \u2014 against the Gemma backend, which correctly 404s, and '4xx -> commit to this backend, no failover' relays that terminal 404 to the client. The primary container's logs show the symmetric case at 04:34:01 and 04:39:01: 'The model coolthor/gemma-4-12B-it-NVFP4A16 does not exist.' logged BY the Qwen container.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h4", + "text": "With the cortex backend forced to 5xx or made unreachable, a request naming the cortex model NEVER reaches the Gemma backend, and the client receives 503 + Retry-After \u2014 reproducible in a unit test with a fake fleet, not just observed in production logs.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c5", + "kind": "why_it_matters", + "text": "A single unlucky turn kills a whole agent loop. A terminal 404 tells a well-behaved client the model will NEVER exist, so colleague work items abort after N steps with status=error and no output \u2014 while the model is listed in /v1/models before and after. And a lobes-discovered senses endpoint degrades instantly (~0.002s per call) because the advertised origin is dead. This has now regressed three times across #87 -> #90 -> #92: the shape keeps breaking because no test ever dials the deployed artifact.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h5", + "text": "A multi-step client loop survives an owner-backend fault: the turn that would previously have died on a terminal 404 instead receives a retryable status, and a client with ordinary backoff completes the run.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c6", + "kind": "after_state", + "text": "Every advertised capability is proven reachable before it is advertised, and every fleet fault surfaces as a retryable status. /capabilities advertises an origin derived from configured truth (never the gateway's internal listen port) and a ready flag backed by a live backend probe. /v1/models lists only backends that actually answer. A request naming a model that /v1/models lists never receives a 404 'does not exist' from the same origin \u2014 an owner that is dead yields 503 + Retry-After. The CLI and the gateway agree on the endpoint byte-for-byte. And an executable verb dials every advertised role endpoint+path so 'advertised implies reachable' is a test, not a promise.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h6", + "text": "Every claim in the after_state is asserted by an executable check, and each of those checks FAILS when run against today's HEAD + today's deployed rig.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c7", + "kind": "before_state", + "text": "The same disease reaches the roles themselves. 'senses' advertises responsibilities=[intake, normalize_input, ...] and the catalog claims image+text and audio+text, but every probe lobes owns is text-only: roles_measure.py:183 sends 'Write a short paragraph.' to senses, and assess.py builds no multimodal content parts at all. The image/audio probes exist only in tests/test_smoke_duo.py:287-347, skipped unless LOBES_SMOKE_BASE_URL is set, with no recorded run. The image+text/audio+text evidence in catalog.py:282 belongs to the sakamakismile coder checkpoint; docs/gemma-4-12b-nvfp4.md:279 states outright that those checks were 'not independently re-run against the base [coolthor] checkpoint specifically'. coolthor became the default in 0.35.0 on MTP decode throughput \u2014 orthogonal to the image+audio precondition #74 set.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h7", + "text": "lobes' own tooling sends a real image and a real audio clip to the senses role and verifies the answer against known ground truth (not merely that the response was 200 with non-empty content), and the recorded evidence names the coolthor checkpoint.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c8", + "kind": "boundary", + "text": "NOT a service mesh, a load balancer, or a health-check-per-request proxy: the gateway stays a stdlib reverse proxy with a bounded, cached readiness signal \u2014 no per-request upstream probe on the hot path. NOT a claim about answer quality: lobes keeps emitting runtime metrics only, never task-quality claims (the #81 h15 honesty condition still holds). NOT a rewrite of the role vocabulary, the tier aliases, or the compose topology. NOT an attempt to keep cortex alive through EngineDeadError \u2014 recovering the vLLM engine is out of scope; only the gateway's HONESTY about a dead engine is in scope.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h8", + "text": "The gateway's POST hot path opens no additional upstream probe connection per request: the readiness signal is read from a cache, exactly as pressure already is.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c9", + "kind": "non_goal", + "text": "Not adding retry/backoff logic inside the gateway on the client's behalf. The gateway's job is to return an honest, correctly-classified status (503 + Retry-After) and let the client's own backoff policy decide. Queuing a request until a backend finishes warming (issue #91's option 2) is explicitly NOT chosen.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c10", + "kind": "success_signal", + "text": "Five falsifiable checks, each runnable: (1) curl GET :8001/capabilities | jq '.[].endpoint' returns the SAME origin as 'lobes capabilities --json' for the same deployment, and that origin answers 200 on its advertised path. (2) With the cortex container stopped, POST /v1/chat/completions model= returns 503 + Retry-After \u2014 never 404, never a Gemma answer. (3) GET /v1/models lists exactly the models whose backends answer. (4) 'lobes doctor' (or the new verify verb) exits non-zero when the deployed gateway's lobes version differs from the CLI's. (5) The live smoke layer posts a real image and a real audio clip at coolthor and records the pass in docs, closing #74 and #69's criterion 7 with evidence rather than assertion.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h9", + "text": "Each of the five success checks is a command a reviewer can paste and watch fail on main, then pass on the branch.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c11", + "kind": "requirement", + "text": "The gateway advertises an origin from CONFIGURED TRUTH, never inference: the fleet compose injects GATEWAY_PUBLIC_URL (derived from the published VLLM_PORT) into the gateway container, and reachable_origin prefers it over the request Host header. When neither configured truth nor a Host header is available the gateway MUST NOT fabricate an absolute URL from its internal listen port.", + "origin": "llm", + "status": "rejected", + "honesty_conditions": [ + { + "id": "h10", + "text": "With GATEWAY_PUBLIC_URL unset AND the request carrying no Host header, the gateway emits an endpoint that is either absent/empty or explicitly relative \u2014 never an absolute URL built from GATEWAY_PORT.", + "status": "rejected" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c12", + "kind": "requirement", + "text": "A model id returned by GET /v1/models never receives a 404 'model does not exist' from POST /v1/chat/completions on the same origin. This is the single invariant issue #91 asks for, stated as a test.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h11", + "text": "A test enumerates GET /v1/models and, for each id, POSTs a minimal completion and asserts the status is never 404 \u2014 run against a fake fleet where the owner backend is dead.", + "status": "rejected" + }, + { + "id": "h23", + "text": "The listed-never-404 invariant is tested at the RACE, not at rest: a fake fleet lists model M, the owner is then killed, and a completion naming M returns 503 + Retry-After \u2014 never 404. A second test asserts the converse: an id that was NEVER in /v1/models does not silently get served by the default backend under a different model's weights.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c13", + "kind": "requirement", + "text": "Failover never forwards a request to a backend that does not serve the resolved model. The failover equivalence class is the SERVED MODEL, not the task family. Cross-backend retry stays legal only when the caller addressed a capability TIER ALIAS (main/minor/multimodal/cheap/normal/hard), in which case the body is re-resolved and re-rewritten per attempt rather than once before the loop.", + "origin": "llm", + "status": "rejected", + "honesty_conditions": [ + { + "id": "h12", + "text": "order_backends(table, ) returns at most the backends that serve that exact model; the existing test_order_backends_generate_still_failovers_between_generate_backends is INVERTED to assert the new contract, and a tier-alias request re-rewrites the body per attempt.", + "status": "rejected" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c14", + "kind": "requirement", + "text": "When the owner backend of a requested model is dead, unreachable, or warming, and no legal failover exists, the gateway returns 503 + Retry-After with a distinguishable error type (e.g. type=backend_unavailable), never 404 and never 502-as-terminal.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h13", + "text": "A dead owner yields status 503 with a Retry-After header and an error type distinguishable from both 'model unknown' and 'all backends down', asserted in a unit test.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c15", + "kind": "requirement", + "text": "GET /v1/models and GET /capabilities reflect a bounded, cached LIVE readiness signal, not a config fact. RoleInfo.ready stops being an alias of loaded for the four gateway-fronted roles, exactly as it already stopped being one for stt/tts in #89. Phantom backends (wired by *_SERVED_NAME with no running container) never appear as ready.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h14", + "text": "A backend wired in config but with no listening container reports ready=false and is absent from GET /v1/models, without the gateway probing on the request hot path.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c16", + "kind": "requirement", + "text": "'advertised implies reachable' becomes executable: a read-only verb dials every role's advertised endpoint+path and reports pass/fail per role, and it is the thing CI and the operator run. It also compares the deployed gateway container's lobes.__version__ against the CLI's and fails on skew \u2014 the defect that made #92 look like a code regression when it was a stale 0.36.0 image.", + "origin": "llm", + "status": "rejected", + "honesty_conditions": [ + { + "id": "h15", + "text": "The verb exits non-zero on the CURRENT live rig (stale 0.36.0 gateway, :8000 endpoint, 404ing audio path) and exits zero after the fix is deployed.", + "status": "rejected" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c17", + "kind": "requirement", + "text": "The multimodal role is proven multimodal by lobes' own tooling: the live smoke layer (tests/test_smoke_duo.py) is executed against coolthor on the DGX Spark and its image+text and audio+text results are recorded in docs/gemma-4-12b-nvfp4.md, replacing the standing admission at line 279 that those checks were never re-run against the base checkpoint. This closes #74's unchecked box and #69's criterion 7 with evidence instead of assertion.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h16", + "text": "docs/gemma-4-12b-nvfp4.md no longer contains the admission that the content-correctness checks 'were not independently re-run against the base checkpoint specifically', because they were re-run and recorded.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c18", + "kind": "before_state", + "text": "NEITHER surface is authoritative, and they are wrong in OPPOSITE directions. For the generate roles the gateway JSON is wrong (endpoint = its internal :8000) and the CLI is right (:8001 from .env VLLM_PORT). For the audio roles the CLI is wrong (ready=true, loaded=true, endpoint=:8001/v1/audio/speech \u2014 read from .env AUDIO_URL) and the gateway is right (ready=false, loaded=false, endpoint=''). Dialing the CLI-advertised tts path returns 404 'audio endpoints are not configured on this deployment'. Root cause: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml:144, so whether stt/tts work depends on whether the operator remembered the overlay -f flag \u2014 while the CLI reads the merged .env and assumes wired. This is #92's exact shape (ready=true on a 404 path) for the audio roles, it is present in main and in the packaged template, and no issue tracks it.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h19", + "text": "A test asserts that 'lobes capabilities' and GET /capabilities agree on ready/loaded/endpoint for ALL SIX roles \u2014 including stt/tts on a deployment where the audio overlay is not composed in \u2014 and that the audio roles never report ready=true while their advertised path returns 404.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c19", + "kind": "before_state", + "text": "GET /v1/models advertises phantom backends. _optional_backend wires a backend when *_BASE_URL OR *_SERVED_NAME is set, and falls back to a default_url naming a container that need not exist. On the live rig that yields 6 advertised models against 4 running vLLM containers: 'multimodal-coder' (http://vllm-multimodal-coder:8000) and 'middle' (http://vllm-middle:8000) have no container at all, yet they are listed to clients AND sit in the generate failover chain, which for the cortex model is [primary, multimodal, multimodal-coder, middle].", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h20", + "text": "On a deployment where MULTIMODAL_CODER_SERVED_NAME and MIDDLE_SERVED_NAME are set but their containers are absent, GET /v1/models returns exactly the ids whose backends answer, and those absent backends never appear in any failover chain.", + "status": "rejected" + }, + { + "id": "h24", + "text": "With MULTIMODAL_CODER_SERVED_NAME and MIDDLE_SERVED_NAME set but no BASE_URL and no container, build_config wires NEITHER backend, GET /v1/models returns exactly the four running ids, and resolve_model('nvidia/Qwen3-14B-NVFP4') does not resolve to the primary's weights.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c20", + "kind": "assumption", + "text": "The vLLM engine crash that triggers all this (vllm.v1.engine.exceptions.EngineDeadError on the cortex container, RestartCount=3) is a separate fault whose ROOT CAUSE is out of scope here. This spec makes lobes honest about a dead engine; it does not attempt to keep the engine alive. If EngineDeadError turns out to be caused by lobes' own serve flags (MTP + max_num_seqs), that is a distinct issue.", + "origin": "llm", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c21", + "kind": "decision", + "text": "The unifying repair is a single invariant with three enforcement points: (1) ONE origin resolver \u2014 configured truth (GATEWAY_PUBLIC_URL, injected from the published VLLM_PORT) beats Host header beats nothing-at-all; the internal listen port is never advertised. (2) ONE readiness source \u2014 a bounded cached probe, shared by /v1/models, /capabilities.ready and the failover order, so 'ready' means 'answered recently'. (3) ONE executable check \u2014 a read-only verb that dials every advertised endpoint+path and the deployed gateway's version, run by CI and the operator.", + "origin": "llm", + "status": "rejected", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c22", + "kind": "requirement", + "text": "A LOCAL live-test of the capabilities contract runs before every PR. It is not a CI job (CI has no GPU and no fleet); it is a local gate the developer runs against the running deployment, in the same live-gated seam tests/test_smoke_duo.py already uses (LOBES_SMOKE_BASE_URL). It must fail loudly rather than skip silently when the operator intended to run it, so that 'advertised implies reachable' is checked against the DEPLOYED artifact before a PR claims to have fixed it \u2014 the exact step whose absence let #87 regress into #92.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h17", + "text": "The local pre-PR live test, pointed at a deployment whose gateway is stale or whose audio overlay is unwired, FAILS \u2014 it does not skip, and it does not pass.", + "status": "confirmed" + }, + { + "id": "h18", + "text": "The gate is ONE trigger and then unattended: a single local command runs the whole capabilities live-test to a pass/fail verdict with no prompts, no manual steps, and no per-role babysitting \u2014 the developer types it, walks away, and reads the exit code.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c23", + "kind": "requirement", + "text": "NO cross-backend failover. Every request resolves to exactly ONE backend and is attempted only there. An explicit model id goes to its owner; a capability tier alias resolves once, up front, via the existing static tier_aliases upward fallback (a tier whose gear is not WIRED still maps to a higher rung at table-build time). There is no runtime retry against a second backend, so a caller who asked for cortex can never silently receive a Gemma answer. A dead/unreachable/warming owner yields 503 + Retry-After. order_backends collapses to a single-element list and rewrite_model stays outside any loop.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h21", + "text": "A unit test with a fake fleet proves that a request naming the cortex model, with the primary backend dead, NEVER opens a connection to the multimodal backend, and returns 503 + Retry-After. order_backends(table, served) returns exactly one backend for every input.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c24", + "kind": "decision", + "text": "Readiness becomes a bounded BACKGROUND cached probe, mirroring the existing PressureCache: each backend's /health is polled on an interval off the request path. The one cached signal feeds GET /v1/models, GET /capabilities .ready, and the wiring/advertisement decisions. The POST hot path opens no probe connection.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c25", + "kind": "decision", + "text": "Phantom backends are stopped by BOTH gates: (a) a backend is wired only when its *_BASE_URL is set \u2014 the 'or *_SERVED_NAME' clause is dropped, matching the MINOR_BASE_URL convention the fleet already documents; and (b) GET /v1/models is filtered by the live readiness signal, so a wired-but-dead backend is not advertised.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c26", + "kind": "decision", + "text": "Issue #69's 'disabled-by-default DSpark experiment entry' is closed as ANSWERED-NEGATIVE: no catalog entry is shipped for deepseek-ai/dspark_gemma4_12b_block7, because #75 proved Gemma4DSparkModel does not load on vLLM 0.23. The repair is documentary \u2014 docs/gemma4-mtp-draft.md still presents DSpark as 'the ONE route task t3 should wire next' with no superseded banner and must carry the invalidity finding.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c27", + "kind": "requirement", + "text": "The pre-PR gate detects DEPLOYED-ARTIFACT skew, not just source correctness: it compares the running gateway container's lobes.__version__ against the CLI's and fails on mismatch, and it dials every advertised role endpoint+path. This is the check whose absence let #87's fix ship in 0.38.0 while the rig kept running 0.36.0 \u2014 making #92 look like a code regression when the code was already correct and merely undeployed.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h22", + "text": "Run against the rig as it stands right now (gateway 0.36.0, CLI 0.39.0, endpoint :8000, /v1/audio/speech 404ing), the gate exits non-zero and names all three faults; after redeploy it exits zero.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + }, + { + "id": "c28", + "kind": "decision", + "text": "The unifying repair is one invariant with three enforcement points: (1) ONE origin resolver \u2014 configured truth (GATEWAY_PUBLIC_URL, injected from the published VLLM_PORT) beats the request Host header beats nothing; the internal listen port is never advertised. (2) ONE readiness source \u2014 a background cached probe shared by /v1/models and /capabilities.ready. (3) ONE executable check \u2014 a LOCAL, single-trigger, unattended pre-PR live test that dials every advertised endpoint+path and the deployed gateway's version.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [], + "hard_questions": [], + "links": [] + }, + { + "id": "c29", + "kind": "requirement", + "text": "The gateway resolves its advertised origin as: an explicit operator override (GATEWAY_PUBLIC_URL \u2014 for a tunnel or a Host-rewriting reverse proxy) FIRST; else the origin the client actually dialed, echoed from the request Host header; else NOTHING \u2014 an empty endpoint. It must never fabricate an absolute URL from its internal listen port (GATEWAY_PORT), and it must never default GATEWAY_PUBLIC_URL to a localhost URL, because a defaulted public_url outranks the Host header and would tell every LAN/tunnel client to dial its own loopback \u2014 reintroducing the #92 defect in a new place. Each caller therefore receives an origin correct for itself. The base fleet compose still passes AUDIO_URL (empty by default; the audio overlay supplies the real value), which is the #96 half of this change.", + "origin": "user", + "status": "confirmed", + "honesty_conditions": [ + { + "id": "h25", + "text": "With GATEWAY_PUBLIC_URL unset and no Host header, the gateway emits an empty endpoint \u2014 never an absolute URL built from GATEWAY_PORT. With GATEWAY_PUBLIC_URL unset and Host='spark.local:8001', it advertises http://spark.local:8001, not http://localhost:8001. With GATEWAY_PUBLIC_URL set, it wins over any Host header.", + "status": "confirmed" + } + ], + "hard_questions": [], + "links": [] + } + ], + "open_vagueness": [ + { + "id": "v1", + "text": "Should the gateway's cached readiness probe be a background thread (like PressureCache) or a lazy TTL-cached probe on the /v1/models + /capabilities read path? Background costs a thread and probes when idle; lazy adds latency to the first read after a TTL expiry. Both keep the hot POST path probe-free.", + "kind": "unknown_nonblocking", + "claim_id": null + }, + { + "id": "v2", + "text": "Whether requiring *_BASE_URL to wire a backend (dropping the 'OR *_SERVED_NAME' clause) breaks any existing deployment that set only the served-name var. The fleet template always sets both, but a hand-edited .env might not.", + "kind": "unknown_nonblocking", + "claim_id": null + }, + { + "id": "v3", + "text": "RESOLVED by user decision (see the DSpark decision claim): close #69's disabled-entry criterion as answered-negative and correct docs/gemma4-mtp-draft.md, which still presents DSpark as the next route to wire. Follow-up: revisit if a vLLM lands that registers Gemma4DSparkModel.", + "kind": "follow_up", + "claim_id": null + } + ] +} diff --git a/.devague/plans/lobes-never-advertises-a-capability-it-cannot-serv.json b/.devague/plans/lobes-never-advertises-a-capability-it-cannot-serv.json new file mode 100644 index 0000000..dd4730a --- /dev/null +++ b/.devague/plans/lobes-never-advertises-a-capability-it-cannot-serv.json @@ -0,0 +1,457 @@ +{ + "slug": "lobes-never-advertises-a-capability-it-cannot-serv", + "title": "lobes never advertises a capability it cannot serve: every endpoint, model and role in the machine-readable contract is proven reachable, and a fleet fault degrades to a retryable 503 instead of a terminal 404", + "frame_slug": "lobes-never-advertises-a-capability-it-cannot-serv", + "schema_version": 1, + "status": "exported", + "created": "2026-07-09T05:37:28Z", + "updated": "2026-07-09T07:03:39Z", + "targets": [ + { + "id": "c1", + "kind": "announcement", + "text": "lobes never advertises a capability it cannot serve: every endpoint, model and role in the machine-readable contract is proven reachable, and a fleet fault degrades to a retryable 503 instead of a terminal 404" + }, + { + "id": "h1", + "kind": "honesty", + "text": "On the live rig, for EVERY role in the contract, the advertised endpoint+path answers a real request with a non-404 status; and no model listed by GET /v1/models returns a 404 'does not exist' from POST /v1/chat/completions on the same origin." + }, + { + "id": "c2", + "kind": "audience", + "text": "Programmatic consumers of the lobes contract \u2014 Colleague's resolve_role_base_url dialing GET /capabilities, and any long-running agent loop driving /v1/chat/completions through the gateway \u2014 plus the lobes operator who runs the fleet on the DGX Spark." + }, + { + "id": "h2", + "kind": "honesty", + "text": "Colleague's resolve_role_base_url, given ONLY GET /capabilities and no COLLEAGUE_*_BASE_URL override, reaches cortex and senses and gets an answer \u2014 the scenario that failed on 2026-07-06 and again on 2026-07-09." + }, + { + "id": "c3", + "kind": "before_state", + "text": "lobes advertises CONFIGURATION and calls it REACHABILITY, and nothing ever dials what it advertises. GET /capabilities returns endpoint=http://localhost:8000 with ready=true for every role while the gateway is published on :8001 and :8000 is an unrelated uvicorn service (reachy-mini-dae) answering {\"detail\":\"Not Found\"}. GET /v1/models lists 6 models while only 4 vLLM containers run. RoleInfo.ready is a config fact (==loaded), not a probe \u2014 its own docstring says so. And the deployed gateway silently runs lobes 0.36.0 while main is 0.39.0, so the #87 fix that shipped in 0.38.0 is not even in the running image." + }, + { + "id": "h3", + "kind": "honesty", + "text": "A test asserts that the origin the gateway advertises never equals its own internal listen port when a published port differs, and that GET /capabilities and 'lobes capabilities --json' return byte-identical endpoint values for the same deployment." + }, + { + "id": "c4", + "kind": "before_state", + "text": "A fleet fault is laundered into a terminal client error. handle_post rewrites the model id ONCE before the failover loop; order_backends offers every same-task generate backend as a failover candidate ([primary, multimodal, multimodal-coder, middle] for cortex). When the cortex vLLM returns >=500 (EngineDeadError, observed 2026-07-09 04:08:34) the gateway retries the same body \u2014 still naming the Qwen model \u2014 against the Gemma backend, which correctly 404s, and '4xx -> commit to this backend, no failover' relays that terminal 404 to the client. The primary container's logs show the symmetric case at 04:34:01 and 04:39:01: 'The model coolthor/gemma-4-12B-it-NVFP4A16 does not exist.' logged BY the Qwen container." + }, + { + "id": "h4", + "kind": "honesty", + "text": "With the cortex backend forced to 5xx or made unreachable, a request naming the cortex model NEVER reaches the Gemma backend, and the client receives 503 + Retry-After \u2014 reproducible in a unit test with a fake fleet, not just observed in production logs." + }, + { + "id": "c5", + "kind": "why_it_matters", + "text": "A single unlucky turn kills a whole agent loop. A terminal 404 tells a well-behaved client the model will NEVER exist, so colleague work items abort after N steps with status=error and no output \u2014 while the model is listed in /v1/models before and after. And a lobes-discovered senses endpoint degrades instantly (~0.002s per call) because the advertised origin is dead. This has now regressed three times across #87 -> #90 -> #92: the shape keeps breaking because no test ever dials the deployed artifact." + }, + { + "id": "h5", + "kind": "honesty", + "text": "A multi-step client loop survives an owner-backend fault: the turn that would previously have died on a terminal 404 instead receives a retryable status, and a client with ordinary backoff completes the run." + }, + { + "id": "c6", + "kind": "after_state", + "text": "Every advertised capability is proven reachable before it is advertised, and every fleet fault surfaces as a retryable status. /capabilities advertises an origin derived from configured truth (never the gateway's internal listen port) and a ready flag backed by a live backend probe. /v1/models lists only backends that actually answer. A request naming a model that /v1/models lists never receives a 404 'does not exist' from the same origin \u2014 an owner that is dead yields 503 + Retry-After. The CLI and the gateway agree on the endpoint byte-for-byte. And an executable verb dials every advertised role endpoint+path so 'advertised implies reachable' is a test, not a promise." + }, + { + "id": "h6", + "kind": "honesty", + "text": "Every claim in the after_state is asserted by an executable check, and each of those checks FAILS when run against today's HEAD + today's deployed rig." + }, + { + "id": "c7", + "kind": "before_state", + "text": "The same disease reaches the roles themselves. 'senses' advertises responsibilities=[intake, normalize_input, ...] and the catalog claims image+text and audio+text, but every probe lobes owns is text-only: roles_measure.py:183 sends 'Write a short paragraph.' to senses, and assess.py builds no multimodal content parts at all. The image/audio probes exist only in tests/test_smoke_duo.py:287-347, skipped unless LOBES_SMOKE_BASE_URL is set, with no recorded run. The image+text/audio+text evidence in catalog.py:282 belongs to the sakamakismile coder checkpoint; docs/gemma-4-12b-nvfp4.md:279 states outright that those checks were 'not independently re-run against the base [coolthor] checkpoint specifically'. coolthor became the default in 0.35.0 on MTP decode throughput \u2014 orthogonal to the image+audio precondition #74 set." + }, + { + "id": "h7", + "kind": "honesty", + "text": "lobes' own tooling sends a real image and a real audio clip to the senses role and verifies the answer against known ground truth (not merely that the response was 200 with non-empty content), and the recorded evidence names the coolthor checkpoint." + }, + { + "id": "c8", + "kind": "boundary", + "text": "NOT a service mesh, a load balancer, or a health-check-per-request proxy: the gateway stays a stdlib reverse proxy with a bounded, cached readiness signal \u2014 no per-request upstream probe on the hot path. NOT a claim about answer quality: lobes keeps emitting runtime metrics only, never task-quality claims (the #81 h15 honesty condition still holds). NOT a rewrite of the role vocabulary, the tier aliases, or the compose topology. NOT an attempt to keep cortex alive through EngineDeadError \u2014 recovering the vLLM engine is out of scope; only the gateway's HONESTY about a dead engine is in scope." + }, + { + "id": "h8", + "kind": "honesty", + "text": "The gateway's POST hot path opens no additional upstream probe connection per request: the readiness signal is read from a cache, exactly as pressure already is." + }, + { + "id": "c10", + "kind": "success_signal", + "text": "Five falsifiable checks, each runnable: (1) curl GET :8001/capabilities | jq '.[].endpoint' returns the SAME origin as 'lobes capabilities --json' for the same deployment, and that origin answers 200 on its advertised path. (2) With the cortex container stopped, POST /v1/chat/completions model= returns 503 + Retry-After \u2014 never 404, never a Gemma answer. (3) GET /v1/models lists exactly the models whose backends answer. (4) 'lobes doctor' (or the new verify verb) exits non-zero when the deployed gateway's lobes version differs from the CLI's. (5) The live smoke layer posts a real image and a real audio clip at coolthor and records the pass in docs, closing #74 and #69's criterion 7 with evidence rather than assertion." + }, + { + "id": "h9", + "kind": "honesty", + "text": "Each of the five success checks is a command a reviewer can paste and watch fail on main, then pass on the branch." + }, + { + "id": "c12", + "kind": "requirement", + "text": "A model id returned by GET /v1/models never receives a 404 'model does not exist' from POST /v1/chat/completions on the same origin. This is the single invariant issue #91 asks for, stated as a test." + }, + { + "id": "h23", + "kind": "honesty", + "text": "The listed-never-404 invariant is tested at the RACE, not at rest: a fake fleet lists model M, the owner is then killed, and a completion naming M returns 503 + Retry-After \u2014 never 404. A second test asserts the converse: an id that was NEVER in /v1/models does not silently get served by the default backend under a different model's weights." + }, + { + "id": "c14", + "kind": "requirement", + "text": "When the owner backend of a requested model is dead, unreachable, or warming, and no legal failover exists, the gateway returns 503 + Retry-After with a distinguishable error type (e.g. type=backend_unavailable), never 404 and never 502-as-terminal." + }, + { + "id": "h13", + "kind": "honesty", + "text": "A dead owner yields status 503 with a Retry-After header and an error type distinguishable from both 'model unknown' and 'all backends down', asserted in a unit test." + }, + { + "id": "c15", + "kind": "requirement", + "text": "GET /v1/models and GET /capabilities reflect a bounded, cached LIVE readiness signal, not a config fact. RoleInfo.ready stops being an alias of loaded for the four gateway-fronted roles, exactly as it already stopped being one for stt/tts in #89. Phantom backends (wired by *_SERVED_NAME with no running container) never appear as ready." + }, + { + "id": "h14", + "kind": "honesty", + "text": "A backend wired in config but with no listening container reports ready=false and is absent from GET /v1/models, without the gateway probing on the request hot path." + }, + { + "id": "c17", + "kind": "requirement", + "text": "The multimodal role is proven multimodal by lobes' own tooling: the live smoke layer (tests/test_smoke_duo.py) is executed against coolthor on the DGX Spark and its image+text and audio+text results are recorded in docs/gemma-4-12b-nvfp4.md, replacing the standing admission at line 279 that those checks were never re-run against the base checkpoint. This closes #74's unchecked box and #69's criterion 7 with evidence instead of assertion." + }, + { + "id": "h16", + "kind": "honesty", + "text": "docs/gemma-4-12b-nvfp4.md no longer contains the admission that the content-correctness checks 'were not independently re-run against the base checkpoint specifically', because they were re-run and recorded." + }, + { + "id": "c18", + "kind": "before_state", + "text": "NEITHER surface is authoritative, and they are wrong in OPPOSITE directions. For the generate roles the gateway JSON is wrong (endpoint = its internal :8000) and the CLI is right (:8001 from .env VLLM_PORT). For the audio roles the CLI is wrong (ready=true, loaded=true, endpoint=:8001/v1/audio/speech \u2014 read from .env AUDIO_URL) and the gateway is right (ready=false, loaded=false, endpoint=''). Dialing the CLI-advertised tts path returns 404 'audio endpoints are not configured on this deployment'. Root cause: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml:144, so whether stt/tts work depends on whether the operator remembered the overlay -f flag \u2014 while the CLI reads the merged .env and assumes wired. This is #92's exact shape (ready=true on a 404 path) for the audio roles, it is present in main and in the packaged template, and no issue tracks it." + }, + { + "id": "h19", + "kind": "honesty", + "text": "A test asserts that 'lobes capabilities' and GET /capabilities agree on ready/loaded/endpoint for ALL SIX roles \u2014 including stt/tts on a deployment where the audio overlay is not composed in \u2014 and that the audio roles never report ready=true while their advertised path returns 404." + }, + { + "id": "c19", + "kind": "before_state", + "text": "GET /v1/models advertises phantom backends. _optional_backend wires a backend when *_BASE_URL OR *_SERVED_NAME is set, and falls back to a default_url naming a container that need not exist. On the live rig that yields 6 advertised models against 4 running vLLM containers: 'multimodal-coder' (http://vllm-multimodal-coder:8000) and 'middle' (http://vllm-middle:8000) have no container at all, yet they are listed to clients AND sit in the generate failover chain, which for the cortex model is [primary, multimodal, multimodal-coder, middle]." + }, + { + "id": "h24", + "kind": "honesty", + "text": "With MULTIMODAL_CODER_SERVED_NAME and MIDDLE_SERVED_NAME set but no BASE_URL and no container, build_config wires NEITHER backend, GET /v1/models returns exactly the four running ids, and resolve_model('nvidia/Qwen3-14B-NVFP4') does not resolve to the primary's weights." + }, + { + "id": "c22", + "kind": "requirement", + "text": "A LOCAL live-test of the capabilities contract runs before every PR. It is not a CI job (CI has no GPU and no fleet); it is a local gate the developer runs against the running deployment, in the same live-gated seam tests/test_smoke_duo.py already uses (LOBES_SMOKE_BASE_URL). It must fail loudly rather than skip silently when the operator intended to run it, so that 'advertised implies reachable' is checked against the DEPLOYED artifact before a PR claims to have fixed it \u2014 the exact step whose absence let #87 regress into #92." + }, + { + "id": "h17", + "kind": "honesty", + "text": "The local pre-PR live test, pointed at a deployment whose gateway is stale or whose audio overlay is unwired, FAILS \u2014 it does not skip, and it does not pass." + }, + { + "id": "h18", + "kind": "honesty", + "text": "The gate is ONE trigger and then unattended: a single local command runs the whole capabilities live-test to a pass/fail verdict with no prompts, no manual steps, and no per-role babysitting \u2014 the developer types it, walks away, and reads the exit code." + }, + { + "id": "c23", + "kind": "requirement", + "text": "NO cross-backend failover. Every request resolves to exactly ONE backend and is attempted only there. An explicit model id goes to its owner; a capability tier alias resolves once, up front, via the existing static tier_aliases upward fallback (a tier whose gear is not WIRED still maps to a higher rung at table-build time). There is no runtime retry against a second backend, so a caller who asked for cortex can never silently receive a Gemma answer. A dead/unreachable/warming owner yields 503 + Retry-After. order_backends collapses to a single-element list and rewrite_model stays outside any loop." + }, + { + "id": "h21", + "kind": "honesty", + "text": "A unit test with a fake fleet proves that a request naming the cortex model, with the primary backend dead, NEVER opens a connection to the multimodal backend, and returns 503 + Retry-After. order_backends(table, served) returns exactly one backend for every input." + }, + { + "id": "c27", + "kind": "requirement", + "text": "The pre-PR gate detects DEPLOYED-ARTIFACT skew, not just source correctness: it compares the running gateway container's lobes.__version__ against the CLI's and fails on mismatch, and it dials every advertised role endpoint+path. This is the check whose absence let #87's fix ship in 0.38.0 while the rig kept running 0.36.0 \u2014 making #92 look like a code regression when the code was already correct and merely undeployed." + }, + { + "id": "h22", + "kind": "honesty", + "text": "Run against the rig as it stands right now (gateway 0.36.0, CLI 0.39.0, endpoint :8000, /v1/audio/speech 404ing), the gate exits non-zero and names all three faults; after redeploy it exits zero." + }, + { + "id": "c29", + "kind": "requirement", + "text": "The gateway resolves its advertised origin as: an explicit operator override (GATEWAY_PUBLIC_URL \u2014 for a tunnel or a Host-rewriting reverse proxy) FIRST; else the origin the client actually dialed, echoed from the request Host header; else NOTHING \u2014 an empty endpoint. It must never fabricate an absolute URL from its internal listen port (GATEWAY_PORT), and it must never default GATEWAY_PUBLIC_URL to a localhost URL, because a defaulted public_url outranks the Host header and would tell every LAN/tunnel client to dial its own loopback \u2014 reintroducing the #92 defect in a new place. Each caller therefore receives an origin correct for itself. The base fleet compose still passes AUDIO_URL (empty by default; the audio overlay supplies the real value), which is the #96 half of this change." + }, + { + "id": "h25", + "kind": "honesty", + "text": "With GATEWAY_PUBLIC_URL unset and no Host header, the gateway emits an empty endpoint \u2014 never an absolute URL built from GATEWAY_PORT. With GATEWAY_PUBLIC_URL unset and Host='spark.local:8001', it advertises http://spark.local:8001, not http://localhost:8001. With GATEWAY_PUBLIC_URL set, it wins over any Host header." + } + ], + "tasks": [ + { + "id": "t1", + "summary": "Backend wiring gate: _optional_backend wires a backend ONLY when its *_BASE_URL is set (drop the 'or *_SERVED_NAME' clause), so no phantom backend is ever invented from a default_url", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "With MULTIMODAL_CODER_SERVED_NAME and MIDDLE_SERVED_NAME set but their *_BASE_URL empty and no container, build_config wires NEITHER backend", + "resolve_model('nvidia/Qwen3-14B-NVFP4') on such a deployment does not resolve to the primary's weights", + "The packaged fleet env.example (which sets both vars when a profile is on) is unaffected: enabling COMPOSE_PROFILES=middle still wires the middle backend", + "Files touched: lobes/gateway/_config.py, tests/test_gateway_config_wiring.py (new)" + ], + "deps": [], + "covers": [ + "c19", + "h24" + ] + }, + { + "id": "t2", + "summary": "No cross-backend failover: order_backends returns exactly ONE backend for every input, and the test that currently asserts cross-model failover is inverted", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "order_backends(table, served) returns a list of length <= 1 for every input, including tier-alias-resolved names", + "tests/test_gateway_routing.py::test_order_backends_generate_still_failovers_between_generate_backends is INVERTED to assert the new contract, not deleted", + "The static tier_aliases upward fallback (an unwired tier maps to a higher rung at table-build time) is preserved and still tested", + "Files touched: lobes/gateway/_routing.py, tests/test_gateway_routing.py" + ], + "deps": [], + "covers": [ + "c23", + "h21" + ] + }, + { + "id": "t3", + "summary": "Backend readiness cache: a bounded BACKGROUND probe of each backend's /health, mirroring PressureCache, exposing a cached tri-state per backend and never probing on the request hot path", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A new lobes/gateway/_readiness.py exposes a cache with .current() returning per-backend readiness, refreshed off the request path on an interval", + "A unit test proves .current() opens no socket (injected probe callable, call count asserted zero across N reads)", + "The probe degrades to 'unknown' (never raises) on OSError, http.client.HTTPException and ValueError \u2014 the malformed-URL lesson from PR #90", + "The background thread is a daemon and stops cleanly on server shutdown", + "Files touched: lobes/gateway/_readiness.py (new), tests/test_gateway_readiness.py (new)" + ], + "deps": [], + "covers": [ + "c8", + "h8" + ] + }, + { + "id": "t4", + "summary": "Fleet template truth: inject GATEWAY_PUBLIC_URL (derived from the published VLLM_PORT) and AUDIO_URL into the gateway container from the BASE fleet compose, so the advertised origin is configured truth and stt/tts stop advertising a 404 path (issue #96)", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "lobes/templates/fleet/docker-compose.yml gateway service passes GATEWAY_PUBLIC_URL defaulted from VLLM_PORT, and AUDIO_URL, without requiring the -f docker-compose.audio.yml overlay", + "A test parses the packaged fleet compose and asserts the gateway environment contains both keys", + "env.example documents GATEWAY_PUBLIC_URL as the tunnel/proxy override and notes it defaults to the published port", + "An audio-less deployment still yields audio_url unset in the gateway (AUDIO_URL empty), so stt/tts report loaded=false rather than a 404ing ready=true", + "Files touched: lobes/templates/fleet/docker-compose.yml, lobes/templates/fleet/env.example, tests/test_fleet_template_gateway_env.py (new)" + ], + "deps": [], + "covers": [ + "c11", + "c18", + "c29" + ] + }, + { + "id": "t5", + "summary": "roles.py: RoleInfo.ready stops being an alias of loaded for the four gateway-fronted roles, and the endpoint is never built from the gateway's internal listen port", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "build_role_registry accepts a per-backend readiness signal; ready reflects it while loaded stays the config fact \u2014 the stt/tts separation from #89, generalised to cortex/senses/embedder/reranker", + "_gateway_base_url no longer returns an absolute URL when only an internal listen port is known: with no gateway_url and no public_url the endpoint is empty, never http://localhost:", + "An unconfigured/unready role is still returned (never omitted, never raises), matching the existing six-roles-always-present contract", + "Files touched: lobes/roles.py, tests/test_roles.py" + ], + "deps": [ + "t3" + ], + "covers": [ + "c15", + "c3" + ] + }, + { + "id": "t6", + "summary": "Gateway core: dead owner yields a retryable 503, the readiness cache feeds /v1/models + /capabilities.ready, and reachable_origin prefers configured truth over Host-header inference", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "handle_post: an owner that refuses, times out, or returns >=500 yields HTTP 503 with a Retry-After header and an OpenAI-shaped error whose type (e.g. backend_unavailable) is distinguishable from both 'model unknown' and the all-backends-down 502", + "A fake-fleet test proves a request naming the cortex model with the primary dead NEVER opens a connection to the multimodal backend (upstream-opener call sites asserted)", + "Race test: a fake fleet lists model M, the owner is then killed, and a completion naming M returns 503 + Retry-After \u2014 never 404", + "Converse test: an id that was never in /v1/models is not silently served by the default backend under a different model's weights", + "GET /v1/models is filtered by the cached readiness signal; GET /capabilities .ready reflects it for all six roles", + "reachable_origin(None, None) never fabricates an absolute URL from GATEWAY_PORT; GATEWAY_PUBLIC_URL > Host header > empty", + "The POST hot path opens no probe connection (asserted by call count against an injected opener)", + "Files touched: lobes/gateway/server.py, tests/test_gateway_server.py, tests/test_gateway_capabilities.py" + ], + "deps": [ + "t1", + "t2", + "t3", + "t5" + ], + "covers": [ + "c4", + "h4", + "c5", + "h5", + "c12", + "h23", + "c14", + "h13", + "h10", + "h14", + "h19", + "c29", + "h25" + ] + }, + { + "id": "t7", + "summary": "CLI truth: lobes capabilities agrees with GET /capabilities byte-for-byte, and lobes doctor detects deployed-gateway version skew", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A test asserts GET /capabilities and 'lobes capabilities --json' return identical endpoint/ready/loaded for all six roles on the same deployment config", + "The CLI never reports stt/tts ready=true purely because AUDIO_URL is a string in .env", + "lobes doctor gains a check comparing the running gateway container's lobes.__version__ against the CLI wheel's, failing with severity=error on mismatch and remediation naming the rebuild command", + "Run against the rig as it stands (gateway 0.36.0, CLI 0.39.0) the version-skew check FAILS; after a rebuild it passes", + "Files touched: lobes/cli/_commands/capabilities.py, lobes/cli/_commands/doctor.py, tests/test_cli_capabilities.py, tests/test_doctor.py" + ], + "deps": [ + "t5", + "t6" + ], + "covers": [ + "h3", + "c27", + "h22" + ] + }, + { + "id": "t8", + "summary": "The senses perception probe: prove coolthor actually PERCEIVES, not merely that the wire accepts content-parts \u2014 ground-truth image and ground-truth audio", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "The image probe generates a solid-colour PNG in-process and asserts the model NAMES that colour (red -> 'red', blue -> 'blue'), not merely that content is non-empty", + "The audio probe synthesizes a known word via the rig's own /v1/audio/speech and asserts the transcription contains it \u2014 this only passes once t4 wires AUDIO_URL (issue #96)", + "The existing 1x1-placeholder assertions are replaced or explicitly relabelled as wire-liveness checks, so 'image+text confirmed' never again means 'HTTP 200 with non-empty content'", + "Files touched: tests/test_smoke_duo.py" + ], + "deps": [ + "t4", + "t6" + ], + "covers": [ + "c7", + "h7", + "c17" + ] + }, + { + "id": "t9", + "summary": "The pre-PR live gate: ONE local command, unattended, that dials every advertised role endpoint+path and the deployed gateway's version, and FAILS rather than skips", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "A single command runs the whole capabilities live-test to a pass/fail exit code with no prompts and no manual steps", + "When the operator asked for the live gate (the env var / flag is set) an unreachable deployment FAILS the run; it never degrades to pytest-skip", + "For every role in GET /capabilities the gate dials endpoint+path and asserts a non-404 status; for every id in GET /v1/models it asserts a completion never returns 404", + "It reproduces Colleague's discovery path: given ONLY the gateway origin and no COLLEAGUE_*_BASE_URL override, resolve cortex and senses from the contract and get an answer", + "It fails on deployed-gateway version skew", + "Run against today's rig it exits non-zero naming the :8000 endpoint, the 404ing audio path, and the 0.36.0-vs-0.39.0 skew; after redeploy it exits zero", + "The repo's pre-PR convention (CLAUDE.md / the run-tests skill) names this command so it is not silently never run", + "Files touched: tests/test_live_capabilities.py (new), scripts or Makefile target for the single trigger, CLAUDE.md" + ], + "deps": [ + "t4", + "t6", + "t7" + ], + "covers": [ + "c1", + "h1", + "c2", + "h2", + "c6", + "h6", + "c10", + "h9", + "c22", + "h17", + "h18" + ] + }, + { + "id": "t10", + "summary": "Documentary repairs: record the perception evidence, retire the DSpark route, and stop README claiming lobes init is single-model", + "origin": "llm", + "status": "confirmed", + "acceptance_criteria": [ + "docs/gemma-4-12b-nvfp4.md no longer contains the line-279 admission that content-correctness checks were 'not independently re-run against the base checkpoint specifically' \u2014 because they were, and the coolthor results are recorded there", + "docs/gemma4-mtp-draft.md carries a superseded banner: DSpark (deepseek-ai/dspark_gemma4_12b_block7) does NOT load on vLLM 0.23 (Gemma4DSparkModel unsupported), per #75 \u2014 it must no longer read as 'the ONE route task t3 should wire next'", + "README.md quickstart no longer documents 'lobes init' as scaffolding the single-model deployment on :8000; the duo is the default and --single opts out", + "Files touched: docs/gemma-4-12b-nvfp4.md, docs/gemma4-mtp-draft.md, README.md" + ], + "deps": [ + "t9", + "t8" + ], + "covers": [ + "h16" + ] + } + ], + "risks": [ + { + "id": "r1", + "text": "Requiring *_BASE_URL to wire a backend could break a hand-edited .env that set only *_SERVED_NAME. The packaged fleet template always sets both when a profile is enabled, but an operator's local file may not \u2014 needs a release note and possibly a warning path.", + "kind": "unknown_nonblocking", + "task_id": "t1" + }, + { + "id": "r2", + "text": "The background readiness thread lives inside a stdlib ThreadingHTTPServer. Daemon-thread lifecycle, clean shutdown, and behaviour under 'docker compose down' need care \u2014 a probe thread that outlives the server or blocks shutdown is a regression.", + "kind": "unknown_nonblocking", + "task_id": "t3" + }, + { + "id": "r3", + "text": "The audio perception probe depends on Chatterbox TTS, which has a recorded history of a poisoned CUDA context (500s cleared only by restarting the container). The probe may need a readiness precondition or a retry, or it will flake for reasons unrelated to senses.", + "kind": "unknown_nonblocking", + "task_id": "t9" + }, + { + "id": "r4", + "text": "The live gate cannot run in CI (no GPU, no fleet), so nothing structurally forces it to run. If it is only a convention it will be skipped exactly when it matters \u2014 which is precisely how #87's fix shipped in 0.38.0 while the rig ran 0.36.0. Consider a pre-push hook or a PR-template checkbox.", + "kind": "follow_up", + "task_id": "t8" + } + ] +} diff --git a/.eidetic/memory/lobes__public.jsonl b/.eidetic/memory/lobes__public.jsonl index b140e1a..f0137a9 100644 --- a/.eidetic/memory/lobes__public.jsonl +++ b/.eidetic/memory/lobes__public.jsonl @@ -1,18 +1,18 @@ {"id": "qwen3.5-4b-verified-facts", "hash": "3f61d119c727d0f228e4d2e9dd8dfd71690bf0d526c8511dca023eb553c7d64f", "content": "Qwen/Qwen3.5-4B verified from HF config.json (2026-06-26): architecture is HYBRID linear-attention (Gated Delta/SSM) + full-attention (32 layers: 24 linear + 8 full), NOT MoE and NOT plain dense -> no vLLM --moe-backend. It is MULTIMODAL (full ViT, image+video tokens) so serve text-only via --language-model-only. Ships a BUILT-IN MTP draft head (mtp_num_hidden_layers=1) -> native speculative decoding, no separate draft checkpoint. 256K native (max_position_embeddings 262144), mRoPE. dtype bf16, public, Apache-2.0. Tool-call format = qwen3_coder (XML ), NOT hermes. No official nvidia/RedHatAI NVFP4 for the 4B; community cosmicproc/Qwen3.5-4B-NVFP4 exists (NVIDIA ModelOpt NVFP4 W4A4 -> vLLM --quantization modelopt_fp4; conv1d + lm_head left bf16; needs Blackwell for W4A4 kernels).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "hf-config-json", "topic": "qwen3.5-4b", "relates_to": "lobes-catalog", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.173778+00:00", "last_recall": "2026-07-03T08:03:33.189858+00:00", "recall_count": 7, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "minor-lobe-spec-issue-64", "hash": "8b97d0eb6db4a548000aa5e6d06068f6173bc562f16c2f8693f0a4ec028da4bb", "content": "lobes issue #64 ('minor lobe') was specced via /think on 2026-06-26 -> docs/specs/2026-06-26-lobes-ships-a-minor-lobe-a-cheap-warm-co-resident.md. Scope (user-confirmed) is the FULL minor-role runtime, not just a catalog entry: add Qwen3.5-4B as catalog gear role_hint=minor, served bf16 WARM CO-RESIDENT behind the gateway, + new read-only verbs lobes run/route/eval with escalation + confidence governance. Key decisions: served checkpoint = bf16 Qwen/Qwen3.5-4B (chosen because unsloth LoRA needs the bf16 base; NVFP4 cosmicproc is doc-only untested); route v1 routes ONLY across lobes GEARS (minor vs primary vs candidate) + escalate, not tools/mesh agents; run/route/eval call the model via the gateway OpenAI endpoint reusing assess.py's stdlib urllib client. Deferred (not yet): actual fine-tuning/lobes train, eval-suite contents, MTP spec-decode for minor. Two code gotchas it requires: (1) runtime/_parser.py infer_parser returns 'hermes' for a qwen3.5 id but the model needs 'qwen3_coder' -> must extend _RULES (catalog test asserts tool_parser==infer_parser(id)); (2) catalog requires non-empty quantization for generate gears, so a bf16 gear needs a quantization-field convention (sentinel that switch/compose translate to omitting --quantization).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "minor-lobe", "relates_to": "issue-64", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.175114+00:00", "last_recall": "2026-07-03T23:37:43.106121+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "minor-lobe-spec-issue-64", "hash": "8b97d0eb6db4a548000aa5e6d06068f6173bc562f16c2f8693f0a4ec028da4bb", "content": "lobes issue #64 ('minor lobe') was specced via /think on 2026-06-26 -> docs/specs/2026-06-26-lobes-ships-a-minor-lobe-a-cheap-warm-co-resident.md. Scope (user-confirmed) is the FULL minor-role runtime, not just a catalog entry: add Qwen3.5-4B as catalog gear role_hint=minor, served bf16 WARM CO-RESIDENT behind the gateway, + new read-only verbs lobes run/route/eval with escalation + confidence governance. Key decisions: served checkpoint = bf16 Qwen/Qwen3.5-4B (chosen because unsloth LoRA needs the bf16 base; NVFP4 cosmicproc is doc-only untested); route v1 routes ONLY across lobes GEARS (minor vs primary vs candidate) + escalate, not tools/mesh agents; run/route/eval call the model via the gateway OpenAI endpoint reusing assess.py's stdlib urllib client. Deferred (not yet): actual fine-tuning/lobes train, eval-suite contents, MTP spec-decode for minor. Two code gotchas it requires: (1) runtime/_parser.py infer_parser returns 'hermes' for a qwen3.5 id but the model needs 'qwen3_coder' -> must extend _RULES (catalog test asserts tool_parser==infer_parser(id)); (2) catalog requires non-empty quantization for generate gears, so a bf16 gear needs a quantization-field convention (sentinel that switch/compose translate to omitting --quantization).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "minor-lobe", "relates_to": "issue-64", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.175114+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "devague-blocking-hard-question-gotcha", "hash": "0b8151753dab00bb4c64218896d4cf470b2bb08e3c1a13b5ad000f6c89e13723", "content": "devague (the /think CLI) gotcha: a BLOCKING hard question (interrogate --hard-question --blocking) is NOT cleared by rejecting the claim it hangs on. convergence._missing_open_uncertainty iterates ALL frame.claims (including rejected) for q.blocking and not q.resolved. There is no CLI verb to resolve a hard question (interrogate has no --resolve; confirm/reject only take c*/h* ids; question --resolve is for the separate 'question' move's items). Workaround: edit the local working-state JSON at .devague/frames/.json and set the hard_question's resolved=true (honest when the question is actually answered). Also: the convergence gate requires a confirmed honesty condition on EVERY confirmed spec-affecting claim (audience/before_state/why_it_matters/after_state/boundary/success_signal/requirement/announcement), not just requirements; decisions/assumptions/non_goals are exempt.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "devague", "relates_to": "think-skill", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.175320+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "qwen3.5-vllm-0.19-fla-gibberish-gb10", "hash": "d038d33600580542974ea8dc412cce46f97641e2f6a77edc77f9633aa1c492a2", "content": "Serving Qwen3.5 (Qwen3_5ForConditionalGeneration; Gated DeltaNet linear-attention hybrid) on the lobes GB10 fleet's vLLM image nvcr.io/nvidia/vllm:26.04-py3 (engine 0.19.0): the architecture IS registered (verified via ModelRegistry: Qwen3_5ForConditionalGeneration + Qwen3_5MTP present) so it LOADS, BUT vLLM 0.19.0 carries a known FLA/Gated-DeltaNet tensor-format bug on Blackwell (sm_120/sm_121) that produces GIBBERISH output. The fix (vLLM PR #43961) shipped only in v0.23.0. So a live Qwen3.5-4B serve on the current production image likely emits garbled text -> NOT a meaningful live test. Also: V2 model runner crashes on the mixed-attention KV cache (workaround --enforce-v1); GDN needs --max-num-batched-tokens 2096 for cache alignment; sm_121 needs forward-compat (TORCH_CUDA_ARCH_LIST=12.0). --language-model-only correctly drops the vision tower for inference of the full multimodal checkpoint. A meaningful live 4B co-resident test needs vLLM >= 0.23.0 (newer image). The minor-lobe CODE (issue #64: catalog gear, qwen3_coder parser rule, run/route/eval verbs, governance, opt-in co-resident fleet service) is independent and CI-green. Fleet facts: gateway published on host :8001 (not :8000); deployment dir ~/.model-gear; HF cache /home/spark/.cache/huggingface; image runtime=runc.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "qwen3.5-serving", "relates_to": "minor-lobe", "date": "2026-06-26"}, "created": "2026-06-26T11:11:12.839972+00:00", "last_recall": "2026-06-30T16:08:24.560923+00:00", "recall_count": 3, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "qwen3.5-vllm-0.19-fla-gibberish-gb10", "hash": "d038d33600580542974ea8dc412cce46f97641e2f6a77edc77f9633aa1c492a2", "content": "Serving Qwen3.5 (Qwen3_5ForConditionalGeneration; Gated DeltaNet linear-attention hybrid) on the lobes GB10 fleet's vLLM image nvcr.io/nvidia/vllm:26.04-py3 (engine 0.19.0): the architecture IS registered (verified via ModelRegistry: Qwen3_5ForConditionalGeneration + Qwen3_5MTP present) so it LOADS, BUT vLLM 0.19.0 carries a known FLA/Gated-DeltaNet tensor-format bug on Blackwell (sm_120/sm_121) that produces GIBBERISH output. The fix (vLLM PR #43961) shipped only in v0.23.0. So a live Qwen3.5-4B serve on the current production image likely emits garbled text -> NOT a meaningful live test. Also: V2 model runner crashes on the mixed-attention KV cache (workaround --enforce-v1); GDN needs --max-num-batched-tokens 2096 for cache alignment; sm_121 needs forward-compat (TORCH_CUDA_ARCH_LIST=12.0). --language-model-only correctly drops the vision tower for inference of the full multimodal checkpoint. A meaningful live 4B co-resident test needs vLLM >= 0.23.0 (newer image). The minor-lobe CODE (issue #64: catalog gear, qwen3_coder parser rule, run/route/eval verbs, governance, opt-in co-resident fleet service) is independent and CI-green. Fleet facts: gateway published on host :8001 (not :8000); deployment dir ~/.model-gear; HF cache /home/spark/.cache/huggingface; image runtime=runc.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "qwen3.5-serving", "relates_to": "minor-lobe", "date": "2026-06-26"}, "created": "2026-06-26T11:11:12.839972+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "devague-no-resolve-park", "hash": "42eca84ddc6356f0a0d522b8953439304b28417a757f6ecab5b4f9d674a6a744", "content": "devague (the /think CLI) has NO resolve-park / edit-vagueness move. A park created with --kind unknown_blocking blocks `converge` by design, and there is no CLI verb to resolve it once the user decides the unknown \u2014 re-running `park` with the same text just creates a DUPLICATE vagueness item (v3), it does not update in place. To converge after a blocking unknown is genuinely decided, edit the frame JSON directly at .devague/frames/.json: reclassify the items \"kind\" from \"unknown_blocking\" to \"follow_up\" (or remove it) and record the resolution in its \"text\". Frame files live under .devague/frames/ (NOT .devague/*.json); spec exports land in docs/specs/-.md. Also: a capture with default origin lands as confirmed (treat as user-stated); --origin llm lands as proposed and needs user confirm. Honesty conditions are required on announcement/audience/before_state/after_state/why_it_matters/requirement/success_signal claims but NOT on decision/non_goal/assumption claims.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"area": "devague", "topic": "think-skill-gotcha"}, "created": "2026-06-26T14:54:28.559433+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "minor-lobe-live-spark-fleet", "hash": "e35daba9e905594346fe19cef3be94a89f381915ae79481a3904bdef2c66b7bb", "content": "GO-LIVE (2026-06-26): the minor lobe (Qwen/Qwen3.5-4B) is LIVE on the spark DGX fleet. Deployment dir is ~/.model-gear (legacy; no ~/.lobes/$LOBES_DIR set), compose project name model-gear, composed from BOTH docker-compose.yml + docker-compose.audio.yml (audio services are NOT behind a profile). Did a full re-scaffold to 0.30.0: overwrote docker-compose.yml + Dockerfile.gateway from the repo 0.30.0 templates (gateway now pip-installs lobes-cli==0.30.0, entrypoint lobes.gateway; was 0.26.0/model_gear pre-rename), and patched .env: MODEL_GEAR_VERSION=0.30.0, added MINOR_BASE_URL=http://vllm-minor:8000, MINOR_SERVED_NAME=Qwen/Qwen3.5-4B, COMPOSE_PROFILES=minor. Applied with: docker compose -f docker-compose.yml -f docker-compose.audio.yml --profile minor up -d --build --no-deps gateway vllm-minor (only gateway recreated + vllm-minor started; primary/embed/rerank/audio untouched). GPU utils: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10 = 0.82 of 128GB unified \u2014 fits. Gateway host port 8001; /v1/models now lists Qwen/Qwen3.5-4B. AUTH is NOT in compose \u2014 it is a HOST cloudflared tunnel process (decoupled), so compose re-scaffold does not touch it; just keep gateway on host port 8001. Rollback files: ~/.model-gear/{docker-compose.yml,Dockerfile.gateway,.env}.preminor.bak. GOTCHAS: (1) gateway reads MINOR_BASE_URL (NOT MINOR_URL like the _URL convention for PRIMARY/EMBED/RERANK); empty default => minor silently unwired. (2) chatterbox TTS hit a poisoned CUDA context (torch.AcceleratorError cudaErrorUnknown, HTTP 500) \u2014 cleared by docker compose restart chatterbox; synthesis works fine WITH the 4B co-resident (the 4B did not cause GPU exhaustion).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"area": "deployment", "host": "spark", "topic": "minor-lobe-golive"}, "created": "2026-06-26T15:35:48.071970+00:00", "last_recall": "2026-07-03T23:57:07.981592+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-68-tiered-routing-spec", "hash": "7d1af62c72622a18e11b0becc36982cf148e2c817a55242ebaf5bef1741673b5", "content": "Issue #68 (tiered model resource routing for Spark unified memory) specced via /think on 2026-06-29 -> docs/specs/2026-06-29-lobes-adds-a-third-middle-model-tier-and-pressure.md. User-confirmed decisions: (1) THIRD tier = a 14B-class NVFP4 checkpoint, INFERENCE-ONLY (~8GB weights) -- it is NOT a LoRA base; LoRA training stays on the existing 4B bf16 minor lobe (so 'scale work + train loras' = 3 inference tiers + 4B trainable base). (2) 27B primary served context trimmed 256K->128K (PRIMARY_MAX_MODEL_LEN=131072) to free ~half the KV cache for the co-resident middle gear. (3) Tier-request surface = gateway model-alias ONLY (caller sends model=cheap|normal|hard to the OpenAI endpoint; gateway resolves to 4B/14B/27B); NO 'lobes select --tier' verb, NO 'route --tier' field. (4) FULL pressure policy IS in scope: read-only swap%/iowait% sampling from /proc, #68 thresholds, degraded-mode state machine, model=hard downgrades with reason=pressure, manual override + downgrade reason cross the OpenAI boundary via HTTP headers, read-only 'lobes status --pressure' emits {tier,model,mode,reason,pressure}. Parked unknowns: exact 14B NVFP4 checkpoint id (same-gen Qwen3.6-14B vs dense Qwen3-14B-NVFP4); exact override/reason header names + whether a streaming response can carry the reason. Risk: 14B NVFP4 must load non-gibberish on the Blackwell vLLM image (dense Qwen3 NVFP4 lower-risk than a 3.5/3.6 hybrid w/ the pre-vLLM-0.23 FLA bug). Current fleet baseline: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "tiered-routing", "relates_to": "issue-68", "date": "2026-06-29"}, "created": "2026-06-29T19:44:26.978729+00:00", "last_recall": "2026-07-03T08:03:33.189858+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "minor-lobe-live-spark-fleet", "hash": "e35daba9e905594346fe19cef3be94a89f381915ae79481a3904bdef2c66b7bb", "content": "GO-LIVE (2026-06-26): the minor lobe (Qwen/Qwen3.5-4B) is LIVE on the spark DGX fleet. Deployment dir is ~/.model-gear (legacy; no ~/.lobes/$LOBES_DIR set), compose project name model-gear, composed from BOTH docker-compose.yml + docker-compose.audio.yml (audio services are NOT behind a profile). Did a full re-scaffold to 0.30.0: overwrote docker-compose.yml + Dockerfile.gateway from the repo 0.30.0 templates (gateway now pip-installs lobes-cli==0.30.0, entrypoint lobes.gateway; was 0.26.0/model_gear pre-rename), and patched .env: MODEL_GEAR_VERSION=0.30.0, added MINOR_BASE_URL=http://vllm-minor:8000, MINOR_SERVED_NAME=Qwen/Qwen3.5-4B, COMPOSE_PROFILES=minor. Applied with: docker compose -f docker-compose.yml -f docker-compose.audio.yml --profile minor up -d --build --no-deps gateway vllm-minor (only gateway recreated + vllm-minor started; primary/embed/rerank/audio untouched). GPU utils: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10 = 0.82 of 128GB unified \u2014 fits. Gateway host port 8001; /v1/models now lists Qwen/Qwen3.5-4B. AUTH is NOT in compose \u2014 it is a HOST cloudflared tunnel process (decoupled), so compose re-scaffold does not touch it; just keep gateway on host port 8001. Rollback files: ~/.model-gear/{docker-compose.yml,Dockerfile.gateway,.env}.preminor.bak. GOTCHAS: (1) gateway reads MINOR_BASE_URL (NOT MINOR_URL like the _URL convention for PRIMARY/EMBED/RERANK); empty default => minor silently unwired. (2) chatterbox TTS hit a poisoned CUDA context (torch.AcceleratorError cudaErrorUnknown, HTTP 500) \u2014 cleared by docker compose restart chatterbox; synthesis works fine WITH the 4B co-resident (the 4B did not cause GPU exhaustion).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"area": "deployment", "host": "spark", "topic": "minor-lobe-golive"}, "created": "2026-06-26T15:35:48.071970+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 6, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-68-tiered-routing-spec", "hash": "7d1af62c72622a18e11b0becc36982cf148e2c817a55242ebaf5bef1741673b5", "content": "Issue #68 (tiered model resource routing for Spark unified memory) specced via /think on 2026-06-29 -> docs/specs/2026-06-29-lobes-adds-a-third-middle-model-tier-and-pressure.md. User-confirmed decisions: (1) THIRD tier = a 14B-class NVFP4 checkpoint, INFERENCE-ONLY (~8GB weights) -- it is NOT a LoRA base; LoRA training stays on the existing 4B bf16 minor lobe (so 'scale work + train loras' = 3 inference tiers + 4B trainable base). (2) 27B primary served context trimmed 256K->128K (PRIMARY_MAX_MODEL_LEN=131072) to free ~half the KV cache for the co-resident middle gear. (3) Tier-request surface = gateway model-alias ONLY (caller sends model=cheap|normal|hard to the OpenAI endpoint; gateway resolves to 4B/14B/27B); NO 'lobes select --tier' verb, NO 'route --tier' field. (4) FULL pressure policy IS in scope: read-only swap%/iowait% sampling from /proc, #68 thresholds, degraded-mode state machine, model=hard downgrades with reason=pressure, manual override + downgrade reason cross the OpenAI boundary via HTTP headers, read-only 'lobes status --pressure' emits {tier,model,mode,reason,pressure}. Parked unknowns: exact 14B NVFP4 checkpoint id (same-gen Qwen3.6-14B vs dense Qwen3-14B-NVFP4); exact override/reason header names + whether a streaming response can carry the reason. Risk: 14B NVFP4 must load non-gibberish on the Blackwell vLLM image (dense Qwen3 NVFP4 lower-risk than a 3.5/3.6 hybrid w/ the pre-vLLM-0.23 FLA bug). Current fleet baseline: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "tiered-routing", "relates_to": "issue-68", "date": "2026-06-29"}, "created": "2026-06-29T19:44:26.978729+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "issue-69-gemma-duo-spec", "hash": "f0fc6928c15f0438a22ae71f7a4f8db22ed8831854c41a361db6f900b8f45b04", "content": "Issue #69 (make Spark default-serve the Qwen3.6-27B + Gemma4-12B duo) specced via /think 2026-06-30 -> docs/specs/2026-06-30-lobes-now-defaults-to-a-spark-duo-the-qwen3-6-27b.md (frame .devague/frames/lobes-now-defaults-to-a-spark-duo-the-qwen3-6-27b.json). USER-confirmed decisions: (1) 'lobes serve' STOPS being single-model -- with no flags it brings up BOTH main (27B MTP primary) + multimodal (Gemma4-12B); minor/14B/legacy need explicit profiles. (2) NEW tier vocabulary main/minor/multimodal (user reframed away from cheap/normal/hard) -- Gemma gets NEW role_hint='multimodal' in catalog.py; old 'middle' role + 'normal' alias DEPRECATED (normal->multimodal back-compat kept); nvidia/Qwen3-14B-NVFP4 demoted role_hint=middle->candidate (KEPT, cite-don't-delete, legacy profile only). (3) Default Gemma serves NVFP4 + NATIVE MTP ON; DeepSeek DSpark draft (deepseek-ai/dspark_gemma4_12b_block7, shipped 2026-06-27 via DeepSpec) = disabled-by-default experiment toggle. (4) Gemma serves FULL multimodal: vision + AUDIO (Gemma4Unified ingests image/video/audio; vLLM supports it; Gemma4-12B is first medium model with native audio-in/ASR) -- chat audio-in is DISTINCT from the /v1/audio/* Parakeet(STT)/Chatterbox(TTS) overlay (unchanged per boundary). Leading checkpoint candidate: sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 (SAME publisher as the lobes primary, NVFP4+MTP). Accepted plan risks (parked unknown_nonblocking): exact checkpoint pick; Gemma4 --tool-call-parser + runtime/_parser.py infer_parser rule (catalog test asserts tool_parser==infer_parser); loads-on-nv26.04-image (LOWER risk: Gemma4UnifiedForConditionalGeneration registered, auto-detects NVFP4, NOT the Qwen3.5 Gated-DeltaNet/FLA arch that gibbered on sm_120/121); native-MTP vs DSpark --speculative-config JSON; measured GPU util for multimodal Gemma (vision+audio embedders+KV vs 14B's 0.12). DESIGN SEAM for spec-to-plan: main/minor/multimodal breaks the linear cheap/normal/hard pressure ladder (issue #68) -- 'multimodal' is a different capability, not a capability rung, so the pressure-downgrade target needs redefining. TOOLING GOTCHA: devague spec_md exporter (render/spec_md.py:66 _follow_up) DROPS unknown_nonblocking vagueness from the exported spec.md (only renders follow_up/out_of_scope) -- had to append an 'Accepted plan risks' section by hand; the frame JSON retains all parked items, so /spec-to-plan (which reads the frame) still gets them. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "gemma-duo", "relates_to": "issue-69", "date": "2026-06-30"}, "created": "2026-06-30T12:37:29.031340+00:00", "last_recall": "2026-07-03T23:57:07.981592+00:00", "recall_count": 8, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "gemma4-unified-unsupported-released-vllm-images", "hash": "34c5ad26dedfe96cb163a6e3e4ad0348a6501f0f8d0be1450f24ebba3fa24840", "content": "t7 live validation (2026-06-30, DGX Spark) for the lobes Gemma 4 12B multimodal gear (issue #69): Gemma 4 12B's architecture is model_type=`gemma4_unified`, and ALL community NVFP4 12B checkpoints use it (sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4, AxionML/Gemma-4-12B-NVFP4, coolthor/gemma-4-12B-it-NVFP4A16). NEITHER released NGC vLLM image registers gemma4_unified: nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0, transformers 4.57.6) NOR nvcr.io/nvidia/vllm:26.05.post1-py3 (vLLM 0.21.0, transformers 5.6.0). vLLM crashes at config load: 'model type gemma4_unified but Transformers does not recognize this architecture ... install Transformers from source'. NOTE: 26.05/vLLM 0.21.0 DOES register Gemma4MTPModel + standard Gemma4ForConditionalGeneration (just not the Unified variant) \u2014 so a standard-arch Gemma4 would load, but the 12B unified multimodal needs nightly/source transformers. Fix path tracked in issue #71 (custom image: 26.05 base + `pip install --pre -U transformers` or git source, verify gemma4_unified registers, check vLLM 0.21.0 pin compat). The lobes catalog gear stays status=configured (correct) until a supporting image lands. Co-resident test approach = zero fleet disruption (run the multimodal container on a free host port alongside the running fleet; primary at util 0.51, +0.12 multimodal = 0.85 < 1.0, no primary bounce). Shipped in PR #72. The vLLM image's --gpus is DeviceRequests nvidia:-1 (=--gpus all), runtime=runc, HF cache /home/spark/.cache/huggingface, compose net model-gear_default, gateway host :8001, deploy dir ~/.model-gear.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-69", "date": "2026-06-30"}, "created": "2026-06-30T15:40:49.826739+00:00", "last_recall": "2026-07-03T23:26:39.010002+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "gemma4-unified-unsupported-released-vllm-images", "hash": "34c5ad26dedfe96cb163a6e3e4ad0348a6501f0f8d0be1450f24ebba3fa24840", "content": "t7 live validation (2026-06-30, DGX Spark) for the lobes Gemma 4 12B multimodal gear (issue #69): Gemma 4 12B's architecture is model_type=`gemma4_unified`, and ALL community NVFP4 12B checkpoints use it (sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4, AxionML/Gemma-4-12B-NVFP4, coolthor/gemma-4-12B-it-NVFP4A16). NEITHER released NGC vLLM image registers gemma4_unified: nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0, transformers 4.57.6) NOR nvcr.io/nvidia/vllm:26.05.post1-py3 (vLLM 0.21.0, transformers 5.6.0). vLLM crashes at config load: 'model type gemma4_unified but Transformers does not recognize this architecture ... install Transformers from source'. NOTE: 26.05/vLLM 0.21.0 DOES register Gemma4MTPModel + standard Gemma4ForConditionalGeneration (just not the Unified variant) \u2014 so a standard-arch Gemma4 would load, but the 12B unified multimodal needs nightly/source transformers. Fix path tracked in issue #71 (custom image: 26.05 base + `pip install --pre -U transformers` or git source, verify gemma4_unified registers, check vLLM 0.21.0 pin compat). The lobes catalog gear stays status=configured (correct) until a supporting image lands. Co-resident test approach = zero fleet disruption (run the multimodal container on a free host port alongside the running fleet; primary at util 0.51, +0.12 multimodal = 0.85 < 1.0, no primary bounce). Shipped in PR #72. The vLLM image's --gpus is DeviceRequests nvidia:-1 (=--gpus all), runtime=runc, HF cache /home/spark/.cache/huggingface, compose net model-gear_default, gateway host :8001, deploy dir ~/.model-gear.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-69", "date": "2026-06-30"}, "created": "2026-06-30T15:40:49.826739+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 7, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "issue-71-gemma4-custom-image-spec", "hash": "167dd266f292a73237b77a2db62529cd14cb3664f363993eb7e34c3d4d1ad09e", "content": "Issue #71 (custom vLLM image for the Gemma 4 12B gemma4_unified multimodal gear) specced via /think 2026-06-30 -> docs/specs/2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.md. USER-confirmed decisions: (1) SCOPE: only vllm-multimodal moves to the custom 26.05-based image NOW; primary/embed/rerank stay on nvcr.io/nvidia/vllm:26.04-py3 (load-tested 27B primary not disturbed); if multimodal works well, open 3 follow-up issues to migrate primary/embed/rerank one-by-one. (2) DELIVERY: local compose build: directive on vllm-multimodal (matches gateway/chatterbox/parakeet/realtime Dockerfile pattern) is the DEFAULT; PLUS an optional MULTIMODAL_IMAGE env override so an operator can point at a ghcr.io/agentculture or local registry:2 tag (registry push feasible but not required). (3) TOOLING: Dockerfile.vllm-gemma4 FROM nvcr.io/nvidia/vllm:26.05.post1-py3 installs transformers with UV (uv pip install --system), NOT pip (user preference). (4) PIN: bake a PINNED transformers ref (version or git SHA) for reproducible rebuilds; exact ref discovered during validation (Q3 left to my default). (5) DELIVERABLE: validate+promote in ONE go on the Spark (spark-f8a9 IS the GB10 Spark, fleet live on 26.04, no vllm-multimodal up) -> the recipe PR merges regardless; status flips configured->load-tested + numbers in docs/gemma-4-12b-nvfp4.md ONLY if all pass (boots, image+text, audio+text, MTP accept>0, util in 0.69 budget); any fail -> stays configured, residual parked. Co-resident validation = zero fleet disruption (free host port alongside running fleet). Parked unknowns v1-v4: exact transformers ref; correct gemma4_mtp method string on vLLM 0.21.0 (r4); measured vision+audio util vs 0.12 (r5); native context (131072 default). Core risk: NO transformers ref may both register gemma4_unified AND keep vLLM 0.21.0 importing -> then await NGC release. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-custom-image", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T16:19:55.749207+00:00", "last_recall": "2026-07-03T23:54:01.568905+00:00", "recall_count": 3, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "gemma4-unified-transformers-ref-resolved-r1", "hash": "7490113c6117e54c0ae4c8a4cec0d0accbd66d0cc4ff212ec58b4f893fdb875a", "content": "RESOLVED (2026-06-30, DGX Spark spark-f8a9, issue #71): the custom vLLM image for the Gemma 4 12B gemma4_unified gear BUILDS and the core risk r1 clears. WORKING RECIPE: FROM nvcr.io/nvidia/vllm:26.05.post1-py3 (vLLM 0.21.0+2325b6f0) + transformers from source pinned to commit 181beb3ba4c47098ed8cbc97ee250d1d45ae0107 (resolves to transformers==5.13.0.dev0). This ref BOTH registers gemma4_unified in transformers AutoConfig CONFIG_MAPPING AND leaves vLLM 0.21.0 importing/serving (the r1 question: does ANY ref do both -> YES). Verified in-image: AutoConfig.from_pretrained(sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4, trust_remote_code=True) -> model_type gemma4_unified; vllm.ModelRegistry.get_supported_archs() shows [Gemma4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4MTPModel] (Gemma4MTPModel present -> good for native MTP / r4); vllm serve --help exits 0 WITH --gpus (without GPU it errors RuntimeError: Failed to infer device type -- a no-GPU artifact, NOT a real break). TWO BUILD GOTCHAS on this base: (1) a multi-line RUN python3 -c \"...\" MUST use trailing-backslash continuations or Docker parses each body line as an instruction (unknown instruction: import); (2) uv pip install --system needs --break-system-packages because the base /usr Python carries a PEP 668 EXTERNALLY-MANAGED marker (the base sets PIP_BREAK_SYSTEM_PACKAGES=1 for pip only, and PIP_CONSTRAINT=/etc/pip/constraint.txt which uv ignores -- fine, we WANT to override the pinned transformers). uv 0.11.17 already ships in the 26.05 base (no bootstrap needed). Image tag built: lobes/vllm-gemma4:local. STILL OPEN (t4): live serve of the actual checkpoint with --speculative-config gemma4_mtp (r4 method string), measured GPU util (r5), native context (v4), image+text + audio+text functional validation.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T16:56:45.301751+00:00", "last_recall": "2026-07-01T05:32:46.663593+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "gemma4-12b-serve-blocked-fp4-marlin-vllm021", "hash": "1070e80076271e9ac88ec328159202179f0ef71bc3daca0704697645a7ad1d4d", "content": "t4 live serve (2026-06-30, DGX Spark, issue #71) of sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 on the custom image (nv26.05.post1 vLLM 0.21.0 + transformers main 181beb3) is BLOCKED by a vLLM kernel bug -- the gear CANNOT serve on this image. Findings, in order hit: (1) QUANT: catalog/compose --quantization modelopt_fp4 is WRONG for this checkpoint; config.json quant_method=compressed-tensors, format nvfp4-pack-quantized (NVFP4A16). Must use --quantization compressed-tensors (or omit -> auto-detect). (2) MTP/r4 NEGATIVE: --speculative-config {method: gemma4_mtp} fails NotImplementedError: Unsupported speculative method mtp. vLLM 0.21.0 speculative.py only enables gemma4 MTP when a SEPARATE DRAFT model has model_type==gemma4_assistant (line 515 hf_config_override; use_gemma4_mtp() needs draft_model_config). There is NO auto-derivation from the gemma4_unified target (unlike deepseek_v3->deepseek_mtp), and THIS checkpoint config has no mtp/assistant/nextn keys despite the -MTP name. So native MTP needs a gemma4_assistant draft repo we do not have. Catalog speculative_config for the gemma gear is wrong -> remove/correct. (3) BLOCKER: with --quantization compressed-tensors + no spec-config, the model LOADS but CRASHES at warmup forward: RuntimeError: Shape mismatch: a.size(1)=4096, size_k=8192 in apply_fp4_marlin_linear (compressed_tensors_w4a16_nvfp4.py apply_weights -> marlin_utils_fp4) on Gemma4 attention o_proj. The CompressedTensorsW4A16Fp4 scheme is MARLIN-ONLY (no cutlass/native alt, no env override; VLLM_NVFP4_GEMM_BACKEND/CT_EMULATIONS dont apply). vLLM 0.21.0s W4A16 NVFP4 Marlin kernel mishandles Gemma4s o_proj shape. RESOLVED at config level: native context=131072 (text_config.max_position_embeddings; v4). NOT measurable: r5 GPU util, functional image/audio (blocked by the crash). Maintenance window: stopped primary(0.51)+minor(0.10) to free ~74GB (user-authorized), embed/rerank untouched, fleet restored after. NEXT OPTIONS: (A) try a different Gemma4-12B checkpoint that avoids W4A16 Marlin -- a W4A4 full-NVFP4 (cutlass path) or a nvidia modelopt_fp4 format (modelopt kernel) checkpoint (AxionML/Gemma-4-12B-NVFP4 fmt unknown; coolthor NVFP4A16 = same W4A16 problem likely). (B) await a vLLM that fixes Marlin W4A16 NVFP4 for gemma4 (or adds a non-marlin W4A16 path). Gear stays status=configured; recipe (image+wiring) + r1 win + quant/spec-config corrections still merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T17:17:56.428563+00:00", "last_recall": "2026-07-03T23:37:43.106121+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "gemma4-serve-root-cause-nonsquare-attn-triton", "hash": "acc43da8b918f91ad0d9a68c3afe107e4a6209635253e269ec1a3314613b2200", "content": "t4 deep-dive (2026-06-30, DGX Spark, issue #71): the Gemma 4 12B serve blocker is PRECISELY the non-square attention, NOT a fundamental wall. config text_config: hidden_size=3840, head_dim=256, global_head_dim=512 (DOUBLE head_dim), num_attention_heads=16, attention_k_eq_v=true. The o_proj weight expects K = num_heads*global_head_dim = 16*512 = 8192, but FLASH_ATTN emits num_heads*head_dim = 16*256 = 4096 -> RuntimeError: Shape mismatch a.size(1)=4096 size_k=8192 in marlin_gemm (o_proj). Fix per ai-muninn.com blog: VLLM_ATTENTION_BACKEND=TRITON_ATTN (Triton handles non-square attn, emits 8192). BUT in my test the env DID NOT ENGAGE: vLLM logged Using FLASH_ATTN out of potential backends [FLASH_ATTN,FLASHINFER,TRITON_ATTN,FLEX_ATTENTION], because gemma4_unified runs via vLLMs Transformers modeling backend (no native vLLM Gemma4Unified impl; native Gemma4ForConditionalGeneration is the non-unified variant) and that path did not honor VLLM_ATTENTION_BACKEND=TRITON_ATTN as a docker -e var. OPEN: how to force TRITON_ATTN on the transformers-backend path (CLI flag? hf attn_implementation override? newer vLLM native unified impl?). Same crash on vLLM 0.21.0 AND 0.22.1. RUNTIME MATRIX TESTED: nv26.05.post1=vLLM0.21.0+torch2.12.0a0nv; nv26.06=vLLM0.22.1+torch2.13.0a0nv (BLOG-PROVEN version) -- both keep NGC torch when transformers overlaid (181beb3); host venv nightly=vLLM0.23.1rc1.dev+torch2.11.0+cu130 (stock torch arch_list sm_80..sm_120 only, sm_121 via fwd-compat, torch loads on GB10 but OOMs unless fleet stopped). NOTE 0.24.x does NOT exist yet (latest nightly 0.23.1rc1). BLOG-PROVEN WORKING RECIPE (untested by us): coolthor/gemma-4-12B-it-NVFP4A16 (7.7GB) + vLLM 0.22.1 + VLLM_ATTENTION_BACKEND=TRITON_ATTN -> image+audio+video all work. Our checkpoint (sakamakismile coder-fable5 variant) may also have a nonstandard quant; coolthor is the safer checkpoint. NEXT: (1) figure out TRITON_ATTN engagement on transformers backend; (2) test coolthor; (3) consider switching catalog default to coolthor.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T17:46:48.319291+00:00", "last_recall": "2026-07-01T05:32:46.663593+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "gemma4-serve-root-cause-nonsquare-attn-triton", "hash": "acc43da8b918f91ad0d9a68c3afe107e4a6209635253e269ec1a3314613b2200", "content": "t4 deep-dive (2026-06-30, DGX Spark, issue #71): the Gemma 4 12B serve blocker is PRECISELY the non-square attention, NOT a fundamental wall. config text_config: hidden_size=3840, head_dim=256, global_head_dim=512 (DOUBLE head_dim), num_attention_heads=16, attention_k_eq_v=true. The o_proj weight expects K = num_heads*global_head_dim = 16*512 = 8192, but FLASH_ATTN emits num_heads*head_dim = 16*256 = 4096 -> RuntimeError: Shape mismatch a.size(1)=4096 size_k=8192 in marlin_gemm (o_proj). Fix per ai-muninn.com blog: VLLM_ATTENTION_BACKEND=TRITON_ATTN (Triton handles non-square attn, emits 8192). BUT in my test the env DID NOT ENGAGE: vLLM logged Using FLASH_ATTN out of potential backends [FLASH_ATTN,FLASHINFER,TRITON_ATTN,FLEX_ATTENTION], because gemma4_unified runs via vLLMs Transformers modeling backend (no native vLLM Gemma4Unified impl; native Gemma4ForConditionalGeneration is the non-unified variant) and that path did not honor VLLM_ATTENTION_BACKEND=TRITON_ATTN as a docker -e var. OPEN: how to force TRITON_ATTN on the transformers-backend path (CLI flag? hf attn_implementation override? newer vLLM native unified impl?). Same crash on vLLM 0.21.0 AND 0.22.1. RUNTIME MATRIX TESTED: nv26.05.post1=vLLM0.21.0+torch2.12.0a0nv; nv26.06=vLLM0.22.1+torch2.13.0a0nv (BLOG-PROVEN version) -- both keep NGC torch when transformers overlaid (181beb3); host venv nightly=vLLM0.23.1rc1.dev+torch2.11.0+cu130 (stock torch arch_list sm_80..sm_120 only, sm_121 via fwd-compat, torch loads on GB10 but OOMs unless fleet stopped). NOTE 0.24.x does NOT exist yet (latest nightly 0.23.1rc1). BLOG-PROVEN WORKING RECIPE (untested by us): coolthor/gemma-4-12B-it-NVFP4A16 (7.7GB) + vLLM 0.22.1 + VLLM_ATTENTION_BACKEND=TRITON_ATTN -> image+audio+video all work. Our checkpoint (sakamakismile coder-fable5 variant) may also have a nonstandard quant; coolthor is the safer checkpoint. NEXT: (1) figure out TRITON_ATTN engagement on transformers backend; (2) test coolthor; (3) consider switching catalog default to coolthor.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T17:46:48.319291+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "issue-75-gemma4-mtp-spec", "hash": "3730e1ff2f26b4d477684710af9197daf4f2f25348579c0ddc834c6e65e8f3d0", "content": "Issue #75 (Gemma4 native MTP: source/build a gemma4_assistant draft for speculative decoding) was specced via /think on 2026-07-01 -> docs/specs/2026-07-01-gemma-4-12b-gear-gets-speculative-decoding-draft.md (frame .devague/frames/the-gemma-4-12b-gear-gets-speculative-decoding-lob.json). USER-confirmed decisions (4 AskUserQuestion answers): (1) SCOPE = any speculative-decoding speedup counts, NOT strictly native gemma4_mtp -> measure the EXISTING DSpark draft_model route (deepseek-ai/dspark_gemma4_12b_block7) FIRST as the cheap path; pursue a native gemma4_assistant draft only if DSpark proves the win is real but insufficient. (2) BUILD APPETITE = NO -- training/distilling a gemma4_assistant draft head is a SEPARATE follow-up, not in #75; #75 stops at sourced-or-DSpark + documented verdict. (3) DONE = a MEASURED VERDICT (restore speculative_config in catalog+compose with recorded acceptance%/speedup beating baseline, OR a documented negative with the numbers that rule it out). (4) SERVE GATING = #75 is BLOCKED on #71 serve-enablement -- does NOT ship draft wiring ahead of serve; only desk-sourcing a candidate draft can start before #71 lands. GROUNDING: gemma4_mtp has NO self-speculation (vLLM 0.21/0.22 need a separate model_type==gemma4_assistant draft; deepseek_v3->deepseek_mtp auto-derivation has no gemma equivalent); {method:gemma4_mtp} is rejected Unsupported speculative method; the served -MTP checkpoint exposes no mtp/assistant/nextn keys. Catalog->compose wiring follows the 27B mtp_compose_command_items() pattern. Hard dependency #71: gear LOADS but does not SERVE (non-square attention global_head_dim 512 != head_dim 256 needs TRITON_ATTN, not honored on vLLM transformers backend -> o_proj GEMM 4096!=8192). EXPORTER GOTCHA confirmed again: devague spec_md drops unknown_nonblocking vagueness (whether a sourceable gemma4_assistant draft EXISTS on HF) -> restored by hand in the spec.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"topic": "gemma4-mtp-spec", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-06-30T22:37:05.301533+00:00", "last_recall": "2026-07-01T18:13:02.194724+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-75-gemma4-mtp-workforce-findings", "hash": "2d373fe4374cd170fa61deb6a15e9435c41300f9d4862b5c494d257410bbaeb8", "content": "Issue #75 workforce run (2026-07-01, /assign-to-workforce, branch spec/issue-75-gemma4-mtp-draft). Wave 0 (t1,t2) built + merged as DOCS; t3 DEFERRED (re-binned behind #71). KEY FINDINGS: (1) t1 web-research RESOLVED the parked unknown: a NATIVE gemma4_assistant-family draft DOES exist for this checkpoint family -- google/gemma-4-12B-it-assistant (model_type gemma4_unified_assistant, Google DeepMind, Apache-2.0). vLLM hf_config_override recognizes BOTH gemma4_assistant (plain E2B/E4B/26B-A4B/31B) and gemma4_unified_assistant (the Unified/multimodal line = our checkpoint family); both normalize to internal gemma4_mtp with forced n_predict=1. Recorded as the ESCALATION candidate; chosen route stays DSpark draft_model (deepseek-ai/dspark_gemma4_12b_block7) per user measure-first. DSpark config.json (fetched): target_model_type gemma4_unified, num_target_layers 48, vocab_size 262144 -- purpose-built for this target. All 3 checkpoints share vocab_size 262144 + GemmaTokenizer (necessary-not-sufficient for acceptance; measured numbers are t4, gated on #71). Live behaviour on vLLM 0.22.1 flagged to-verify-at-serve. Doc: docs/gemma4-mtp-draft.md. (2) PLANNING CORRECTION: t3 (make gemma wiring catalog-driven) is NOT buildable-now -- the repo GUARDS the no-spec invariant with 3 tests naming #75 as the follow-up: test_gemma_has_no_speculative_config (asserts gemma.speculative_config==\"\"), test_fleet_compose_multimodal_vision_active_no_spec_decode (asserts vllm-multimodal has NO --speculative-config), + the MTP-items drift guard. Adding DSpark speculative_config + flipping those guards IS the restore action, gated behind measurement (t5) + #71. So only t1+t2 were buildable-now; t3->t5 resume when #71 serve-enablement lands. Docs PR opened for the #75 spec/plan/grounding. NOTE test_speculative_config_only_on_mtp_checkpoints passes for gemma (id contains MTP) so it is NOT a blocker -- the two blockers are the explicit gemma/no-spec assertions.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"topic": "gemma4-mtp-workforce", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T05:07:24.807686+00:00", "last_recall": "2026-07-01T18:13:02.194724+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-75-gemma4-mtp-workforce-findings", "hash": "2d373fe4374cd170fa61deb6a15e9435c41300f9d4862b5c494d257410bbaeb8", "content": "Issue #75 workforce run (2026-07-01, /assign-to-workforce, branch spec/issue-75-gemma4-mtp-draft). Wave 0 (t1,t2) built + merged as DOCS; t3 DEFERRED (re-binned behind #71). KEY FINDINGS: (1) t1 web-research RESOLVED the parked unknown: a NATIVE gemma4_assistant-family draft DOES exist for this checkpoint family -- google/gemma-4-12B-it-assistant (model_type gemma4_unified_assistant, Google DeepMind, Apache-2.0). vLLM hf_config_override recognizes BOTH gemma4_assistant (plain E2B/E4B/26B-A4B/31B) and gemma4_unified_assistant (the Unified/multimodal line = our checkpoint family); both normalize to internal gemma4_mtp with forced n_predict=1. Recorded as the ESCALATION candidate; chosen route stays DSpark draft_model (deepseek-ai/dspark_gemma4_12b_block7) per user measure-first. DSpark config.json (fetched): target_model_type gemma4_unified, num_target_layers 48, vocab_size 262144 -- purpose-built for this target. All 3 checkpoints share vocab_size 262144 + GemmaTokenizer (necessary-not-sufficient for acceptance; measured numbers are t4, gated on #71). Live behaviour on vLLM 0.22.1 flagged to-verify-at-serve. Doc: docs/gemma4-mtp-draft.md. (2) PLANNING CORRECTION: t3 (make gemma wiring catalog-driven) is NOT buildable-now -- the repo GUARDS the no-spec invariant with 3 tests naming #75 as the follow-up: test_gemma_has_no_speculative_config (asserts gemma.speculative_config==\"\"), test_fleet_compose_multimodal_vision_active_no_spec_decode (asserts vllm-multimodal has NO --speculative-config), + the MTP-items drift guard. Adding DSpark speculative_config + flipping those guards IS the restore action, gated behind measurement (t5) + #71. So only t1+t2 were buildable-now; t3->t5 resume when #71 serve-enablement lands. Docs PR opened for the #75 spec/plan/grounding. NOTE test_speculative_config_only_on_mtp_checkpoints passes for gemma (id contains MTP) so it is NOT a blocker -- the two blockers are the explicit gemma/no-spec assertions.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"topic": "gemma4-mtp-workforce", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T05:07:24.807686+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "gemma4-unified-needs-vllm-nightly-native-class", "hash": "7e085f2a0642b8f4e57b99db304ee50a8b4d77e61170630a49d14956f73fba03", "content": "DEFINITIVE (2026-07-01, DGX Spark GB10, issue #71/#73): the Gemma 4 12B gemma4_unified checkpoint (sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4) CANNOT serve on our pinned custom image (NGC 26.06 = vLLM 0.22.1+7b9cb5b7.dev + transformers 181beb3). ROOT CAUSE proven by live test: gemma4_unified has HETEROGENEOUS PER-LAYER head sizes -- 40 sliding_attention layers at head_dim=256 (o_proj in=4096) + 8 full_attention layers at global_head_dim=512 (o_proj in=8192); verified from safetensors weight shapes (L5/L47 q_proj=[8192], o_proj in=8192; sliding layers q_proj=[4096], o_proj in=4096). vLLM 0.22.1 has NO native Gemma4UnifiedForConditionalGeneration class (only gemma4/gemma4_mm/gemma4_mtp), so it falls to the Transformers modeling backend (TransformersMultiModalForCausalLM), which builds EVERY layer's Attention with a SINGLE head_size=get_head_size()=256 -> the 8 full-attention layers emit 16*256=4096 but their o_proj wants 16*512=8192 -> RuntimeError: Shape mismatch a.size(1)=4096 size_k=8192 in torch.ops._C.marlin_gemm at profile_run/_dummy_run. PRIOR TRITON HYPOTHESIS DISPROVEN: I engaged TRITON via the untried --attention-backend TRITON_ATTN CLI FLAG (log: 'Using AttentionBackendEnum.TRITON_ATTN backend') -- a different code path than the env var the prior session tried -- and it CRASHED IDENTICALLY. The kernel was never the issue; the single head_size is. No attention-backend flag or head_size override fixes it (256 breaks full layers, 512 breaks sliding layers). Weights DO load fine on the transformers backend (7.97GiB, no missing-tower errors) -- gemma4_unified is ENCODER-FREE early-fusion: no vision_tower/audio_tower, just model.vision_embedder(patch Dense+LN+2D pos, 13 wts) + model.embed_audio/embed_vision projections. sakamakismile AND coolthor/gemma-4-12B-it-NVFP4A16 are BYTE-IDENTICAL structure (verified via HTTP-range read of coolthor's remote safetensors header: same 1335 tensors) -- #74's 'coolthor is a safer checkpoint' premise is FALSE; both are gemma4_unified and hit the same wall. THE FIX (evidence-backed, not yet run): vLLM NIGHTLY ships the native Gemma4UnifiedForConditionalGeneration class (docs.vllm.ai/en/latest/api/vllm/model_executor/models/gemma4_unified/) which unifies the physical page size by giving the two attention types DIFFERENT block_sizes; recipe = nightly vLLM + VLLM_ATTENTION_BACKEND=TRITON_ATTN (+ vllm[audio], --limit-mm-per-prompt '{\"image\":4,\"audio\":1}'). REFS: vllm-project/recipes Google/Gemma4.md (uv pip install -U vllm --pre); lna-lab/gemma4-12b-vllm-sm120 (Blackwell SM120 proven via stock vllm/vllm-openai:nightly image, single-GPU; released <=0.22.1 cannot load gemma4_unified). SO #73 serve-enablement = BUMP the custom image's vLLM to NIGHTLY (native unified class), NOT just add a flag or the transformers overlay. #71 gated on that. Nightly torch loads on GB10 (sm_121 via fwd-compat) but is memory-hungry (may need primary down). Serve validation used docker run standalone on a free port 8100, --max-model-len 4096 --gpu-mem-util 0.12 --quantization compressed-tensors, minor gear stopped to free ~13GB (user-authorized), primary kept up, minor restored after.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "gemma4-unified-vllm", "relates_to": "issue-71", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T06:04:05.814462+00:00", "last_recall": "2026-07-01T18:13:02.194724+00:00", "recall_count": 1, "links": ["gemma4-serve-root-cause-nonsquare-attn-triton", "gemma4-12b-serve-blocked-fp4-marlin-vllm021"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "gemma4-unified-serve-resolved-nightly", "hash": "d870f6ec640f5480ff61016667fac7632b3cb73778627a73d4874bc6ee0fe3bd", "content": "RESOLVED & SHIPPED (2026-07-01, lobes v0.34.0, branch issue-73-gemma4-serve-nightly): the Gemma 4 12B gemma4_unified gear now SERVES on the DGX Spark GB10. FIX = Dockerfile.vllm-gemma4 rebased FROM vllm/vllm-openai:nightly (pinned by digest sha256:7c5a10e9; vLLM 0.23.1rc1.dev, native Gemma4UnifiedForConditionalGeneration class + transformers 5.12.1) + the vllm[audio] extra (av/soundfile/librosa/soxr installed via uv; audio input RESAMPLES via av/PyAV -- text+image worked without it, audio needed `av` specifically). Live-validated standalone (docker run, port 8100): text OK (Tokyo/42), image+text OK (described red-circle+GEMMA test image), audio+text OK (transcribed a 24kHz chatterbox TTS clip verbatim). Footprint ~15.7GiB (weights 8.1 + cudagraph pool 0.46 actual + KV 7.2) ~= 0.12 of 128GB. TWO vLLM 0.23 config gotchas shipped in compose/env.example: (1) VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -- vLLM's cudagraph memory ESTIMATE (12.74GiB) >> actual (0.46GiB) starved KV so util 0.12 failed 'No available memory for the cache blocks'; (2) MULTIMODAL_MAX_MODEL_LEN default 131072->8192 -- at util 0.12 the KV holds only ~24k tokens, 128k native is unserveable at the co-resident lane budget. Catalog gemma status configured->load-tested; test_gemma4_dockerfile.py + test_catalog.py updated. Closes #71 + #73; #74 (coolthor) mooted -- coolthor is byte-identical gemma4_unified, same fix. See [[gemma4-unified-needs-vllm-nightly-native-class]] for root-cause detail. OPS NOTE: the nightly serve needed a PRIMARY-DOWN window -- the GB10 (128GB unified, ~113GB baseline used with the full fleet) cannot reliably co-host gemma's ~15GB even minor-down (freed 13GB gets reabsorbed); a memory-profiling race (free memory increasing mid-profile) fails startup, and repeated attempts OOM-restarted the primary once (it recovered).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "gemma4-unified-vllm", "relates_to": "issue-71", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T07:13:59.431728+00:00", "last_recall": null, "recall_count": 0, "links": ["gemma4-unified-needs-vllm-nightly-native-class"], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "gb10-cuda-free-and-gemma-image-2026-07-01", "hash": "a433d6112ec6c07ee256a8ef33e001655b56ef29015cf4f95e9b6cec0b33984a", "content": "Benchmarking a co-resident gear on the shared DGX Spark GB10: the box runs near-full (steady ~114/121 GiB used, swap often full). `free -h` MISLEADS for GPU work \u2014 on the unified LPDDR5X memory, other vLLM gears CUDA reservations count against a NEW process, so CUDA-visible free is far less than free shows (measured ~19 GiB free when free -h showed ~33 GiB available). Check the real number with: docker run --rm --gpus all --entrypoint python3 -c \"import torch;print(torch.cuda.mem_get_info())\". To bench the Gemma 4 12B multimodal gear standalone we freed room by stopping minor and reducing the 27B primary to 64K context + util 0.38 (edit PRIMARY_MAX_MODEL_LEN/PRIMARY_GPU_MEM_UTIL in ~/.model-gear/.env, then docker compose up -d vllm-primary). Also: the on-disk lobes/vllm-gemma4:local tag is STALE (vLLM 0.21, falls back to TransformersMultiModalForCausalLM and crashes/OOMs); the validated image is lobes/vllm-gemma4:nightly-audio (vLLM 0.23.1rc1.dev672, native Gemma4UnifiedForConditionalGeneration). lobes fleet up --build rebuilds :local fresh from Dockerfile.vllm-gemma4. Gemma 12B bench result: ~23 tok/s single-stream decode, no spec-decode.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "session", "repo": "lobes-cli", "topic": "gemma4-benchmark"}, "created": "2026-07-01T16:36:32.758857+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} @@ -24,7 +24,13 @@ {"id": "always-on-duo-budget-validated-2026-07-02", "hash": "166b20b11f5aa62b85d2712aa96eed3abd9329bbfcb6ffe29a665a504d2057be", "content": "ALWAYS-ON DUO serving config LIVE-VALIDATED (2026-07-02, DGX Spark GB10, branch spec/fleet-nightly-unification, wired into templates). User wanted: Gemma base always-on at MAX context + Qwen 27B always-on (lower context OK) + both co-resident. VALIDATED co-residence: Gemma base (coolthor/gemma-4-12B-it-NVFP4A16 + native MTP) at 128K context / util 0.22 (KV holds 128K at 4.67x concurrency; footprint ~26 GiB = weights 9.25 incl assistant draft + graph 4.17 + KV ~12.6) AND Qwen3.6-27B-Text-NVFP4-MTP at 64K context / util 0.35-measured (64K at 6.36x, KV 17.5 GiB) both HEALTHY + serving together (~108 GiB used / ~13 GiB free, with embed+rerank+co-tenant services eidetic/nova/realtime/gateway up). Answer to 'can Gemma-128K + Qwen-64K co-reside?' = YES. TEMPLATE DEFAULTS now: PRIMARY_MAX_MODEL_LEN=65536 (64K, trimmed from 128K), PRIMARY_GPU_MEM_UTIL=0.30 (shaved from validated 0.35 for headroom), MULTIMODAL_MAX_MODEL_LEN=131072 (128K native, up from co-resident 8192), MULTIMODAL_GPU_MEM_UTIL=0.22 (up from 0.12). New default-fleet budget = 0.30 + 0.22 + 0.06 (embed) + 0.06 (rerank) = 0.64. KEY LESSON: Gemma-128K is CHEAP -- at util 0.45 it got 42.88 GiB KV = 13.9x the 128K context, so 128K needs only ~util 0.20-0.22 (earlier 0.40 estimate was wrong; the #71 'util 0.12 holds 24K, 128K needs much larger util' claim is SUPERSEDED). The 27B KV is util-bound not context-bound: 32K vs 64K at same util just trades concurrency, same memory. NOTE: these are TEMPLATE changes; the RUNNING fleet (~/.model-gear) still has the old config -- a live redeploy (lobes init/serve with new template) is a SEPARATE disruptive step to actually run the duo. Recorded docs/vllm-nightly-migration.md section 8. Version 0.35.0.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-02"}, "created": "2026-07-01T23:19:56.814616+00:00", "last_recall": "2026-07-03T08:03:33.189858+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "claudemd-256k-context-stale-fleet-runs-64k", "hash": "d3732d72586287b2de6491267aaef6479142800831a9b486565654656e707132", "content": "CLAUDE.md's '256K native context served at the full 256K' for the Qwen 27B primary is STALE for the FLEET (duo) deployment (verified 2026-07-03, code map for issue #81). The 256K figure (VLLM_MAX_MODEL_LEN=262144) is ONLY the legacy single-model scaffold (lobes/templates/docker-compose.yml + env.example). The always-on fleet duo template (lobes/templates/fleet/) trims the 27B primary to 64K: PRIMARY_MAX_MODEL_LEN=65536 (util 0.30), so the Gemma 4 12B multimodal gear serves its full 128K native (MULTIMODAL_MAX_MODEL_LEN=131072, util 0.22). Live-validated co-resident on the DGX Spark GB10 2026-07-02 (PR #80). Issue #81 REBALANCES this: cortex(Qwen)=128K, senses(Gemma)=32K. Don't trust the 256K line when reasoning about the fleet.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "fleet-context-window", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.843433+00:00", "last_recall": "2026-07-03T23:54:01.568905+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "fleet-containers-omits-gemma-multimodal", "hash": "a87d77b4a8f3e681b57f2edbf43d8b83e4b9563f71713d306675ec5f7e17ae09", "content": "GAP (found 2026-07-03, #81 code map): FLEET_CONTAINERS in lobes/runtime/_compose.py:42 = (FLEET_PRIMARY, FLEET_EMBED, FLEET_RERANK, FLEET_GATEWAY) OMITS the Gemma/multimodal container (no FLEET_MULTIMODAL constant), even though vllm-multimodal is default-on in the duo. So 'lobes fleet status' (fleet.py iterates _compose.fleet_containers()) does NOT report model-gear-vllm-multimodal's container state at all. Making 'senses' a first-class discoverable role (#81) REQUIRES adding this container to FLEET_CONTAINERS. Captured as decision c26 in docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "fleet-status-gap", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845251+00:00", "last_recall": "2026-07-03T23:37:43.106121+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "issue-81-cortex-senses-roles-specced", "hash": "37fc19bb17bdcac85e4d013d2815fafeb6d8a5bbe624f01c592feb3aedaed6de", "content": "Issue #81 (expose full Colleague stack as cortex/senses role-based lobes) specced via /think 2026-07-03 -> docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md. USER-confirmed FULL-#81 scope: six first-class roles cortex/senses/stt/tts/embedder/reranker; REBALANCE cortex(Qwen3.6-27B-MTP)=128K (up from fleet's 64K), senses(Gemma4-12B)=32K (down from 128K), util retuned, pressure degrades cortex->minor (senses is a distinct capability not a rung); 'lobes capabilities'/'lobes endpoint ' + gateway GET /capabilities JSON contract for Colleague; 'lobes up ' + 'colleague-stack' profile; per-role RUNTIME metrics only (Colleague owns task-quality); cortex-only-vs-cortex+senses bench profiles. KEY DECISIONS: (c24) ROLE LAYER not rename -- cortex->primary, senses->multimodal via catalog.TIER_ROLE + mirrors (_pressure_policy._TIER_ROLE, _tier_request); internal service/env/container names (vllm-primary/vllm-multimodal, PRIMARY_*/MULTIMODAL_*) NOT renamed; main|multimodal|hard|normal kept as back-compat aliases. (c25) contract transport = CLI 'lobes capabilities --json' AND gateway GET /capabilities. (c26) fix FLEET_CONTAINERS gap. 'brain' FORBIDDEN as role name/alias. Parked (nonblocking): exact utils, canonical responsibilities word-lists, 'lobes up' verb shape, whether colleague-stack bundles audio overlay. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "cortex-senses-roles", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845819+00:00", "last_recall": "2026-07-03T23:54:01.568905+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} -{"id": "pr-90-fleet-reachability-truth-review-reflex", "hash": "91a2aafd09c1fb6859b760c68c8417e2866db446a86a7edc7c3c87dadb80c26c", "content": "PR #90 (agentculture/lobes-cli, 0.38.0, 2026-07-04) shipped fleet reachability truth as ONE PR fixing #84/#87/#89: (#84) lobes status fleet-aware via _compose.is_fleet() Dockerfile.gateway marker, single-model output byte-identical; (#87) GET /capabilities + every role endpoint advertises a client-reachable origin from the request Host header with a GATEWAY_PUBLIC_URL override for tunnels; (#89) stt/tts ready is a LIVE probe of the realtime bridge GET /v1/health/ready aggregate, gateway returns 503-warming vs 502-unreachable for /v1/audio/*, Chatterbox reports honest 503 cuda_context_poisoned. Built via the devague pipeline (/think spec -> /spec-to-plan 6 tasks/3 waves decomposed by FILE for disjointness -> /assign-to-workforce) with colleague local-vLLM as the wave-0 workforce and opus for the gateway integration. LESSON \u2014 the diverse-review reflex earns its keep: the ask-colleague review AND the Qodo agentic review each independently caught real semantic bugs in the SAME audio loaded/ready area. colleague caught loaded/ready conflation (a warming backend read as not-deployed). Qodo caught two more: probe_audio_ready caught only OSError so a malformed AUDIO_URL (non-numeric port -> urlsplit(...).port raises ValueError) crashed the handler uncaught; and build_role_registry did not clamp ready on audio_configured, so a caller passing audio_ready=True with AUDIO_URL unset got ready=True/loaded=False/endpoint=empty. Fix patterns: catch ValueError in URL-parsing probes exactly like open_upstream does; make public builders self-enforce the invariants their own docstrings promise. Run BOTH review passes on a committed diff before merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "pr-90-fleet-reachability", "relates_to": "issues-84-87-89", "date": "2026-07-04"}, "created": "2026-07-04T04:03:43.104710+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "issue-81-cortex-senses-roles-specced", "hash": "37fc19bb17bdcac85e4d013d2815fafeb6d8a5bbe624f01c592feb3aedaed6de", "content": "Issue #81 (expose full Colleague stack as cortex/senses role-based lobes) specced via /think 2026-07-03 -> docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md. USER-confirmed FULL-#81 scope: six first-class roles cortex/senses/stt/tts/embedder/reranker; REBALANCE cortex(Qwen3.6-27B-MTP)=128K (up from fleet's 64K), senses(Gemma4-12B)=32K (down from 128K), util retuned, pressure degrades cortex->minor (senses is a distinct capability not a rung); 'lobes capabilities'/'lobes endpoint ' + gateway GET /capabilities JSON contract for Colleague; 'lobes up ' + 'colleague-stack' profile; per-role RUNTIME metrics only (Colleague owns task-quality); cortex-only-vs-cortex+senses bench profiles. KEY DECISIONS: (c24) ROLE LAYER not rename -- cortex->primary, senses->multimodal via catalog.TIER_ROLE + mirrors (_pressure_policy._TIER_ROLE, _tier_request); internal service/env/container names (vllm-primary/vllm-multimodal, PRIMARY_*/MULTIMODAL_*) NOT renamed; main|multimodal|hard|normal kept as back-compat aliases. (c25) contract transport = CLI 'lobes capabilities --json' AND gateway GET /capabilities. (c26) fix FLEET_CONTAINERS gap. 'brain' FORBIDDEN as role name/alias. Parked (nonblocking): exact utils, canonical responsibilities word-lists, 'lobes up' verb shape, whether colleague-stack bundles audio overlay. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "cortex-senses-roles", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845819+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 3, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "pr-90-fleet-reachability-truth-review-reflex", "hash": "91a2aafd09c1fb6859b760c68c8417e2866db446a86a7edc7c3c87dadb80c26c", "content": "PR #90 (agentculture/lobes-cli, 0.38.0, 2026-07-04) shipped fleet reachability truth as ONE PR fixing #84/#87/#89: (#84) lobes status fleet-aware via _compose.is_fleet() Dockerfile.gateway marker, single-model output byte-identical; (#87) GET /capabilities + every role endpoint advertises a client-reachable origin from the request Host header with a GATEWAY_PUBLIC_URL override for tunnels; (#89) stt/tts ready is a LIVE probe of the realtime bridge GET /v1/health/ready aggregate, gateway returns 503-warming vs 502-unreachable for /v1/audio/*, Chatterbox reports honest 503 cuda_context_poisoned. Built via the devague pipeline (/think spec -> /spec-to-plan 6 tasks/3 waves decomposed by FILE for disjointness -> /assign-to-workforce) with colleague local-vLLM as the wave-0 workforce and opus for the gateway integration. LESSON \u2014 the diverse-review reflex earns its keep: the ask-colleague review AND the Qodo agentic review each independently caught real semantic bugs in the SAME audio loaded/ready area. colleague caught loaded/ready conflation (a warming backend read as not-deployed). Qodo caught two more: probe_audio_ready caught only OSError so a malformed AUDIO_URL (non-numeric port -> urlsplit(...).port raises ValueError) crashed the handler uncaught; and build_role_registry did not clamp ready on audio_configured, so a caller passing audio_ready=True with AUDIO_URL unset got ready=True/loaded=False/endpoint=empty. Fix patterns: catch ValueError in URL-parsing probes exactly like open_upstream does; make public builders self-enforce the invariants their own docstrings promise. Run BOTH review passes on a committed diff before merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "pr-90-fleet-reachability", "relates_to": "issues-84-87-89", "date": "2026-07-04"}, "created": "2026-07-04T04:03:43.104710+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "vllm-preserve-thinking-qwen36", "hash": "9aea7c52e41a22b743843280df65110b2d74c238bece4d66fff194a48dac1617", "content": "vLLM preserve_thinking for Qwen3.6 cortex is real and supported: --default-chat-template-kwargs '{\"preserve_thinking\": true}' is a stable vLLM serve flag since 0.9.0 (both pinned images \u2014 nvcr.io/nvidia/vllm:26.04-py3 and the 2026 vllm/vllm-openai nightly \u2014 are newer). preserve_thinking is a genuine variable in the SERVED chat template (cortex serves with --tokenizer=mmangkad/Qwen3.6-27B-NVFP4), gating historical retention via `preserve_thinking is true or loop.index0 > ns.last_query_index`. Default keeps only for turns after the last user query; preserve_thinking=true keeps ALL. Older Qwen3-30B template lacks the var \u2014 it is Qwen3.6-specific. Request-level chat_template_kwargs override the server default. Open risk is the INPUT side: whether vLLM re-injects an assistant-history reasoning field back into the template \u2014 that is what issue #93's token-delta diagnostic proves. Spec: docs/specs/2026-07-07-lobes-preserves-qwen-thinking-traces-across-multi.md (issue #93).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "issue-93-think", "issue": "93"}, "created": "2026-07-07T05:22:37.832377+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} {"id": "devague-no-resolve-verb-for-blocking-items", "hash": "7ed18e35e93824f581b137e6fd605f7e9193eadd3e201e9d259b4e62f0cc4b29", "content": "devague (/think) tooling gap: there is NO verb to resolve a blocking hard question or an unknown_blocking vagueness. `reject` only accepts claim (c*) / honesty (h*) ids \u2014 not q* hard-questions or v* vagueness. `question --resolve` only handles standalone question-move items, not hard questions on claims. Re-parking a vagueness spawns a NEW v2, it does not flip the original. The convergence gate blocks ONLY on hard_questions with blocking:true and open_vagueness kind unknown_blocking; non-blocking ones ship as caveats in the exported spec. Practical rule: record hard questions/risks as NON-blocking from the start, or hand-edit .devague/frames/.json to set blocking:false / kind:unknown_nonblocking to converge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "issue-93-think"}, "created": "2026-07-07T05:22:37.836620+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "lobes-advertised-implies-reachable-2026-07-09", "hash": "4095aa5c41e5be9ecc1273f6a11bbd8de1161d6e9d81974e75f156fdb7242d9c", "content": "lobes-cli 2026-07-09 live-rig investigation unifying #92/#91/#95/#74/#69 into ONE spec (docs/specs/2026-07-09-lobes-never-advertises-a-capability-it-cannot-serv.md, branch spec/advertised-implies-reachable-92-91-74-69). THE ISSUES NAMED THE SYMPTOMS RIGHT AND THE CAUSES WRONG. (1) #92 is NOT a regression of #87: reachable_origin shipped in 0.38.0/PR#90 (merged 2026-07-04), but the deployed gateway image was built 2026-07-03 carrying lobes 0.36.0 (verified: docker exec model-gear-gateway python -c \"import lobes; lobes.__version__\" -> 0.36.0; hasattr(server,\"reachable_origin\") -> False). Pre-fix code falls back to _gateway_base_url() = the gateway INTERNAL container port :8000, while the published host port is VLLM_PORT=8001. NOTHING DETECTS GATEWAY CONTAINER VERSION SKEW \u2014 that absence is the real defect. Extra hazard: host :8000 is NOT dead, it is reachy-mini-dae (an unrelated uvicorn app) answering {\"detail\":\"Not Found\"}, so the advertised endpoint points at a foreign live service. (2) #91 is NOT a backend-reload window: handle_post calls rewrite_model ONCE before the failover loop, and order_backends returns EVERY same-task generate backend as a failover candidate \u2014 for the cortex model that is [primary, multimodal, multimodal-coder, middle]. Cortex returns >=500 (vllm.v1.engine.exceptions.EngineDeadError 04:08:34) -> gateway retries the SAME body, still naming the Qwen id, against the Gemma backend -> Gemma correctly 404s NotFoundError -> handle_post rule \"2xx or 4xx -> commit to this backend (4xx is a client error; no failover)\" relays it as a TERMINAL 404. PROOF, not theory: model-gear-vllm-primary (the Qwen container) logs \"The model `coolthor/gemma-4-12B-it-NVFP4A16` does not exist.\" at 04:34:01 and 04:39:01 \u2014 the symmetric direction. tests/test_gateway_routing.py::test_order_backends_generate_still_failovers_between_generate_backends ASSERTS THE BUG AS INTENDED BEHAVIOUR. (3) TWO UNFILED BUGS of the same shape, both found here, now #96 and #97. #96: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml:144, never the base fleet template, so ServerConfig.audio_url is empty and POST /v1/audio/speech 404s \"audio endpoints are not configured\" \u2014 yet `lobes capabilities` reads the merged .env (which HAS AUDIO_URL) and reports stt/tts ready=true/loaded=true. #97: _optional_backend wires a backend on *_BASE_URL OR *_SERVED_NAME with a default_url naming a container that need not exist -> /v1/models lists 6 models against 4 running containers. (4) KEY INSIGHT: NEITHER contract surface is authoritative and they are wrong in OPPOSITE directions \u2014 for the generate roles the gateway JSON is wrong and the CLI is right; for the audio roles the CLI is wrong and the gateway is right. Both derive readiness from CONFIGURATION and neither dials anything. build_role_registry is genuinely one builder, but it is fed two different gateway_url values (CLI: .env VLLM_PORT; gateway: its own GATEWAY_PORT), so \"one source of truth\" holds for the SHAPE and fails for the ORIGIN. (5) EVIDENCE GATHERED: coolthor/gemma-4-12B-it-NVFP4A16 genuinely PERCEIVES images \u2014 a stdlib-generated solid-red PNG -> \"Red\", solid-blue -> \"Blue\" (ground-truth checked). tests/test_smoke_duo.py:287-347 only asserts HTTP 200 + non-empty content on a 1x1 placeholder, i.e. it proves the WIRE not the PERCEPTION; \"image+text confirmed\" in the docs was that weaker claim. Audio perception could NOT be proven because generating real speech needs /v1/audio/speech, which is bug #96. (6) USER DECISIONS in the frame: NO cross-backend failover at all (one backend per request; dead owner -> 503 + Retry-After; order_backends collapses to a single-element list) \u2014 stricter than model-exact-with-tier-rewrite, chosen because a caller asking for cortex must never silently receive Gemma; readiness becomes a BACKGROUND cached probe mirroring PressureCache; phantom backends stopped by BOTH a config gate (require *_BASE_URL) and the readiness filter; #69 DSpark disabled-entry criterion closed ANSWERED-NEGATIVE (#75 proved Gemma4DSparkModel does not load on vLLM 0.23) with docs/gemma4-mtp-draft.md to be corrected; EngineDeadError root cause OUT of scope -> filed as #98. (7) NEW REQUIREMENT from user: a LOCAL, single-trigger, UNATTENDED pre-PR live test of the capabilities contract (not CI \u2014 no GPU there), which must FAIL not skip. It must also compare the deployed gateway container lobes.__version__ against the CLI wheel. (8) TOOL GAP: devague has NO move to un-park a blocking vagueness \u2014 once parked as unknown_blocking, converge can never pass. Had to hand-edit .devague/frames/.json to re-kind v3 to follow_up. Worth an upstream issue on agentculture/devague. (9) #69 audit: 6/8 acceptance criteria DONE; README.md:52-68 still documents `lobes init` as scaffolding the SINGLE-model deployment though fleet is the default since e51ffc8 (stale, contradicts init.py:106-111). #74 audit: the coolthor default switched in 0.35.0 justified by MTP decode throughput, NOT by the image/audio precondition #74 itself set.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "advertised-implies-reachable", "relates_to": "issues-92-91-95-74-69-96-97-98", "date": "2026-07-09"}, "created": "2026-07-09T05:37:04.254050+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "gateway-public-url-must-not-default-2026-07-09", "hash": "b00294589b18a9dcf624e32a58ee3c93699fa5c87323463f7b86d30e100d62d0", "content": "lobes-cli gateway origin advertising: NEVER default GATEWAY_PUBLIC_URL to a localhost URL in the compose template. reachable_origin(host_header, public_url) checks `if public_url: return public_url` BEFORE `if host_header: ...`, so a defaulted public_url (e.g. GATEWAY_PUBLIC_URL=${GATEWAY_PUBLIC_URL:-http://localhost:${VLLM_PORT:-8000}}) is ALWAYS set and the Host header is never consulted. A LAN or tunnel client then GETs /capabilities and is told to dial http://localhost:8001 \u2014 which on ITS machine is a different service. That is the exact #92 defect (advertised endpoint points at a foreign daemon) reintroduced by the fix for it. Correct precedence: explicit operator override (GATEWAY_PUBLIC_URL, for a tunnel / Host-rewriting reverse proxy) > the origin the client actually dialed (Host header) > NOTHING (empty endpoint). Never fabricate an absolute URL from the internal GATEWAY_PORT. This was caught only because the spec claim (c11) was tested against a simulated LAN Host header before merging the subagent's faithful implementation of it \u2014 the REQUIREMENT was wrong, not the code. Amended to c29 in the frame lobes-never-advertises-a-capability-it-cannot-serv. LESSON: when a spec says 'configured truth beats inference', check whether the 'configured truth' is actually per-deployment truth or per-CALLER truth. An origin is per-caller.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "gateway-origin-advertising", "relates_to": "issues-92-96", "date": "2026-07-09"}, "created": "2026-07-09T07:07:45.762815+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "devague-cli-gaps-2026-07-09", "hash": "6493960c999eb7f5248596d5c45c83c2daaed559149de80320a916a4de6cd53a", "content": "devague CLI gaps hit while running /think -> /spec-to-plan -> /assign-to-workforce on lobes-cli (2026-07-09), worth upstream issues on agentculture/devague: (1) NO MOVE TO UN-PARK A BLOCKING VAGUENESS. Once `devague park \"...\" --kind unknown_blocking` is recorded, converge can never pass \u2014 `confirm`/`reject` accept only claim ids (c*) and honesty ids (h*), not vagueness ids (v*). Workaround: hand-edit .devague/frames/.json to re-kind the item to follow_up. A decided unknown is no longer vagueness, so there should be a `resolve` move. (2) `devague plan confirm` takes ONE task id, unlike `devague confirm` which is transactional over many ids \u2014 `plan confirm t1 t2 t3` errors with 'unrecognized arguments'. Loop instead. (3) After amending the source frame (reject a claim, capture a replacement), `devague plan cover --target ` FAILS with 'run devague plan show to see targets' until you first run `devague plan status` (or converge), which re-derives and persists the plan's targets from the live frame. Run `plan status` immediately after any frame amendment. (4) The plan JSON key is `targets`, not `coverage_targets`. Also: `devague plan converge` correctly refuses when the source frame has regressed below convergence \u2014 re-converge the frame first.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "devague-cli", "relates_to": "think-spec-to-plan-workforce", "date": "2026-07-09"}, "created": "2026-07-09T07:07:45.847201+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "senses-gemma-cannot-hear-2026-07-09", "hash": "93dee119b623b8d35a77422ef56604c09119e3fb72196f1433f91c04a8514dbf", "content": "lobes `senses` role (coolthor/gemma-4-12B-it-NVFP4A16 on vLLM 0.23.1rc1.dev672+g93d8f834d) CANNOT HEAR \u2014 vLLM silently DROPS `input_audio` OpenAI content parts (filed as issue #101). Proof by prompt-token accounting via model=multimodal: text-only = 15 prompt_tokens; text+image (96x96 solid PNG) = 273 (+258, and the model correctly answers \"Red\"/\"Blue\"); text+audio (0.68s WAV @24kHz from the rig's own Chatterbox) = 34 (+19 placeholder tokens) with empty content; same clip resampled to 16kHz = 48 tokens and the model replies \"I cannot hear any audio because you haven't provided a file or a link\". Sample rate is NOT the cause. The checkpoint is fine: AutoConfig shows audio_config + audio_token_id + vision_config. So vLLM's gemma4_unified path wires the vision encoder and not the audio encoder, and DROPS rather than REJECTS the audio part \u2014 caller gets HTTP 200 and a fluent answer that ignored the audio. HOW IT WENT UNNOTICED FOR SO LONG: tests/test_smoke_duo.py's live layer asserted only `assert content.strip()` (HTTP 200 + non-empty) against a 1x1 placeholder PNG and a tiny WAV. That proves the WIRE, not the PERCEPTION. The docs' \"image+text \u2713 / audio+text \u2713\" for the sakamakismile coder checkpoint rest on the same weak check. METHOD THAT WORKS: generate a solid-colour PNG with stdlib zlib+struct, ask \"What single colour fills this image? Answer with one word.\", assert the colour name appears; ALWAYS run a negative control (feed blue, assert \"red\" fails) or the test is vacuous. For audio, synthesize a known word with the rig's own Chatterbox TTS (POST /v1/audio/speech) and assert the transcription contains it. The audio probe is now @pytest.mark.xfail(strict=True) citing #101, so it XPASSes and fails the suite the day audio starts working. NOTE the `stt` role (Parakeet, POST /v1/audio/transcriptions) is a separate purpose-built path and WORKS: tts(\"banana\") -> stt -> \"Banana.\"; tts(\"the quick brown fox\") -> \"The Quick Brown Fox\".", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "senses-audio", "relates_to": "issues-101-74", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.311269+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "lobes-pressure-swap-occupancy-sticky-2026-07-09", "hash": "7f666a9ea25527288cadc52f8b87e5499164ea353ffed40d2e3eda9aa7868e6b", "content": "lobes pressure policy sheds 100% of generate traffic FOREVER on a box whose swap has ever filled (filed as issue #100). lobes/runtime/_pressure.py computes swap_used_percent = (SwapTotal - SwapFree)/SwapTotal from /proc/meminfo \u2014 an OCCUPANCY metric. Swap occupancy is STICKY: the kernel does not reclaim swapped pages just because RAM freed, so once it crosses the 75% default threshold it stays there until swapoff/swapon. On the DGX Spark GB10 (2026-07-09): swap 16322/16383 MB = 99.6% occupancy, BUT `vmstat` si=0 so=0 (nothing being swapped) and /proc/pressure/memory shows `full avg10=0.00 avg60=0.00` (ZERO memory stall), with 26 GB RAM available. Since #88 (0.37.0) the policy SHEDS with 429 server_busy instead of degrading, so every cortex/senses request 429s while `X-Lobes-Override: 1` still returns 200. The rig had been fine on gateway 0.36.0 (pre-shed) and broke the instant it was upgraded to 0.39.0 \u2014 the wire had been armed for weeks. WORKAROUND (documented, but it DISABLES the policy rather than fixing it): set LOBES_SWAP_DEGRADED_THRESHOLD=100 and LOBES_IOWAIT_DEGRADED_THRESHOLD=100 in the deployment .env and recreate the gateway. Note LOBES_IOWAIT_DEGRADED_THRESHOLD=100 was ALREADY set by a previous operator for the same class of reason (0.36.2 CHANGELOG documents \"phantom high iowait on an idle disk, e.g. the DGX Spark GB10\") \u2014 half the policy had already been switched off, which is the tell that the METRIC is wrong, not the threshold. RIGHT FIX: read PSI (/proc/pressure/memory, /proc/pressure/io avg10) or swap RATE (pswpin/pswpout deltas from /proc/vmstat), not occupancy; keep occupancy at most as a secondary AND-condition. Also: a gateway that has shed 100% of requests for an hour is misconfigured, not busy \u2014 nothing currently warns.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "pressure-policy", "relates_to": "issues-100-85-86-88", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.369909+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} +{"id": "lobes-fleet-frozen-image-pin-2026-07-09", "hash": "08f35f73d1ad1f81fc3dd0a6d348a2d8f7f6c2ef4581caa92dfba6c7a56b14c5", "content": "lobes-cli fleet deployments FREEZE their lobes version and nothing bumps it (issue #99). Dockerfile.gateway/realtime/stt/chatterbox all run: pip install --no-cache-dir lobes-cli==${MODEL_GEAR_VERSION}. `lobes init` writes MODEL_GEAR_VERSION into .env ONCE at scaffold time and no verb ever rewrites it, so `docker compose up -d --build gateway` faithfully reinstalls the OLD pinned version. On 2026-07-09 the reference rig ran THREE lobes versions at once: gateway 0.36.0 (image built 07-03), realtime 0.34.1 (07-02), stt/chatterbox 0.34.1-era (07-02) \u2014 against a CLI at 0.39.0, with 0.36.1/0.36.2/0.37.0/0.38.0/0.39.0 ALL published on PyPI. CONSEQUENCE: three bugs were reported against lobes CODE that were merely undeployed fixes. (a) #92 'advertised endpoint :8000 404s' \u2014 reachable_origin shipped in 0.38.0/PR#90 on 07-04; the gateway image predated it by one day, so it fell back to _gateway_base_url() = its INTERNAL container port 8000 while the published host port is 8001. Worse, host :8000 was an unrelated uvicorn service (reachy-mini-dae), so the advertised endpoint pointed at a foreign daemon. (b) the stt sidecar answered /v1/health/ready with 200 status=ready while its CUDA context was poisoned and transcription 500d \u2014 #89 (07-04) had replaced the old liveness-only handler with a model_loaded+cuda_ok probe, but the stt image was built 07-02. (c) chatterbox likewise. CHEAP FIX (no model reload): edit ~/.lobes/.env MODEL_GEAR_VERSION=, then from ~/.lobes run: docker compose -f docker-compose.yml -f docker-compose.audio.yml up -d --build --no-deps gateway realtime stt chatterbox. GOTCHA 1: the compose PROJECT is named lobes (from the dir basename) while containers are named model-gear-* via container_name \u2014 check the com.docker.compose.project label before invoking compose or you create a duplicate project. GOTCHA 2: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml, so a gateway recreated without the -f audio overlay silently loses the audio lane (issue #96). GOTCHA 3: a from-source checkout can be AHEAD of PyPI, in which case the gateway image cannot be built from the pin at all \u2014 the dev path needs a locally built wheel COPYd in.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "fleet-version-pin", "relates_to": "issues-99-92-89-96", "date": "2026-07-09"}, "created": "2026-07-09T09:38:41.844728+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}} diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bf9e6e..9e69209 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,42 @@ All notable changes to this project are documented here. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.40.0] - 2026-07-09 + +### Security + +- The `Host` request header is validated against a strict host-authority allowlist before it is echoed as an advertised origin in `GET /capabilities`. Previously the c29 origin-resolution change reflected an unsanitized, client-controlled `Host` header into every role's `endpoint` (e.g. `http://evil.example/../