lobes__public.jsonl

{"id": "qwen3.5-4b-verified-facts", "hash": "3f61d119c727d0f228e4d2e9dd8dfd71690bf0d526c8511dca023eb553c7d64f", "content": "Qwen/Qwen3.5-4B verified from HF config.json (2026-06-26): architecture is HYBRID linear-attention (Gated Delta/SSM) + full-attention (32 layers: 24 linear + 8 full), NOT MoE and NOT plain dense -> no vLLM --moe-backend. It is MULTIMODAL (full ViT, image+video tokens) so serve text-only via --language-model-only. Ships a BUILT-IN MTP draft head (mtp_num_hidden_layers=1) -> native speculative decoding, no separate draft checkpoint. 256K native (max_position_embeddings 262144), mRoPE. dtype bf16, public, Apache-2.0. Tool-call format = qwen3_coder (XML <tool_call><function=...>), NOT hermes. No official nvidia/RedHatAI NVFP4 for the 4B; community cosmicproc/Qwen3.5-4B-NVFP4 exists (NVIDIA ModelOpt NVFP4 W4A4 -> vLLM --quantization modelopt_fp4; conv1d + lm_head left bf16; needs Blackwell for W4A4 kernels).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "hf-config-json", "topic": "qwen3.5-4b", "relates_to": "lobes-catalog", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.173778+00:00", "last_recall": "2026-07-03T08:03:33.189858+00:00", "recall_count": 7, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "minor-lobe-spec-issue-64", "hash": "8b97d0eb6db4a548000aa5e6d06068f6173bc562f16c2f8693f0a4ec028da4bb", "content": "lobes issue #64 ('minor lobe') was specced via /think on 2026-06-26 -> docs/specs/2026-06-26-lobes-ships-a-minor-lobe-a-cheap-warm-co-resident.md. Scope (user-confirmed) is the FULL minor-role runtime, not just a catalog entry: add Qwen3.5-4B as catalog gear role_hint=minor, served bf16 WARM CO-RESIDENT behind the gateway, + new read-only verbs lobes run/route/eval with escalation + confidence governance. Key decisions: served checkpoint = bf16 Qwen/Qwen3.5-4B (chosen because unsloth LoRA needs the bf16 base; NVFP4 cosmicproc is doc-only untested); route v1 routes ONLY across lobes GEARS (minor vs primary vs candidate) + escalate, not tools/mesh agents; run/route/eval call the model via the gateway OpenAI endpoint reusing assess.py's stdlib urllib client. Deferred (not yet): actual fine-tuning/lobes train, eval-suite contents, MTP spec-decode for minor. Two code gotchas it requires: (1) runtime/_parser.py infer_parser returns 'hermes' for a qwen3.5 id but the model needs 'qwen3_coder' -> must extend _RULES (catalog test asserts tool_parser==infer_parser(id)); (2) catalog requires non-empty quantization for generate gears, so a bf16 gear needs a quantization-field convention (sentinel that switch/compose translate to omitting --quantization).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "minor-lobe", "relates_to": "issue-64", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.175114+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "devague-blocking-hard-question-gotcha", "hash": "0b8151753dab00bb4c64218896d4cf470b2bb08e3c1a13b5ad000f6c89e13723", "content": "devague (the /think CLI) gotcha: a BLOCKING hard question (interrogate --hard-question --blocking) is NOT cleared by rejecting the claim it hangs on. convergence._missing_open_uncertainty iterates ALL frame.claims (including rejected) for q.blocking and not q.resolved. There is no CLI verb to resolve a hard question (interrogate has no --resolve; confirm/reject only take c*/h* ids; question --resolve is for the separate 'question' move's items). Workaround: edit the local working-state JSON at .devague/frames/<slug>.json and set the hard_question's resolved=true (honest when the question is actually answered). Also: the convergence gate requires a confirmed honesty condition on EVERY confirmed spec-affecting claim (audience/before_state/why_it_matters/after_state/boundary/success_signal/requirement/announcement), not just requirements; decisions/assumptions/non_goals are exempt.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "devague", "relates_to": "think-skill", "date": "2026-06-26"}, "created": "2026-06-26T09:44:18.175320+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "qwen3.5-vllm-0.19-fla-gibberish-gb10", "hash": "d038d33600580542974ea8dc412cce46f97641e2f6a77edc77f9633aa1c492a2", "content": "Serving Qwen3.5 (Qwen3_5ForConditionalGeneration; Gated DeltaNet linear-attention hybrid) on the lobes GB10 fleet's vLLM image nvcr.io/nvidia/vllm:26.04-py3 (engine 0.19.0): the architecture IS registered (verified via ModelRegistry: Qwen3_5ForConditionalGeneration + Qwen3_5MTP present) so it LOADS, BUT vLLM 0.19.0 carries a known FLA/Gated-DeltaNet tensor-format bug on Blackwell (sm_120/sm_121) that produces GIBBERISH output. The fix (vLLM PR #43961) shipped only in v0.23.0. So a live Qwen3.5-4B serve on the current production image likely emits garbled text -> NOT a meaningful live test. Also: V2 model runner crashes on the mixed-attention KV cache (workaround --enforce-v1); GDN needs --max-num-batched-tokens 2096 for cache alignment; sm_121 needs forward-compat (TORCH_CUDA_ARCH_LIST=12.0). --language-model-only correctly drops the vision tower for inference of the full multimodal checkpoint. A meaningful live 4B co-resident test needs vLLM >= 0.23.0 (newer image). The minor-lobe CODE (issue #64: catalog gear, qwen3_coder parser rule, run/route/eval verbs, governance, opt-in co-resident fleet service) is independent and CI-green. Fleet facts: gateway published on host :8001 (not :8000); deployment dir ~/.model-gear; HF cache /home/spark/.cache/huggingface; image runtime=runc.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "qwen3.5-serving", "relates_to": "minor-lobe", "date": "2026-06-26"}, "created": "2026-06-26T11:11:12.839972+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "devague-no-resolve-park", "hash": "42eca84ddc6356f0a0d522b8953439304b28417a757f6ecab5b4f9d674a6a744", "content": "devague (the /think CLI) has NO resolve-park / edit-vagueness move. A park created with --kind unknown_blocking blocks `converge` by design, and there is no CLI verb to resolve it once the user decides the unknown \u2014 re-running `park` with the same text just creates a DUPLICATE vagueness item (v3), it does not update in place. To converge after a blocking unknown is genuinely decided, edit the frame JSON directly at .devague/frames/<slug>.json: reclassify the items \"kind\" from \"unknown_blocking\" to \"follow_up\" (or remove it) and record the resolution in its \"text\". Frame files live under .devague/frames/ (NOT .devague/*.json); spec exports land in docs/specs/<date>-<slug>.md. Also: a capture with default origin lands as confirmed (treat as user-stated); --origin llm lands as proposed and needs user confirm. Honesty conditions are required on announcement/audience/before_state/after_state/why_it_matters/requirement/success_signal claims but NOT on decision/non_goal/assumption claims.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"area": "devague", "topic": "think-skill-gotcha"}, "created": "2026-06-26T14:54:28.559433+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "minor-lobe-live-spark-fleet", "hash": "e35daba9e905594346fe19cef3be94a89f381915ae79481a3904bdef2c66b7bb", "content": "GO-LIVE (2026-06-26): the minor lobe (Qwen/Qwen3.5-4B) is LIVE on the spark DGX fleet. Deployment dir is ~/.model-gear (legacy; no ~/.lobes/$LOBES_DIR set), compose project name model-gear, composed from BOTH docker-compose.yml + docker-compose.audio.yml (audio services are NOT behind a profile). Did a full re-scaffold to 0.30.0: overwrote docker-compose.yml + Dockerfile.gateway from the repo 0.30.0 templates (gateway now pip-installs lobes-cli==0.30.0, entrypoint lobes.gateway; was 0.26.0/model_gear pre-rename), and patched .env: MODEL_GEAR_VERSION=0.30.0, added MINOR_BASE_URL=http://vllm-minor:8000, MINOR_SERVED_NAME=Qwen/Qwen3.5-4B, COMPOSE_PROFILES=minor. Applied with: docker compose -f docker-compose.yml -f docker-compose.audio.yml --profile minor up -d --build --no-deps gateway vllm-minor (only gateway recreated + vllm-minor started; primary/embed/rerank/audio untouched). GPU utils: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10 = 0.82 of 128GB unified \u2014 fits. Gateway host port 8001; /v1/models now lists Qwen/Qwen3.5-4B. AUTH is NOT in compose \u2014 it is a HOST cloudflared tunnel process (decoupled), so compose re-scaffold does not touch it; just keep gateway on host port 8001. Rollback files: ~/.model-gear/{docker-compose.yml,Dockerfile.gateway,.env}.preminor.bak. GOTCHAS: (1) gateway reads MINOR_BASE_URL (NOT MINOR_URL like the <NAME>_URL convention for PRIMARY/EMBED/RERANK); empty default => minor silently unwired. (2) chatterbox TTS hit a poisoned CUDA context (torch.AcceleratorError cudaErrorUnknown, HTTP 500) \u2014 cleared by docker compose restart chatterbox; synthesis works fine WITH the 4B co-resident (the 4B did not cause GPU exhaustion).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"area": "deployment", "host": "spark", "topic": "minor-lobe-golive"}, "created": "2026-06-26T15:35:48.071970+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 6, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-68-tiered-routing-spec", "hash": "7d1af62c72622a18e11b0becc36982cf148e2c817a55242ebaf5bef1741673b5", "content": "Issue #68 (tiered model resource routing for Spark unified memory) specced via /think on 2026-06-29 -> docs/specs/2026-06-29-lobes-adds-a-third-middle-model-tier-and-pressure.md. User-confirmed decisions: (1) THIRD tier = a 14B-class NVFP4 checkpoint, INFERENCE-ONLY (~8GB weights) -- it is NOT a LoRA base; LoRA training stays on the existing 4B bf16 minor lobe (so 'scale work + train loras' = 3 inference tiers + 4B trainable base). (2) 27B primary served context trimmed 256K->128K (PRIMARY_MAX_MODEL_LEN=131072) to free ~half the KV cache for the co-resident middle gear. (3) Tier-request surface = gateway model-alias ONLY (caller sends model=cheap|normal|hard to the OpenAI endpoint; gateway resolves to 4B/14B/27B); NO 'lobes select --tier' verb, NO 'route --tier' field. (4) FULL pressure policy IS in scope: read-only swap%/iowait% sampling from /proc, #68 thresholds, degraded-mode state machine, model=hard downgrades with reason=pressure, manual override + downgrade reason cross the OpenAI boundary via HTTP headers, read-only 'lobes status --pressure' emits {tier,model,mode,reason,pressure}. Parked unknowns: exact 14B NVFP4 checkpoint id (same-gen Qwen3.6-14B vs dense Qwen3-14B-NVFP4); exact override/reason header names + whether a streaming response can carry the reason. Risk: 14B NVFP4 must load non-gibberish on the Blackwell vLLM image (dense Qwen3 NVFP4 lower-risk than a 3.5/3.6 hybrid w/ the pre-vLLM-0.23 FLA bug). Current fleet baseline: primary 0.6 + embed 0.06 + rerank 0.06 + minor 0.10. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "tiered-routing", "relates_to": "issue-68", "date": "2026-06-29"}, "created": "2026-06-29T19:44:26.978729+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 5, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-69-gemma-duo-spec", "hash": "f0fc6928c15f0438a22ae71f7a4f8db22ed8831854c41a361db6f900b8f45b04", "content": "Issue #69 (make Spark default-serve the Qwen3.6-27B + Gemma4-12B duo) specced via /think 2026-06-30 -> docs/specs/2026-06-30-lobes-now-defaults-to-a-spark-duo-the-qwen3-6-27b.md (frame .devague/frames/lobes-now-defaults-to-a-spark-duo-the-qwen3-6-27b.json). USER-confirmed decisions: (1) 'lobes serve' STOPS being single-model -- with no flags it brings up BOTH main (27B MTP primary) + multimodal (Gemma4-12B); minor/14B/legacy need explicit profiles. (2) NEW tier vocabulary main/minor/multimodal (user reframed away from cheap/normal/hard) -- Gemma gets NEW role_hint='multimodal' in catalog.py; old 'middle' role + 'normal' alias DEPRECATED (normal->multimodal back-compat kept); nvidia/Qwen3-14B-NVFP4 demoted role_hint=middle->candidate (KEPT, cite-don't-delete, legacy profile only). (3) Default Gemma serves NVFP4 + NATIVE MTP ON; DeepSeek DSpark draft (deepseek-ai/dspark_gemma4_12b_block7, shipped 2026-06-27 via DeepSpec) = disabled-by-default experiment toggle. (4) Gemma serves FULL multimodal: vision + AUDIO (Gemma4Unified ingests image/video/audio; vLLM supports it; Gemma4-12B is first medium model with native audio-in/ASR) -- chat audio-in is DISTINCT from the /v1/audio/* Parakeet(STT)/Chatterbox(TTS) overlay (unchanged per boundary). Leading checkpoint candidate: sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 (SAME publisher as the lobes primary, NVFP4+MTP). Accepted plan risks (parked unknown_nonblocking): exact checkpoint pick; Gemma4 --tool-call-parser + runtime/_parser.py infer_parser rule (catalog test asserts tool_parser==infer_parser); loads-on-nv26.04-image (LOWER risk: Gemma4UnifiedForConditionalGeneration registered, auto-detects NVFP4, NOT the Qwen3.5 Gated-DeltaNet/FLA arch that gibbered on sm_120/121); native-MTP vs DSpark --speculative-config JSON; measured GPU util for multimodal Gemma (vision+audio embedders+KV vs 14B's 0.12). DESIGN SEAM for spec-to-plan: main/minor/multimodal breaks the linear cheap/normal/hard pressure ladder (issue #68) -- 'multimodal' is a different capability, not a capability rung, so the pressure-downgrade target needs redefining. TOOLING GOTCHA: devague spec_md exporter (render/spec_md.py:66 _follow_up) DROPS unknown_nonblocking vagueness from the exported spec.md (only renders follow_up/out_of_scope) -- had to append an 'Accepted plan risks' section by hand; the frame JSON retains all parked items, so /spec-to-plan (which reads the frame) still gets them. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "gemma-duo", "relates_to": "issue-69", "date": "2026-06-30"}, "created": "2026-06-30T12:37:29.031340+00:00", "last_recall": "2026-07-03T23:57:07.981592+00:00", "recall_count": 8, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-unified-unsupported-released-vllm-images", "hash": "34c5ad26dedfe96cb163a6e3e4ad0348a6501f0f8d0be1450f24ebba3fa24840", "content": "t7 live validation (2026-06-30, DGX Spark) for the lobes Gemma 4 12B multimodal gear (issue #69): Gemma 4 12B's architecture is model_type=`gemma4_unified`, and ALL community NVFP4 12B checkpoints use it (sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4, AxionML/Gemma-4-12B-NVFP4, coolthor/gemma-4-12B-it-NVFP4A16). NEITHER released NGC vLLM image registers gemma4_unified: nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0, transformers 4.57.6) NOR nvcr.io/nvidia/vllm:26.05.post1-py3 (vLLM 0.21.0, transformers 5.6.0). vLLM crashes at config load: 'model type gemma4_unified but Transformers does not recognize this architecture ... install Transformers from source'. NOTE: 26.05/vLLM 0.21.0 DOES register Gemma4MTPModel + standard Gemma4ForConditionalGeneration (just not the Unified variant) \u2014 so a standard-arch Gemma4 would load, but the 12B unified multimodal needs nightly/source transformers. Fix path tracked in issue #71 (custom image: 26.05 base + `pip install --pre -U transformers` or git source, verify gemma4_unified registers, check vLLM 0.21.0 pin compat). The lobes catalog gear stays status=configured (correct) until a supporting image lands. Co-resident test approach = zero fleet disruption (run the multimodal container on a free host port alongside the running fleet; primary at util 0.51, +0.12 multimodal = 0.85 < 1.0, no primary bounce). Shipped in PR #72. The vLLM image's --gpus is DeviceRequests nvidia:-1 (=--gpus all), runtime=runc, HF cache /home/spark/.cache/huggingface, compose net model-gear_default, gateway host :8001, deploy dir ~/.model-gear.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-69", "date": "2026-06-30"}, "created": "2026-06-30T15:40:49.826739+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 7, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-71-gemma4-custom-image-spec", "hash": "167dd266f292a73237b77a2db62529cd14cb3664f363993eb7e34c3d4d1ad09e", "content": "Issue #71 (custom vLLM image for the Gemma 4 12B gemma4_unified multimodal gear) specced via /think 2026-06-30 -> docs/specs/2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.md. USER-confirmed decisions: (1) SCOPE: only vllm-multimodal moves to the custom 26.05-based image NOW; primary/embed/rerank stay on nvcr.io/nvidia/vllm:26.04-py3 (load-tested 27B primary not disturbed); if multimodal works well, open 3 follow-up issues to migrate primary/embed/rerank one-by-one. (2) DELIVERY: local compose build: directive on vllm-multimodal (matches gateway/chatterbox/parakeet/realtime Dockerfile pattern) is the DEFAULT; PLUS an optional MULTIMODAL_IMAGE env override so an operator can point at a ghcr.io/agentculture or local registry:2 tag (registry push feasible but not required). (3) TOOLING: Dockerfile.vllm-gemma4 FROM nvcr.io/nvidia/vllm:26.05.post1-py3 installs transformers with UV (uv pip install --system), NOT pip (user preference). (4) PIN: bake a PINNED transformers ref (version or git SHA) for reproducible rebuilds; exact ref discovered during validation (Q3 left to my default). (5) DELIVERABLE: validate+promote in ONE go on the Spark (spark-f8a9 IS the GB10 Spark, fleet live on 26.04, no vllm-multimodal up) -> the recipe PR merges regardless; status flips configured->load-tested + numbers in docs/gemma-4-12b-nvfp4.md ONLY if all pass (boots, image+text, audio+text, MTP accept>0, util in 0.69 budget); any fail -> stays configured, residual parked. Co-resident validation = zero fleet disruption (free host port alongside running fleet). Parked unknowns v1-v4: exact transformers ref; correct gemma4_mtp method string on vLLM 0.21.0 (r4); measured vision+audio util vs 0.12 (r5); native context (131072 default). Core risk: NO transformers ref may both register gemma4_unified AND keep vLLM 0.21.0 importing -> then await NGC release. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-custom-image", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T16:19:55.749207+00:00", "last_recall": "2026-07-03T23:54:01.568905+00:00", "recall_count": 3, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-unified-transformers-ref-resolved-r1", "hash": "7490113c6117e54c0ae4c8a4cec0d0accbd66d0cc4ff212ec58b4f893fdb875a", "content": "RESOLVED (2026-06-30, DGX Spark spark-f8a9, issue #71): the custom vLLM image for the Gemma 4 12B gemma4_unified gear BUILDS and the core risk r1 clears. WORKING RECIPE: FROM nvcr.io/nvidia/vllm:26.05.post1-py3 (vLLM 0.21.0+2325b6f0) + transformers from source pinned to commit 181beb3ba4c47098ed8cbc97ee250d1d45ae0107 (resolves to transformers==5.13.0.dev0). This ref BOTH registers gemma4_unified in transformers AutoConfig CONFIG_MAPPING AND leaves vLLM 0.21.0 importing/serving (the r1 question: does ANY ref do both -> YES). Verified in-image: AutoConfig.from_pretrained(sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4, trust_remote_code=True) -> model_type gemma4_unified; vllm.ModelRegistry.get_supported_archs() shows [Gemma4ForCausalLM, Gemma4ForConditionalGeneration, Gemma4MTPModel] (Gemma4MTPModel present -> good for native MTP / r4); vllm serve --help exits 0 WITH --gpus (without GPU it errors RuntimeError: Failed to infer device type -- a no-GPU artifact, NOT a real break). TWO BUILD GOTCHAS on this base: (1) a multi-line RUN python3 -c \"...\" MUST use trailing-backslash continuations or Docker parses each body line as an instruction (unknown instruction: import); (2) uv pip install --system needs --break-system-packages because the base /usr Python carries a PEP 668 EXTERNALLY-MANAGED marker (the base sets PIP_BREAK_SYSTEM_PACKAGES=1 for pip only, and PIP_CONSTRAINT=/etc/pip/constraint.txt which uv ignores -- fine, we WANT to override the pinned transformers). uv 0.11.17 already ships in the 26.05 base (no bootstrap needed). Image tag built: lobes/vllm-gemma4:local. STILL OPEN (t4): live serve of the actual checkpoint with --speculative-config gemma4_mtp (r4 method string), measured GPU util (r5), native context (v4), image+text + audio+text functional validation.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T16:56:45.301751+00:00", "last_recall": "2026-07-01T05:32:46.663593+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-12b-serve-blocked-fp4-marlin-vllm021", "hash": "1070e80076271e9ac88ec328159202179f0ef71bc3daca0704697645a7ad1d4d", "content": "t4 live serve (2026-06-30, DGX Spark, issue #71) of sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4 on the custom image (nv26.05.post1 vLLM 0.21.0 + transformers main 181beb3) is BLOCKED by a vLLM kernel bug -- the gear CANNOT serve on this image. Findings, in order hit: (1) QUANT: catalog/compose --quantization modelopt_fp4 is WRONG for this checkpoint; config.json quant_method=compressed-tensors, format nvfp4-pack-quantized (NVFP4A16). Must use --quantization compressed-tensors (or omit -> auto-detect). (2) MTP/r4 NEGATIVE: --speculative-config {method: gemma4_mtp} fails NotImplementedError: Unsupported speculative method mtp. vLLM 0.21.0 speculative.py only enables gemma4 MTP when a SEPARATE DRAFT model has model_type==gemma4_assistant (line 515 hf_config_override; use_gemma4_mtp() needs draft_model_config). There is NO auto-derivation from the gemma4_unified target (unlike deepseek_v3->deepseek_mtp), and THIS checkpoint config has no mtp/assistant/nextn keys despite the -MTP name. So native MTP needs a gemma4_assistant draft repo we do not have. Catalog speculative_config for the gemma gear is wrong -> remove/correct. (3) BLOCKER: with --quantization compressed-tensors + no spec-config, the model LOADS but CRASHES at warmup forward: RuntimeError: Shape mismatch: a.size(1)=4096, size_k=8192 in apply_fp4_marlin_linear (compressed_tensors_w4a16_nvfp4.py apply_weights -> marlin_utils_fp4) on Gemma4 attention o_proj. The CompressedTensorsW4A16Fp4 scheme is MARLIN-ONLY (no cutlass/native alt, no env override; VLLM_NVFP4_GEMM_BACKEND/CT_EMULATIONS dont apply). vLLM 0.21.0s W4A16 NVFP4 Marlin kernel mishandles Gemma4s o_proj shape. RESOLVED at config level: native context=131072 (text_config.max_position_embeddings; v4). NOT measurable: r5 GPU util, functional image/audio (blocked by the crash). Maintenance window: stopped primary(0.51)+minor(0.10) to free ~74GB (user-authorized), embed/rerank untouched, fleet restored after. NEXT OPTIONS: (A) try a different Gemma4-12B checkpoint that avoids W4A16 Marlin -- a W4A4 full-NVFP4 (cutlass path) or a nvidia modelopt_fp4 format (modelopt kernel) checkpoint (AxionML/Gemma-4-12B-NVFP4 fmt unknown; coolthor NVFP4A16 = same W4A16 problem likely). (B) await a vLLM that fixes Marlin W4A16 NVFP4 for gemma4 (or adds a non-marlin W4A16 path). Gear stays status=configured; recipe (image+wiring) + r1 win + quant/spec-config corrections still merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T17:17:56.428563+00:00", "last_recall": "2026-07-03T23:37:43.106121+00:00", "recall_count": 4, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-serve-root-cause-nonsquare-attn-triton", "hash": "acc43da8b918f91ad0d9a68c3afe107e4a6209635253e269ec1a3314613b2200", "content": "t4 deep-dive (2026-06-30, DGX Spark, issue #71): the Gemma 4 12B serve blocker is PRECISELY the non-square attention, NOT a fundamental wall. config text_config: hidden_size=3840, head_dim=256, global_head_dim=512 (DOUBLE head_dim), num_attention_heads=16, attention_k_eq_v=true. The o_proj weight expects K = num_heads*global_head_dim = 16*512 = 8192, but FLASH_ATTN emits num_heads*head_dim = 16*256 = 4096 -> RuntimeError: Shape mismatch a.size(1)=4096 size_k=8192 in marlin_gemm (o_proj). Fix per ai-muninn.com blog: VLLM_ATTENTION_BACKEND=TRITON_ATTN (Triton handles non-square attn, emits 8192). BUT in my test the env DID NOT ENGAGE: vLLM logged Using FLASH_ATTN out of potential backends [FLASH_ATTN,FLASHINFER,TRITON_ATTN,FLEX_ATTENTION], because gemma4_unified runs via vLLMs Transformers modeling backend (no native vLLM Gemma4Unified impl; native Gemma4ForConditionalGeneration is the non-unified variant) and that path did not honor VLLM_ATTENTION_BACKEND=TRITON_ATTN as a docker -e var. OPEN: how to force TRITON_ATTN on the transformers-backend path (CLI flag? hf attn_implementation override? newer vLLM native unified impl?). Same crash on vLLM 0.21.0 AND 0.22.1. RUNTIME MATRIX TESTED: nv26.05.post1=vLLM0.21.0+torch2.12.0a0nv; nv26.06=vLLM0.22.1+torch2.13.0a0nv (BLOG-PROVEN version) -- both keep NGC torch when transformers overlaid (181beb3); host venv nightly=vLLM0.23.1rc1.dev+torch2.11.0+cu130 (stock torch arch_list sm_80..sm_120 only, sm_121 via fwd-compat, torch loads on GB10 but OOMs unless fleet stopped). NOTE 0.24.x does NOT exist yet (latest nightly 0.23.1rc1). BLOG-PROVEN WORKING RECIPE (untested by us): coolthor/gemma-4-12B-it-NVFP4A16 (7.7GB) + vLLM 0.22.1 + VLLM_ATTENTION_BACKEND=TRITON_ATTN -> image+audio+video all work. Our checkpoint (sakamakismile coder-fable5 variant) may also have a nonstandard quant; coolthor is the safer checkpoint. NEXT: (1) figure out TRITON_ATTN engagement on transformers backend; (2) test coolthor; (3) consider switching catalog default to coolthor.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "gemma4-unified-vllm", "relates_to": "issue-71", "date": "2026-06-30"}, "created": "2026-06-30T17:46:48.319291+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-75-gemma4-mtp-spec", "hash": "3730e1ff2f26b4d477684710af9197daf4f2f25348579c0ddc834c6e65e8f3d0", "content": "Issue #75 (Gemma4 native MTP: source/build a gemma4_assistant draft for speculative decoding) was specced via /think on 2026-07-01 -> docs/specs/2026-07-01-gemma-4-12b-gear-gets-speculative-decoding-draft.md (frame .devague/frames/the-gemma-4-12b-gear-gets-speculative-decoding-lob.json). USER-confirmed decisions (4 AskUserQuestion answers): (1) SCOPE = any speculative-decoding speedup counts, NOT strictly native gemma4_mtp -> measure the EXISTING DSpark draft_model route (deepseek-ai/dspark_gemma4_12b_block7) FIRST as the cheap path; pursue a native gemma4_assistant draft only if DSpark proves the win is real but insufficient. (2) BUILD APPETITE = NO -- training/distilling a gemma4_assistant draft head is a SEPARATE follow-up, not in #75; #75 stops at sourced-or-DSpark + documented verdict. (3) DONE = a MEASURED VERDICT (restore speculative_config in catalog+compose with recorded acceptance%/speedup beating baseline, OR a documented negative with the numbers that rule it out). (4) SERVE GATING = #75 is BLOCKED on #71 serve-enablement -- does NOT ship draft wiring ahead of serve; only desk-sourcing a candidate draft can start before #71 lands. GROUNDING: gemma4_mtp has NO self-speculation (vLLM 0.21/0.22 need a separate model_type==gemma4_assistant draft; deepseek_v3->deepseek_mtp auto-derivation has no gemma equivalent); {method:gemma4_mtp} is rejected Unsupported speculative method; the served -MTP checkpoint exposes no mtp/assistant/nextn keys. Catalog->compose wiring follows the 27B mtp_compose_command_items() pattern. Hard dependency #71: gear LOADS but does not SERVE (non-square attention global_head_dim 512 != head_dim 256 needs TRITON_ATTN, not honored on vLLM transformers backend -> o_proj GEMM 4096!=8192). EXPORTER GOTCHA confirmed again: devague spec_md drops unknown_nonblocking vagueness (whether a sourceable gemma4_assistant draft EXISTS on HF) -> restored by hand in the spec.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"topic": "gemma4-mtp-spec", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-06-30T22:37:05.301533+00:00", "last_recall": "2026-07-01T18:13:02.194724+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-75-gemma4-mtp-workforce-findings", "hash": "2d373fe4374cd170fa61deb6a15e9435c41300f9d4862b5c494d257410bbaeb8", "content": "Issue #75 workforce run (2026-07-01, /assign-to-workforce, branch spec/issue-75-gemma4-mtp-draft). Wave 0 (t1,t2) built + merged as DOCS; t3 DEFERRED (re-binned behind #71). KEY FINDINGS: (1) t1 web-research RESOLVED the parked unknown: a NATIVE gemma4_assistant-family draft DOES exist for this checkpoint family -- google/gemma-4-12B-it-assistant (model_type gemma4_unified_assistant, Google DeepMind, Apache-2.0). vLLM hf_config_override recognizes BOTH gemma4_assistant (plain E2B/E4B/26B-A4B/31B) and gemma4_unified_assistant (the Unified/multimodal line = our checkpoint family); both normalize to internal gemma4_mtp with forced n_predict=1. Recorded as the ESCALATION candidate; chosen route stays DSpark draft_model (deepseek-ai/dspark_gemma4_12b_block7) per user measure-first. DSpark config.json (fetched): target_model_type gemma4_unified, num_target_layers 48, vocab_size 262144 -- purpose-built for this target. All 3 checkpoints share vocab_size 262144 + GemmaTokenizer (necessary-not-sufficient for acceptance; measured numbers are t4, gated on #71). Live behaviour on vLLM 0.22.1 flagged to-verify-at-serve. Doc: docs/gemma4-mtp-draft.md. (2) PLANNING CORRECTION: t3 (make gemma wiring catalog-driven) is NOT buildable-now -- the repo GUARDS the no-spec invariant with 3 tests naming #75 as the follow-up: test_gemma_has_no_speculative_config (asserts gemma.speculative_config==\"\"), test_fleet_compose_multimodal_vision_active_no_spec_decode (asserts vllm-multimodal has NO --speculative-config), + the MTP-items drift guard. Adding DSpark speculative_config + flipping those guards IS the restore action, gated behind measurement (t5) + #71. So only t1+t2 were buildable-now; t3->t5 resume when #71 serve-enablement lands. Docs PR opened for the #75 spec/plan/grounding. NOTE test_speculative_config_only_on_mtp_checkpoints passes for gemma (id contains MTP) so it is NOT a blocker -- the two blockers are the explicit gemma/no-spec assertions.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"topic": "gemma4-mtp-workforce", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T05:07:24.807686+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-unified-needs-vllm-nightly-native-class", "hash": "7e085f2a0642b8f4e57b99db304ee50a8b4d77e61170630a49d14956f73fba03", "content": "DEFINITIVE (2026-07-01, DGX Spark GB10, issue #71/#73): the Gemma 4 12B gemma4_unified checkpoint (sakamakismile/gemma-4-12B-coder-fable5-composer2.5-MTP-NVFP4) CANNOT serve on our pinned custom image (NGC 26.06 = vLLM 0.22.1+7b9cb5b7.dev + transformers 181beb3). ROOT CAUSE proven by live test: gemma4_unified has HETEROGENEOUS PER-LAYER head sizes -- 40 sliding_attention layers at head_dim=256 (o_proj in=4096) + 8 full_attention layers at global_head_dim=512 (o_proj in=8192); verified from safetensors weight shapes (L5/L47 q_proj=[8192], o_proj in=8192; sliding layers q_proj=[4096], o_proj in=4096). vLLM 0.22.1 has NO native Gemma4UnifiedForConditionalGeneration class (only gemma4/gemma4_mm/gemma4_mtp), so it falls to the Transformers modeling backend (TransformersMultiModalForCausalLM), which builds EVERY layer's Attention with a SINGLE head_size=get_head_size()=256 -> the 8 full-attention layers emit 16*256=4096 but their o_proj wants 16*512=8192 -> RuntimeError: Shape mismatch a.size(1)=4096 size_k=8192 in torch.ops._C.marlin_gemm at profile_run/_dummy_run. PRIOR TRITON HYPOTHESIS DISPROVEN: I engaged TRITON via the untried --attention-backend TRITON_ATTN CLI FLAG (log: 'Using AttentionBackendEnum.TRITON_ATTN backend') -- a different code path than the env var the prior session tried -- and it CRASHED IDENTICALLY. The kernel was never the issue; the single head_size is. No attention-backend flag or head_size override fixes it (256 breaks full layers, 512 breaks sliding layers). Weights DO load fine on the transformers backend (7.97GiB, no missing-tower errors) -- gemma4_unified is ENCODER-FREE early-fusion: no vision_tower/audio_tower, just model.vision_embedder(patch Dense+LN+2D pos, 13 wts) + model.embed_audio/embed_vision projections. sakamakismile AND coolthor/gemma-4-12B-it-NVFP4A16 are BYTE-IDENTICAL structure (verified via HTTP-range read of coolthor's remote safetensors header: same 1335 tensors) -- #74's 'coolthor is a safer checkpoint' premise is FALSE; both are gemma4_unified and hit the same wall. THE FIX (evidence-backed, not yet run): vLLM NIGHTLY ships the native Gemma4UnifiedForConditionalGeneration class (docs.vllm.ai/en/latest/api/vllm/model_executor/models/gemma4_unified/) which unifies the physical page size by giving the two attention types DIFFERENT block_sizes; recipe = nightly vLLM + VLLM_ATTENTION_BACKEND=TRITON_ATTN (+ vllm[audio], --limit-mm-per-prompt '{\"image\":4,\"audio\":1}'). REFS: vllm-project/recipes Google/Gemma4.md (uv pip install -U vllm --pre); lna-lab/gemma4-12b-vllm-sm120 (Blackwell SM120 proven via stock vllm/vllm-openai:nightly image, single-GPU; released <=0.22.1 cannot load gemma4_unified). SO #73 serve-enablement = BUMP the custom image's vLLM to NIGHTLY (native unified class), NOT just add a flag or the transformers overlay. #71 gated on that. Nightly torch loads on GB10 (sm_121 via fwd-compat) but is memory-hungry (may need primary down). Serve validation used docker run standalone on a free port 8100, --max-model-len 4096 --gpu-mem-util 0.12 --quantization compressed-tensors, minor gear stopped to free ~13GB (user-authorized), primary kept up, minor restored after.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "gemma4-unified-vllm", "relates_to": "issue-71", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T06:04:05.814462+00:00", "last_recall": "2026-07-01T18:13:02.194724+00:00", "recall_count": 1, "links": ["gemma4-serve-root-cause-nonsquare-attn-triton", "gemma4-12b-serve-blocked-fp4-marlin-vllm021"], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma4-unified-serve-resolved-nightly", "hash": "d870f6ec640f5480ff61016667fac7632b3cb73778627a73d4874bc6ee0fe3bd", "content": "RESOLVED & SHIPPED (2026-07-01, lobes v0.34.0, branch issue-73-gemma4-serve-nightly): the Gemma 4 12B gemma4_unified gear now SERVES on the DGX Spark GB10. FIX = Dockerfile.vllm-gemma4 rebased FROM vllm/vllm-openai:nightly (pinned by digest sha256:7c5a10e9; vLLM 0.23.1rc1.dev, native Gemma4UnifiedForConditionalGeneration class + transformers 5.12.1) + the vllm[audio] extra (av/soundfile/librosa/soxr installed via uv; audio input RESAMPLES via av/PyAV -- text+image worked without it, audio needed `av` specifically). Live-validated standalone (docker run, port 8100): text OK (Tokyo/42), image+text OK (described red-circle+GEMMA test image), audio+text OK (transcribed a 24kHz chatterbox TTS clip verbatim). Footprint ~15.7GiB (weights 8.1 + cudagraph pool 0.46 actual + KV 7.2) ~= 0.12 of 128GB. TWO vLLM 0.23 config gotchas shipped in compose/env.example: (1) VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 -- vLLM's cudagraph memory ESTIMATE (12.74GiB) >> actual (0.46GiB) starved KV so util 0.12 failed 'No available memory for the cache blocks'; (2) MULTIMODAL_MAX_MODEL_LEN default 131072->8192 -- at util 0.12 the KV holds only ~24k tokens, 128k native is unserveable at the co-resident lane budget. Catalog gemma status configured->load-tested; test_gemma4_dockerfile.py + test_catalog.py updated. Closes #71 + #73; #74 (coolthor) mooted -- coolthor is byte-identical gemma4_unified, same fix. See [[gemma4-unified-needs-vllm-nightly-native-class]] for root-cause detail. OPS NOTE: the nightly serve needed a PRIMARY-DOWN window -- the GB10 (128GB unified, ~113GB baseline used with the full fleet) cannot reliably co-host gemma's ~15GB even minor-down (freed 13GB gets reabsorbed); a memory-profiling race (free memory increasing mid-profile) fails startup, and repeated attempts OOM-restarted the primary once (it recovered).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "gemma4-unified-vllm", "relates_to": "issue-71", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T07:13:59.431728+00:00", "last_recall": null, "recall_count": 0, "links": ["gemma4-unified-needs-vllm-nightly-native-class"], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gb10-cuda-free-and-gemma-image-2026-07-01", "hash": "a433d6112ec6c07ee256a8ef33e001655b56ef29015cf4f95e9b6cec0b33984a", "content": "Benchmarking a co-resident gear on the shared DGX Spark GB10: the box runs near-full (steady ~114/121 GiB used, swap often full). `free -h` MISLEADS for GPU work \u2014 on the unified LPDDR5X memory, other vLLM gears CUDA reservations count against a NEW process, so CUDA-visible free is far less than free shows (measured ~19 GiB free when free -h showed ~33 GiB available). Check the real number with: docker run --rm --gpus all --entrypoint python3 <img> -c \"import torch;print(torch.cuda.mem_get_info())\". To bench the Gemma 4 12B multimodal gear standalone we freed room by stopping minor and reducing the 27B primary to 64K context + util 0.38 (edit PRIMARY_MAX_MODEL_LEN/PRIMARY_GPU_MEM_UTIL in ~/.model-gear/.env, then docker compose up -d vllm-primary). Also: the on-disk lobes/vllm-gemma4:local tag is STALE (vLLM 0.21, falls back to TransformersMultiModalForCausalLM and crashes/OOMs); the validated image is lobes/vllm-gemma4:nightly-audio (vLLM 0.23.1rc1.dev672, native Gemma4UnifiedForConditionalGeneration). lobes fleet up --build rebuilds :local fresh from Dockerfile.vllm-gemma4. Gemma 12B bench result: ~23 tok/s single-stream decode, no spec-decode.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "session", "repo": "lobes-cli", "topic": "gemma4-benchmark"}, "created": "2026-07-01T16:36:32.758857+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "spec-fleet-nightly-unification-2026-07-01", "hash": "b1e55dde298f5940101250b964a9f89f6c7e847da4c9342fce3b747b51b224c0", "content": "Fleet-wide vLLM-nightly unification + #75 Gemma-MTP follow-through: SPEC + PLAN both exported 2026-07-01 on branch spec/fleet-nightly-unification (commits a272b02 spec, 3c6ffa1 plan; no PR yet). spec: docs/specs/2026-07-01-lobes-unifies-its-generate-lane-on-one-vllm-nightl.md; plan: docs/plans/2026-07-01-lobes-unifies-its-generate-lane-on-one-vllm-nightl.md. CONTEXT: #71/#73 landed (Gemma 4 12B serves on vLLM nightly 0.23.1rc1 native Gemma4UnifiedForConditionalGeneration, ~23 tok/s no-spec) -> unblocked #75 t3-t5 (Gemma MTP measure-leg) AND proved the nightly base image runs on GB10 (sm_121), reopening the 27B primary vLLM upgrade the qwen3.6-27b doc said was blocked on the 26.04 base. 5 USER-confirmed decisions: (1) COMBINED (Gemma MTP + Qwen 27B nightly upgrade + head-to-head = one spec). (2) COMMIT-TO-NIGHTLY (27B default image flips nvcr.io/nvidia/vllm:26.04-py3/0.19.0 -> nightly; rollback = pinned 0.19.0 until nightly >= parity). (3) BENCHMARK INFORM-ONLY (no auto primary-swap). (4) DRAFT ROUTE DSpark-FIRST (deepseek-ai/dspark_gemma4_12b_block7 via draft_model; native google/gemma-4-12B-it-assistant is escalation). (5) FLEET-WIDE (user added mid-session: ALL vLLM gears move to nightly; realtime Parakeet/Chatterbox EXCLUDED, not vLLM). PLAN = 9 tasks / 6 waves: w0 (parallel de-risk, ZERO fleet mutation) t1 before-state+baselines doc, t2 spike 27B standalone on nightly, t3 spike embed+rerank pooling; w1 t4 flip primary/embed/rerank default images to one pinned nightly digest; w2 t5 wire Gemma DSpark speculative_config catalog-driven+round-trip (NOT default-on yet); w3 t6 measure Gemma MTP + same-nightly head-to-head, t8 trailing minor/14B; w4 t7 commit verdict (flip 3 guard tests default-on if wins, else document negative); w5 t9 shipped-state docs. t4/t5/t8 all edit fleet docker-compose.yml -> serialized by deps (t4->t5->t8). Covers all 24 targets. 6 first-class risks: r1 qwen3_5_mtp deprecated on 0.19.0 -> nightly may need method=mtp or gibber via Qwen3.5 hybrid/FLA (resolve via t2 spike, fallback method=mtp); r2 nightly memory-hungry -> fleet may not co-reside on GB10 -> standalone benchmark; r3 nightly image digest to standardize (v1); r4 DSpark num_speculative_tokens (block_size=7, experiment used 3) (v2); r5 minor/14B blocks-or-trails (v3, planned trailing); r6 native gemma4_assistant escalation (v4, follow_up). BASELINES TO BEAT: 27B ~19 tok/s + 72-79% accept; Gemma ~23 tok/s no-spec. NEXT LEG: /assign-to-workforce (fan out waves) OR open the PR.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "decision", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T18:44:23.097262+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "t2-27b-nightly-spike-GO-2026-07-01", "hash": "97e494eb38321df01785d2532944c1e33d8622d12911ef53fb3634caf3944db1", "content": "GO/NO-GO RESOLVED (2026-07-01, DGX Spark GB10, branch spec/fleet-nightly-unification): the 27B primary sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP SERVES on vLLM nightly (vllm/vllm-openai:nightly = 0.23.1rc1.dev672) with MTP ACTIVE. This is the t2 spike of the fleet-nightly-unification plan and it PASSED, retiring risk r1 and validating assumption c22. FINDINGS (standalone docker run, free port 8100, util 0.40, max-model-len 8192, primary stopped in an authorized maintenance window to free ~72GB then restarted): (1) Resolved architecture Qwen3_5ForConditionalGeneration + draft Qwen3_5MTP. (2) --speculative-config method qwen3_5_mtp logs 'deprecated and replaced with mtp' and AUTO-MAPS to SpeculativeConfig(method='mtp', num_spec_tokens=3) -- MTP stays on, NO caller/config change needed (r1 resolved). (3) --quantization modelopt resolves to modelopt_fp4, served via FlashInferCutlassNvFp4LinearKernel (native NVFP4 cutlass). (4) Qwen3.5 hybrid attention: 'Using Triton/FLA GDN prefill kernel' -- no gibber. (5) Correctness 17x23=391 finish=stop. (6) Valid qwen3_coder tool-call get_weather({city:Paris}) finish=tool_calls. (7) MTP draft acceptance 69.9% (174/249 tokens, small 83-draft sample -- at/near the 72-79% 0.19.0 baseline). (8) ~20.7 tok/s incl prefill (256 tok/12.4s) -- parity with ~18.7-19.1 baseline. Load 18.65GB weights + 25.68GB KV (224824-tok cache), init 141s (compile 39s). CAVEATS: acceptance/throughput are functional-parity signals not production benchmark (t6 does apples-to-apples). RISK r2 CONFIRMED: GB10 was memory-saturated (118/121GB used, swap 15/15 full) so the fleet CANNOT co-reside a 2nd large gear on nightly -> t6 head-to-head will need sequential/standalone; t4 image-flip must mind util budget. Nightly image already local (no pull). PROGRESS: t1 (before-state doc) + t5 (gemma DSpark round-trip test, guards preserved) + t2 all done/committed. REMAINING: t3 embed/rerank spike (needs a lighter minor-only window), t4 flip images (code, unblocked by t2 GO but needs t3), t6 head-to-head (window+standalone), t7 verdict, t8 minor/14B, t9 docs. Recorded in docs/vllm-nightly-migration.md section 4.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T19:18:10.194386+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "t6-headtohead-dspark-invalid-2026-07-01", "hash": "321495e51a6d65267860a72b7aeb701576fb4fbd0979fb0b1971a81f90f903e8", "content": "t6 head-to-head + DSpark verdict (2026-07-01, DGX Spark GB10, full maintenance window, branch spec/fleet-nightly-unification). Both generate gears benchmarked STANDALONE on the SAME engine vLLM 0.23.1rc1.dev672 (r2 blocks co-residence). RESULTS (batch=1 greedy, max_len 8192, ignore_eos 800-tok decode + 3201-tok prefill): Qwen 27B MTP = ~17.6-18.0 tok/s decode (vLLM-logged) / ~2190 tok/s prefill / 60.6% MTP draft acceptance (modelopt_fp4, util 0.40). Gemma 12B NO-SPEC = ~22.8-23.0 tok/s decode / ~1966 tok/s prefill (compressed-tensors, util 0.30). Gemma OUT-DECODES the 27B (~23 vs ~18) just by being smaller -- inform-only, NO swap (c19). NOTE: 27B on nightly is ~PARITY, slightly BELOW its 0.19.0 baseline (18.7-19.1 tok/s, 72-79% accept) on this spike -- nightly is a unification move (c18), not a 27B speedup. DSPARK ROUTE = INVALID ON vLLM 0.23 (h5 resolved NEGATIVE): serving Gemma with deepseek-ai/dspark_gemma4_12b_block7 fails -> (1) config key `draft_model_id` is OUTDATED, vLLM 0.23 SpeculativeConfig wants `model` (the doc/t5-recorded config is wrong); (2) DECISIVE: `Model architectures ['Gemma4DSparkModel'] are not supported for now` -- DSpark drafter arch not in vLLM 0.23 supported speculative-draft set. BUT `Gemma4MTPModel` IS in that supported list -> the NATIVE route google/gemma-4-12B-it-assistant (gemma4_unified_assistant->gemma4_mtp, method mtp, n_predict=1) is the VIABLE Gemma-MTP path; measuring it is the follow-up (r6/v4), NOT done. t7 VERDICT: keep Gemma no-spec by default, do NOT wire DSpark, document the negative, fix the config-key note. Side finding: VLLM_ATTENTION_BACKEND is flagged 'unknown env var' on nightly (native Gemma4Unified class auto-forces TRITON) -> the compose env line is a no-op warning now (clean up in t7/t9). PROGRESS: t1-t6 done; remaining t7 (verdict, code), t8 (minor/14B), t9 (docs). All recorded in docs/vllm-nightly-migration.md sections 4-6.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-01"}, "created": "2026-07-01T20:11:57.319090+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma-mtp-coder-vs-base-2026-07-02", "hash": "85d26068d6b2c9d100da8ab3a133a731e7b4ba30b09297ac11390e88f683d8bc", "content": "Gemma MTP checkpoint choice RESOLVED (2026-07-02, DGX Spark, native google/gemma-4-12B-it-assistant draft via method=mtp n_predict=1, all on nightly 0.23.1rc1.dev672, batch=1 greedy max_len 8192). KEY: MTP acceptance is dominated by target-vs-assistant match. Gemma CODER (sakamakismile NVFP4 coder-fable5) + MTP = 30.8% accept, ~24 tok/s (~1.04x over ~23 no-spec) -- MARGINAL, the coder fine-tune's distribution shift kills acceptance. Gemma BASE (google/gemma-4-12B-it bf16, the assistant's EXACT target) + MTP = 93.9% accept, 14.6 tok/s vs 6.5 no-spec = ~2.25x MTP speedup -- acceptance is EXCELLENT on the matched base. BUT the base is bf16 -> compute-bound: 14.6 tok/s absolute is STILL SLOWER than the NVFP4 coder (23-24) because bf16 12B no-spec is only 6.5 tok/s vs NVFP4 ~23. CONCLUSION: the winning 'less-coder-enjoy-MTP-more' gear = an NVFP4 BASE-IT checkpoint (NVFP4 speed ~23 x base ~90% MTP accept ~= genuinely fast high-MTP gear). Candidate coolthor/gemma-4-12B-it-NVFP4A16 (NVFP4 of the base it-model; NVFP4A16 W4A16, needs nightly's native Gemma4Unified class which we have). The bf16 base is the acceptance PROOF, too slow to ship co-resident. SUPPORT-BOTH plan: catalog carries coder (NVFP4, coding, no useful MTP) + base (NVFP4, general, native MTP default-on) so callers pick coding-strength vs MTP-throughput. Recorded docs/vllm-nightly-migration.md section 7. Correct native-MTP config: {\"method\":\"mtp\",\"model\":\"google/gemma-4-12B-it-assistant\",\"num_speculative_tokens\":1} (NOT draft_model_id; that key is rejected on 0.23). Assistant is text-only drafter (falls back text-only mode), KV-shares draft layers 46/47 with target.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-02"}, "created": "2026-07-01T21:36:07.502625+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gemma-nvfp4-base-mtp-winner-2026-07-02", "hash": "c08f1ed5bed11e7ec6d723ff092a34e6829616ab751c66c22bd3e71fc9043810", "content": "WINNING Gemma-MTP gear MEASURED (2026-07-02, DGX Spark, nightly 0.23.1rc1.dev672): coolthor/gemma-4-12B-it-NVFP4A16 (NVFP4 compressed-tensors nvfp4-pack, gemma4_unified, 7.7GB, the NVFP4 quant of base google/gemma-4-12B-it) + native MTP draft google/gemma-4-12B-it-assistant (method=mtp, num_speculative_tokens=1) = 28.6 tok/s decode (19.8 no-spec x ~1.45x MTP @ 57.9% acceptance). This is the FASTEST Gemma config measured, beating coder-NVFP4 no-spec (~23), coder+MTP (~24, 30.8% accept), and bf16 base+MTP (14.6, 93.9% accept). KEY LESSON: MTP acceptance tracks target-vs-assistant match AND quant -- coder fine-tune 30.8%, NVFP4 base 57.9%, bf16 base 93.9% (NVFP4 quant shifts the target distribution vs what the bf16 assistant expects, halving acceptance -- but NVFP4 speed more than compensates). Serve flags = same as coder gemma (--quantization compressed-tensors, TRITON auto via native Gemma4Unified class on nightly, --tool-call-parser pythonic, util 0.30, max-model-len 8192) PLUS --speculative-config. SUPPORT-BOTH (user request, next code leg): catalog.py carries TWO gemma gears -- coder (sakamakismile NVFP4, coding-strong, NO MTP wired) + base (coolthor NVFP4, native MTP default-on ~28.6 tok/s). Callers pick coding-strength vs MTP-throughput. The base gear pins google/gemma-4-12B-it-assistant (838MB bf16 drafter) as a dep. DSpark route stays DEAD (Gemma4DSparkModel unsupported on 0.23). Recorded docs/vllm-nightly-migration.md section 7.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-02"}, "created": "2026-07-01T22:13:13.178800+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "always-on-duo-budget-validated-2026-07-02", "hash": "166b20b11f5aa62b85d2712aa96eed3abd9329bbfcb6ffe29a665a504d2057be", "content": "ALWAYS-ON DUO serving config LIVE-VALIDATED (2026-07-02, DGX Spark GB10, branch spec/fleet-nightly-unification, wired into templates). User wanted: Gemma base always-on at MAX context + Qwen 27B always-on (lower context OK) + both co-resident. VALIDATED co-residence: Gemma base (coolthor/gemma-4-12B-it-NVFP4A16 + native MTP) at 128K context / util 0.22 (KV holds 128K at 4.67x concurrency; footprint ~26 GiB = weights 9.25 incl assistant draft + graph 4.17 + KV ~12.6) AND Qwen3.6-27B-Text-NVFP4-MTP at 64K context / util 0.35-measured (64K at 6.36x, KV 17.5 GiB) both HEALTHY + serving together (~108 GiB used / ~13 GiB free, with embed+rerank+co-tenant services eidetic/nova/realtime/gateway up). Answer to 'can Gemma-128K + Qwen-64K co-reside?' = YES. TEMPLATE DEFAULTS now: PRIMARY_MAX_MODEL_LEN=65536 (64K, trimmed from 128K), PRIMARY_GPU_MEM_UTIL=0.30 (shaved from validated 0.35 for headroom), MULTIMODAL_MAX_MODEL_LEN=131072 (128K native, up from co-resident 8192), MULTIMODAL_GPU_MEM_UTIL=0.22 (up from 0.12). New default-fleet budget = 0.30 + 0.22 + 0.06 (embed) + 0.06 (rerank) = 0.64. KEY LESSON: Gemma-128K is CHEAP -- at util 0.45 it got 42.88 GiB KV = 13.9x the 128K context, so 128K needs only ~util 0.20-0.22 (earlier 0.40 estimate was wrong; the #71 'util 0.12 holds 24K, 128K needs much larger util' claim is SUPERSEDED). The 27B KV is util-bound not context-bound: 32K vs 64K at same util just trades concurrency, same memory. NOTE: these are TEMPLATE changes; the RUNNING fleet (~/.model-gear) still has the old config -- a live redeploy (lobes init/serve with new template) is a SEPARATE disruptive step to actually run the duo. Recorded docs/vllm-nightly-migration.md section 8. Version 0.35.0.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"topic": "fleet-nightly-unification", "relates_to": "issue-75", "source": "claude-code-session", "date": "2026-07-02"}, "created": "2026-07-01T23:19:56.814616+00:00", "last_recall": "2026-07-03T08:03:33.189858+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "claudemd-256k-context-stale-fleet-runs-64k", "hash": "d3732d72586287b2de6491267aaef6479142800831a9b486565654656e707132", "content": "CLAUDE.md's '256K native context served at the full 256K' for the Qwen 27B primary is STALE for the FLEET (duo) deployment (verified 2026-07-03, code map for issue #81). The 256K figure (VLLM_MAX_MODEL_LEN=262144) is ONLY the legacy single-model scaffold (lobes/templates/docker-compose.yml + env.example). The always-on fleet duo template (lobes/templates/fleet/) trims the 27B primary to 64K: PRIMARY_MAX_MODEL_LEN=65536 (util 0.30), so the Gemma 4 12B multimodal gear serves its full 128K native (MULTIMODAL_MAX_MODEL_LEN=131072, util 0.22). Live-validated co-resident on the DGX Spark GB10 2026-07-02 (PR #80). Issue #81 REBALANCES this: cortex(Qwen)=128K, senses(Gemma)=32K. Don't trust the 256K line when reasoning about the fleet.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "fleet-context-window", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.843433+00:00", "last_recall": "2026-07-03T23:54:01.568905+00:00", "recall_count": 1, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "fleet-containers-omits-gemma-multimodal", "hash": "a87d77b4a8f3e681b57f2edbf43d8b83e4b9563f71713d306675ec5f7e17ae09", "content": "GAP (found 2026-07-03, #81 code map): FLEET_CONTAINERS in lobes/runtime/_compose.py:42 = (FLEET_PRIMARY, FLEET_EMBED, FLEET_RERANK, FLEET_GATEWAY) OMITS the Gemma/multimodal container (no FLEET_MULTIMODAL constant), even though vllm-multimodal is default-on in the duo. So 'lobes fleet status' (fleet.py iterates _compose.fleet_containers()) does NOT report model-gear-vllm-multimodal's container state at all. Making 'senses' a first-class discoverable role (#81) REQUIRES adding this container to FLEET_CONTAINERS. Captured as decision c26 in docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "claude-code-session", "topic": "fleet-status-gap", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845251+00:00", "last_recall": "2026-07-03T23:37:43.106121+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "issue-81-cortex-senses-roles-specced", "hash": "37fc19bb17bdcac85e4d013d2815fafeb6d8a5bbe624f01c592feb3aedaed6de", "content": "Issue #81 (expose full Colleague stack as cortex/senses role-based lobes) specced via /think 2026-07-03 -> docs/specs/2026-07-03-lobes-exposes-the-full-colleague-runtime-stack-as.md. USER-confirmed FULL-#81 scope: six first-class roles cortex/senses/stt/tts/embedder/reranker; REBALANCE cortex(Qwen3.6-27B-MTP)=128K (up from fleet's 64K), senses(Gemma4-12B)=32K (down from 128K), util retuned, pressure degrades cortex->minor (senses is a distinct capability not a rung); 'lobes capabilities'/'lobes endpoint <role>' + gateway GET /capabilities JSON contract for Colleague; 'lobes up <role>' + 'colleague-stack' profile; per-role RUNTIME metrics only (Colleague owns task-quality); cortex-only-vs-cortex+senses bench profiles. KEY DECISIONS: (c24) ROLE LAYER not rename -- cortex->primary, senses->multimodal via catalog.TIER_ROLE + mirrors (_pressure_policy._TIER_ROLE, _tier_request); internal service/env/container names (vllm-primary/vllm-multimodal, PRIMARY_*/MULTIMODAL_*) NOT renamed; main|multimodal|hard|normal kept as back-compat aliases. (c25) contract transport = CLI 'lobes capabilities --json' AND gateway GET /capabilities. (c26) fix FLEET_CONTAINERS gap. 'brain' FORBIDDEN as role name/alias. Parked (nonblocking): exact utils, canonical responsibilities word-lists, 'lobes up' verb shape, whether colleague-stack bundles audio overlay. Next leg: /spec-to-plan.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "cortex-senses-roles", "relates_to": "issue-81", "date": "2026-07-03"}, "created": "2026-07-03T08:22:08.845819+00:00", "last_recall": "2026-07-09T05:13:17.278668+00:00", "recall_count": 3, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "pr-90-fleet-reachability-truth-review-reflex", "hash": "91a2aafd09c1fb6859b760c68c8417e2866db446a86a7edc7c3c87dadb80c26c", "content": "PR #90 (agentculture/lobes-cli, 0.38.0, 2026-07-04) shipped fleet reachability truth as ONE PR fixing #84/#87/#89: (#84) lobes status fleet-aware via _compose.is_fleet() Dockerfile.gateway marker, single-model output byte-identical; (#87) GET /capabilities + every role endpoint advertises a client-reachable origin from the request Host header with a GATEWAY_PUBLIC_URL override for tunnels; (#89) stt/tts ready is a LIVE probe of the realtime bridge GET /v1/health/ready aggregate, gateway returns 503-warming vs 502-unreachable for /v1/audio/*, Chatterbox reports honest 503 cuda_context_poisoned. Built via the devague pipeline (/think spec -> /spec-to-plan 6 tasks/3 waves decomposed by FILE for disjointness -> /assign-to-workforce) with colleague local-vLLM as the wave-0 workforce and opus for the gateway integration. LESSON \u2014 the diverse-review reflex earns its keep: the ask-colleague review AND the Qodo agentic review each independently caught real semantic bugs in the SAME audio loaded/ready area. colleague caught loaded/ready conflation (a warming backend read as not-deployed). Qodo caught two more: probe_audio_ready caught only OSError so a malformed AUDIO_URL (non-numeric port -> urlsplit(...).port raises ValueError) crashed the handler uncaught; and build_role_registry did not clamp ready on audio_configured, so a caller passing audio_ready=True with AUDIO_URL unset got ready=True/loaded=False/endpoint=empty. Fix patterns: catch ValueError in URL-parsing probes exactly like open_upstream does; make public builders self-enforce the invariants their own docstrings promise. Run BOTH review passes on a committed diff before merge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "project", "record_metadata": {"source": "claude-code-session", "topic": "pr-90-fleet-reachability", "relates_to": "issues-84-87-89", "date": "2026-07-04"}, "created": "2026-07-04T04:03:43.104710+00:00", "last_recall": "2026-07-09T05:13:18.802890+00:00", "recall_count": 2, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "vllm-preserve-thinking-qwen36", "hash": "9aea7c52e41a22b743843280df65110b2d74c238bece4d66fff194a48dac1617", "content": "vLLM preserve_thinking for Qwen3.6 cortex is real and supported: --default-chat-template-kwargs '{\"preserve_thinking\": true}' is a stable vLLM serve flag since 0.9.0 (both pinned images \u2014 nvcr.io/nvidia/vllm:26.04-py3 and the 2026 vllm/vllm-openai nightly \u2014 are newer). preserve_thinking is a genuine variable in the SERVED chat template (cortex serves with --tokenizer=mmangkad/Qwen3.6-27B-NVFP4), gating historical <think> retention via `preserve_thinking is true or loop.index0 > ns.last_query_index`. Default keeps <think> only for turns after the last user query; preserve_thinking=true keeps ALL. Older Qwen3-30B template lacks the var \u2014 it is Qwen3.6-specific. Request-level chat_template_kwargs override the server default. Open risk is the INPUT side: whether vLLM re-injects an assistant-history reasoning field back into the template \u2014 that is what issue #93's token-delta diagnostic proves. Spec: docs/specs/2026-07-07-lobes-preserves-qwen-thinking-traces-across-multi.md (issue #93).", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "issue-93-think", "issue": "93"}, "created": "2026-07-07T05:22:37.832377+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "devague-no-resolve-verb-for-blocking-items", "hash": "7ed18e35e93824f581b137e6fd605f7e9193eadd3e201e9d259b4e62f0cc4b29", "content": "devague (/think) tooling gap: there is NO verb to resolve a blocking hard question or an unknown_blocking vagueness. `reject` only accepts claim (c*) / honesty (h*) ids \u2014 not q* hard-questions or v* vagueness. `question --resolve` only handles standalone question-move items, not hard questions on claims. Re-parking a vagueness spawns a NEW v2, it does not flip the original. The convergence gate blocks ONLY on hard_questions with blocking:true and open_vagueness kind unknown_blocking; non-blocking ones ship as caveats in the exported spec. Practical rule: record hard questions/risks as NON-blocking from the start, or hand-edit .devague/frames/<slug>.json to set blocking:false / kind:unknown_nonblocking to converge.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "reference", "record_metadata": {"source": "issue-93-think"}, "created": "2026-07-07T05:22:37.836620+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "lobes-advertised-implies-reachable-2026-07-09", "hash": "4095aa5c41e5be9ecc1273f6a11bbd8de1161d6e9d81974e75f156fdb7242d9c", "content": "lobes-cli 2026-07-09 live-rig investigation unifying #92/#91/#95/#74/#69 into ONE spec (docs/specs/2026-07-09-lobes-never-advertises-a-capability-it-cannot-serv.md, branch spec/advertised-implies-reachable-92-91-74-69). THE ISSUES NAMED THE SYMPTOMS RIGHT AND THE CAUSES WRONG. (1) #92 is NOT a regression of #87: reachable_origin shipped in 0.38.0/PR#90 (merged 2026-07-04), but the deployed gateway image was built 2026-07-03 carrying lobes 0.36.0 (verified: docker exec model-gear-gateway python -c \"import lobes; lobes.__version__\" -> 0.36.0; hasattr(server,\"reachable_origin\") -> False). Pre-fix code falls back to _gateway_base_url() = the gateway INTERNAL container port :8000, while the published host port is VLLM_PORT=8001. NOTHING DETECTS GATEWAY CONTAINER VERSION SKEW \u2014 that absence is the real defect. Extra hazard: host :8000 is NOT dead, it is reachy-mini-dae (an unrelated uvicorn app) answering {\"detail\":\"Not Found\"}, so the advertised endpoint points at a foreign live service. (2) #91 is NOT a backend-reload window: handle_post calls rewrite_model ONCE before the failover loop, and order_backends returns EVERY same-task generate backend as a failover candidate \u2014 for the cortex model that is [primary, multimodal, multimodal-coder, middle]. Cortex returns >=500 (vllm.v1.engine.exceptions.EngineDeadError 04:08:34) -> gateway retries the SAME body, still naming the Qwen id, against the Gemma backend -> Gemma correctly 404s NotFoundError -> handle_post rule \"2xx or 4xx -> commit to this backend (4xx is a client error; no failover)\" relays it as a TERMINAL 404. PROOF, not theory: model-gear-vllm-primary (the Qwen container) logs \"The model `coolthor/gemma-4-12B-it-NVFP4A16` does not exist.\" at 04:34:01 and 04:39:01 \u2014 the symmetric direction. tests/test_gateway_routing.py::test_order_backends_generate_still_failovers_between_generate_backends ASSERTS THE BUG AS INTENDED BEHAVIOUR. (3) TWO UNFILED BUGS of the same shape, both found here, now #96 and #97. #96: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml:144, never the base fleet template, so ServerConfig.audio_url is empty and POST /v1/audio/speech 404s \"audio endpoints are not configured\" \u2014 yet `lobes capabilities` reads the merged .env (which HAS AUDIO_URL) and reports stt/tts ready=true/loaded=true. #97: _optional_backend wires a backend on *_BASE_URL OR *_SERVED_NAME with a default_url naming a container that need not exist -> /v1/models lists 6 models against 4 running containers. (4) KEY INSIGHT: NEITHER contract surface is authoritative and they are wrong in OPPOSITE directions \u2014 for the generate roles the gateway JSON is wrong and the CLI is right; for the audio roles the CLI is wrong and the gateway is right. Both derive readiness from CONFIGURATION and neither dials anything. build_role_registry is genuinely one builder, but it is fed two different gateway_url values (CLI: .env VLLM_PORT; gateway: its own GATEWAY_PORT), so \"one source of truth\" holds for the SHAPE and fails for the ORIGIN. (5) EVIDENCE GATHERED: coolthor/gemma-4-12B-it-NVFP4A16 genuinely PERCEIVES images \u2014 a stdlib-generated solid-red PNG -> \"Red\", solid-blue -> \"Blue\" (ground-truth checked). tests/test_smoke_duo.py:287-347 only asserts HTTP 200 + non-empty content on a 1x1 placeholder, i.e. it proves the WIRE not the PERCEPTION; \"image+text confirmed\" in the docs was that weaker claim. Audio perception could NOT be proven because generating real speech needs /v1/audio/speech, which is bug #96. (6) USER DECISIONS in the frame: NO cross-backend failover at all (one backend per request; dead owner -> 503 + Retry-After; order_backends collapses to a single-element list) \u2014 stricter than model-exact-with-tier-rewrite, chosen because a caller asking for cortex must never silently receive Gemma; readiness becomes a BACKGROUND cached probe mirroring PressureCache; phantom backends stopped by BOTH a config gate (require *_BASE_URL) and the readiness filter; #69 DSpark disabled-entry criterion closed ANSWERED-NEGATIVE (#75 proved Gemma4DSparkModel does not load on vLLM 0.23) with docs/gemma4-mtp-draft.md to be corrected; EngineDeadError root cause OUT of scope -> filed as #98. (7) NEW REQUIREMENT from user: a LOCAL, single-trigger, UNATTENDED pre-PR live test of the capabilities contract (not CI \u2014 no GPU there), which must FAIL not skip. It must also compare the deployed gateway container lobes.__version__ against the CLI wheel. (8) TOOL GAP: devague has NO move to un-park a blocking vagueness \u2014 once parked as unknown_blocking, converge can never pass. Had to hand-edit .devague/frames/<slug>.json to re-kind v3 to follow_up. Worth an upstream issue on agentculture/devague. (9) #69 audit: 6/8 acceptance criteria DONE; README.md:52-68 still documents `lobes init` as scaffolding the SINGLE-model deployment though fleet is the default since e51ffc8 (stale, contradicts init.py:106-111). #74 audit: the coolthor default switched in 0.35.0 justified by MTP decode throughput, NOT by the image/audio precondition #74 itself set.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "advertised-implies-reachable", "relates_to": "issues-92-91-95-74-69-96-97-98", "date": "2026-07-09"}, "created": "2026-07-09T05:37:04.254050+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "gateway-public-url-must-not-default-2026-07-09", "hash": "b00294589b18a9dcf624e32a58ee3c93699fa5c87323463f7b86d30e100d62d0", "content": "lobes-cli gateway origin advertising: NEVER default GATEWAY_PUBLIC_URL to a localhost URL in the compose template. reachable_origin(host_header, public_url) checks `if public_url: return public_url` BEFORE `if host_header: ...`, so a defaulted public_url (e.g. GATEWAY_PUBLIC_URL=${GATEWAY_PUBLIC_URL:-http://localhost:${VLLM_PORT:-8000}}) is ALWAYS set and the Host header is never consulted. A LAN or tunnel client then GETs /capabilities and is told to dial http://localhost:8001 \u2014 which on ITS machine is a different service. That is the exact #92 defect (advertised endpoint points at a foreign daemon) reintroduced by the fix for it. Correct precedence: explicit operator override (GATEWAY_PUBLIC_URL, for a tunnel / Host-rewriting reverse proxy) > the origin the client actually dialed (Host header) > NOTHING (empty endpoint). Never fabricate an absolute URL from the internal GATEWAY_PORT. This was caught only because the spec claim (c11) was tested against a simulated LAN Host header before merging the subagent's faithful implementation of it \u2014 the REQUIREMENT was wrong, not the code. Amended to c29 in the frame lobes-never-advertises-a-capability-it-cannot-serv. LESSON: when a spec says 'configured truth beats inference', check whether the 'configured truth' is actually per-deployment truth or per-CALLER truth. An origin is per-caller.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "gateway-origin-advertising", "relates_to": "issues-92-96", "date": "2026-07-09"}, "created": "2026-07-09T07:07:45.762815+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "devague-cli-gaps-2026-07-09", "hash": "6493960c999eb7f5248596d5c45c83c2daaed559149de80320a916a4de6cd53a", "content": "devague CLI gaps hit while running /think -> /spec-to-plan -> /assign-to-workforce on lobes-cli (2026-07-09), worth upstream issues on agentculture/devague: (1) NO MOVE TO UN-PARK A BLOCKING VAGUENESS. Once `devague park \"...\" --kind unknown_blocking` is recorded, converge can never pass \u2014 `confirm`/`reject` accept only claim ids (c*) and honesty ids (h*), not vagueness ids (v*). Workaround: hand-edit .devague/frames/<slug>.json to re-kind the item to follow_up. A decided unknown is no longer vagueness, so there should be a `resolve` move. (2) `devague plan confirm` takes ONE task id, unlike `devague confirm` which is transactional over many ids \u2014 `plan confirm t1 t2 t3` errors with 'unrecognized arguments'. Loop instead. (3) After amending the source frame (reject a claim, capture a replacement), `devague plan cover <tN> --target <newId>` FAILS with 'run devague plan show to see targets' until you first run `devague plan status` (or converge), which re-derives and persists the plan's targets from the live frame. Run `plan status` immediately after any frame amendment. (4) The plan JSON key is `targets`, not `coverage_targets`. Also: `devague plan converge` correctly refuses when the source frame has regressed below convergence \u2014 re-converge the frame first.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "devague-cli", "relates_to": "think-spec-to-plan-workforce", "date": "2026-07-09"}, "created": "2026-07-09T07:07:45.847201+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "senses-gemma-cannot-hear-2026-07-09", "hash": "93dee119b623b8d35a77422ef56604c09119e3fb72196f1433f91c04a8514dbf", "content": "lobes `senses` role (coolthor/gemma-4-12B-it-NVFP4A16 on vLLM 0.23.1rc1.dev672+g93d8f834d) CANNOT HEAR \u2014 vLLM silently DROPS `input_audio` OpenAI content parts (filed as issue #101). Proof by prompt-token accounting via model=multimodal: text-only = 15 prompt_tokens; text+image (96x96 solid PNG) = 273 (+258, and the model correctly answers \"Red\"/\"Blue\"); text+audio (0.68s WAV @24kHz from the rig's own Chatterbox) = 34 (+19 placeholder tokens) with empty content; same clip resampled to 16kHz = 48 tokens and the model replies \"I cannot hear any audio because you haven't provided a file or a link\". Sample rate is NOT the cause. The checkpoint is fine: AutoConfig shows audio_config + audio_token_id + vision_config. So vLLM's gemma4_unified path wires the vision encoder and not the audio encoder, and DROPS rather than REJECTS the audio part \u2014 caller gets HTTP 200 and a fluent answer that ignored the audio. HOW IT WENT UNNOTICED FOR SO LONG: tests/test_smoke_duo.py's live layer asserted only `assert content.strip()` (HTTP 200 + non-empty) against a 1x1 placeholder PNG and a tiny WAV. That proves the WIRE, not the PERCEPTION. The docs' \"image+text \u2713 / audio+text \u2713\" for the sakamakismile coder checkpoint rest on the same weak check. METHOD THAT WORKS: generate a solid-colour PNG with stdlib zlib+struct, ask \"What single colour fills this image? Answer with one word.\", assert the colour name appears; ALWAYS run a negative control (feed blue, assert \"red\" fails) or the test is vacuous. For audio, synthesize a known word with the rig's own Chatterbox TTS (POST /v1/audio/speech) and assert the transcription contains it. The audio probe is now @pytest.mark.xfail(strict=True) citing #101, so it XPASSes and fails the suite the day audio starts working. NOTE the `stt` role (Parakeet, POST /v1/audio/transcriptions) is a separate purpose-built path and WORKS: tts(\"banana\") -> stt -> \"Banana.\"; tts(\"the quick brown fox\") -> \"The Quick Brown Fox\".", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "investigation", "record_metadata": {"source": "claude-code-session", "topic": "senses-audio", "relates_to": "issues-101-74", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.311269+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "lobes-pressure-swap-occupancy-sticky-2026-07-09", "hash": "7f666a9ea25527288cadc52f8b87e5499164ea353ffed40d2e3eda9aa7868e6b", "content": "lobes pressure policy sheds 100% of generate traffic FOREVER on a box whose swap has ever filled (filed as issue #100). lobes/runtime/_pressure.py computes swap_used_percent = (SwapTotal - SwapFree)/SwapTotal from /proc/meminfo \u2014 an OCCUPANCY metric. Swap occupancy is STICKY: the kernel does not reclaim swapped pages just because RAM freed, so once it crosses the 75% default threshold it stays there until swapoff/swapon. On the DGX Spark GB10 (2026-07-09): swap 16322/16383 MB = 99.6% occupancy, BUT `vmstat` si=0 so=0 (nothing being swapped) and /proc/pressure/memory shows `full avg10=0.00 avg60=0.00` (ZERO memory stall), with 26 GB RAM available. Since #88 (0.37.0) the policy SHEDS with 429 server_busy instead of degrading, so every cortex/senses request 429s while `X-Lobes-Override: 1` still returns 200. The rig had been fine on gateway 0.36.0 (pre-shed) and broke the instant it was upgraded to 0.39.0 \u2014 the wire had been armed for weeks. WORKAROUND (documented, but it DISABLES the policy rather than fixing it): set LOBES_SWAP_DEGRADED_THRESHOLD=100 and LOBES_IOWAIT_DEGRADED_THRESHOLD=100 in the deployment .env and recreate the gateway. Note LOBES_IOWAIT_DEGRADED_THRESHOLD=100 was ALREADY set by a previous operator for the same class of reason (0.36.2 CHANGELOG documents \"phantom high iowait on an idle disk, e.g. the DGX Spark GB10\") \u2014 half the policy had already been switched off, which is the tell that the METRIC is wrong, not the threshold. RIGHT FIX: read PSI (/proc/pressure/memory, /proc/pressure/io avg10) or swap RATE (pswpin/pswpout deltas from /proc/vmstat), not occupancy; keep occupancy at most as a secondary AND-condition. Also: a gateway that has shed 100% of requests for an hour is misconfigured, not busy \u2014 nothing currently warns.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "pressure-policy", "relates_to": "issues-100-85-86-88", "date": "2026-07-09"}, "created": "2026-07-09T09:38:15.369909+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}
{"id": "lobes-fleet-frozen-image-pin-2026-07-09", "hash": "08f35f73d1ad1f81fc3dd0a6d348a2d8f7f6c2ef4581caa92dfba6c7a56b14c5", "content": "lobes-cli fleet deployments FREEZE their lobes version and nothing bumps it (issue #99). Dockerfile.gateway/realtime/stt/chatterbox all run: pip install --no-cache-dir lobes-cli==${MODEL_GEAR_VERSION}. `lobes init` writes MODEL_GEAR_VERSION into .env ONCE at scaffold time and no verb ever rewrites it, so `docker compose up -d --build gateway` faithfully reinstalls the OLD pinned version. On 2026-07-09 the reference rig ran THREE lobes versions at once: gateway 0.36.0 (image built 07-03), realtime 0.34.1 (07-02), stt/chatterbox 0.34.1-era (07-02) \u2014 against a CLI at 0.39.0, with 0.36.1/0.36.2/0.37.0/0.38.0/0.39.0 ALL published on PyPI. CONSEQUENCE: three bugs were reported against lobes CODE that were merely undeployed fixes. (a) #92 'advertised endpoint :8000 404s' \u2014 reachable_origin shipped in 0.38.0/PR#90 on 07-04; the gateway image predated it by one day, so it fell back to _gateway_base_url() = its INTERNAL container port 8000 while the published host port is 8001. Worse, host :8000 was an unrelated uvicorn service (reachy-mini-dae), so the advertised endpoint pointed at a foreign daemon. (b) the stt sidecar answered /v1/health/ready with 200 status=ready while its CUDA context was poisoned and transcription 500d \u2014 #89 (07-04) had replaced the old liveness-only handler with a model_loaded+cuda_ok probe, but the stt image was built 07-02. (c) chatterbox likewise. CHEAP FIX (no model reload): edit ~/.lobes/.env MODEL_GEAR_VERSION=<cli version>, then from ~/.lobes run: docker compose -f docker-compose.yml -f docker-compose.audio.yml up -d --build --no-deps gateway realtime stt chatterbox. GOTCHA 1: the compose PROJECT is named lobes (from the dir basename) while containers are named model-gear-* via container_name \u2014 check the com.docker.compose.project label before invoking compose or you create a duplicate project. GOTCHA 2: AUDIO_URL reaches the gateway ONLY via docker-compose.audio.yml, so a gateway recreated without the -f audio overlay silently loses the audio lane (issue #96). GOTCHA 3: a from-source checkout can be AHEAD of PyPI, in which case the gateway image cannot be built from the pin at all \u2014 the dev path needs a locally built wheel COPYd in.", "scope": {"name": "lobes", "visibility": "public"}, "metadata": {"type": "gotcha", "record_metadata": {"source": "claude-code-session", "topic": "fleet-version-pin", "relates_to": "issues-99-92-89-96", "date": "2026-07-09"}, "created": "2026-07-09T09:38:41.844728+00:00", "last_recall": null, "recall_count": 0, "links": [], "supersedes": null, "lifecycle": "active", "added_by": null}}