Custom vLLM image for the Gemma 4 12B gemma4_unified gear (#71) (v0.33.0)

[OriNachum]

Resolves the core of #71 and corrects the multimodal gear's config from live validation on the DGX Spark (spark-f8a9, GB10). Specced/planned via /think → /spec-to-plan; wave-0 scaffold fanned out via /assign-to-workforce; live waves run on the Spark.

What #71 asked, and what's delivered

#71 asked for a custom vLLM image whose Transformers registers gemma4_unified so the multimodal gear can load. Done and verified live:

• Dockerfile.vllm-gemma4 — FROM nvcr.io/nvidia/vllm:26.06-py3 (vLLM 0.22.1, the serve-proven version; keeps NGC's Blackwell torch 2.13.0a0) + a uv-installed, pinned from-source Transformers (181beb3, 5.13.0.dev0). Build-time, network-free verify asserts gemma4_unified ∈ CONFIG_MAPPING, import vllm, and a Gemma4 arch in ModelRegistry. Confirmed: AutoConfig.from_pretrained(checkpoint) → model_type: gemma4_unified; weights load.
• Compose wiring — vllm-multimodal builds the image (build:) with a MULTIMODAL_IMAGE override (local build by default; optional ghcr.io/local-registry tag). primary/embed/rerank stay on 26.04.

Corrections from live validation

• Quantization modelopt_fp4 → compressed-tensors (the checkpoint's actual quant_method; modelopt_fp4 fails with a method mismatch).
• Removed the gemma4_mtp --speculative-config — vLLM 0.21/0.22 enable Gemma4 MTP only via a separate gemma4_assistant draft the checkpoint doesn't expose (gemma4_mtp is rejected).
• VLLM_ATTENTION_BACKEND=TRITON_ATTN added — Gemma 4's non-square attention (global_head_dim 512 ≠ head_dim 256) needs it.
• Native context confirmed 128K (text_config.max_position_embeddings=131072).

Status: still configured (serve-enablement deferred)

The gear loads but does not yet serve: it crashes at warmup with Shape mismatch: a.size(1)=4096, size_k=8192 (o_proj) because vLLM runs gemma4_unified via its transformers-modeling backend, which didn't honor TRITON_ATTN in our runs. This is well-characterized and split into follow-ups:

• Gemma4 serve-enablement: force TRITON_ATTN on vLLM's transformers-modeling backend #73 — force TRITON_ATTN on the transformers-modeling backend (the serve blocker)
• Validate coolthor/gemma-4-12B-it-NVFP4A16 (blog-proven) and consider it as the multimodal default #74 — validate the blog-proven coolthor/gemma-4-12B-it-NVFP4A16; maybe switch the default
• Gemma4 native MTP: source/build a gemma4_assistant draft model for speculative decoding #75 — source/build a gemma4_assistant draft for native MTP

docs/gemma-4-12b-nvfp4.md carries the full writeup (runtime matrix, root cause, resolved-vs-open).

Tests / version

• 825 passed, 6 skipped; lint (black/isort/flake8) clean; rubric gate (afi cli doctor --strict) passes.
• Tests updated to the corrected reality (quant, no spec-config, TRITON_ATTN, 26.06 base) + a new fleet assertion for the Triton backend.
• Version 0.32.1 → 0.33.0; CHANGELOG updated.

— lobes (Claude)

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add custom vLLM image + fleet wiring for Gemma4 gemma4_unified multimodal gear

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Add a custom NGC-based vLLM image that registers gemma4_unified via pinned Transformers.
• Wire vllm-multimodal to build locally by default, with MULTIMODAL_IMAGE override.
• Correct multimodal runtime flags (quantization, speculative decode removal, Triton attention env)
 from live validation.

Diagram

graph TD
A["Operator / CI"] --> B["fleet/docker-compose.yml"] --> C["vllm-multimodal service"] --> D["vLLM server"]
B --> E["Dockerfile.vllm-gemma4"] --> F["Transformers @181beb3"] --> C
D --> G["Gemma4 multimodal gear"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Wait for an NGC vLLM release that bundles `gemma4_unified`

• ➕ Avoids maintaining a pinned-from-source Transformers overlay
• ➕ Reduces divergence from vendor base images
• ➖ Blocks bringing the multimodal gear up in the near term
• ➖ No guarantee on release timing for required Transformers/vLLM combo

2. Build vLLM + Transformers fully from source (no NGC base)

• ➕ Maximum control over attention backend behavior and feature flags
• ➕ Can track upstream fixes quickly (e.g., transformers-backend TRITON forcing)
• ➖ Higher maintenance and larger security/compat surface (CUDA/torch toolchain)
• ➖ Loses the proven NGC Blackwell torch stack unless carefully replicated

3. Use a prebuilt Transformers wheel/nightly instead of git ref

• ➕ Faster builds and simpler provenance than git+https installs
• ➕ Potentially less brittle than pinning to a commit SHA
• ➖ May not exist for the needed commit window; can drift unexpectedly
• ➖ Harder to ensure exact behavior parity across rebuilds

Recommendation: Current approach (NGC vLLM base + pinned Transformers overlay + static build-time assertions + compose override) is the best tradeoff for near-term operability and reproducibility. Keep the pin strategy, but consider a follow-up to make lobes fleet up optionally skip --build to better support MULTIMODAL_IMAGE registry workflows, and to address the remaining serve blocker by ensuring TRITON attention is honored on the transformers-modeling backend.

Files changed (20) +1300 / -107

Enhancement (2) +62 / -0

_compose.pyInclude Gemma4 custom Dockerfile in fleet template mapping +4/-0

Include Gemma4 custom Dockerfile in fleet template mapping

• Adds Dockerfile.vllm-gemma4 to the runtime compose template set so 'lobes init/fleet' scaffolding includes it.

lobes/runtime/_compose.py

Dockerfile.vllm-gemma4Add custom NGC vLLM image with pinned Transformers registering gemma4_unified +58/-0

Add custom NGC vLLM image with pinned Transformers registering gemma4_unified

• Introduces a Dockerfile based on nvcr.io/nvidia/vllm:26.06-py3 that overlays a pinned from-source Transformers ref via uv and performs network-free verification of gemma4_unified registration and vLLM importability.

lobes/templates/fleet/Dockerfile.vllm-gemma4

Bug fix (1) +27 / -19

catalog.pyCorrect Gemma4 multimodal catalog entry (quantization, no MTP, serving notes) +27/-19

Correct Gemma4 multimodal catalog entry (quantization, no MTP, serving notes)

• Updates the Gemma4 multimodal SupportedModel metadata to use compressed-tensors, removes speculative_config, and clarifies configured status due to serve blocker and TRITON attention requirement.

lobes/catalog.py

Tests (5) +329 / -18

test_catalog.pyUpdate catalog tests for Gemma4 quantization and lack of speculative_config +15/-12

Update catalog tests for Gemma4 quantization and lack of speculative_config

• Adjusts assertions to require compressed-tensors quantization and an empty speculative_config for the Gemma4 multimodal entry.

tests/test_catalog.py

test_cli_fleet.pyUpdate fleet compose tests for no spec-decode and TRITON attention env +18/-6

Update fleet compose tests for no spec-decode and TRITON attention env

• Renames/updates the multimodal compose expectations to ensure no --speculative-config and to assert VLLM_ATTENTION_BACKEND=TRITON_ATTN is present.

tests/test_cli_fleet.py

test_gemma4_compose.pyAdd tests for multimodal custom build block and MULTIMODAL_IMAGE override +132/-0

Add tests for multimodal custom build block and MULTIMODAL_IMAGE override

• Adds static YAML-based tests ensuring vllm-multimodal has build+image override, other vLLM services remain on the stock 26.04 image, and the audio overlay compose is untouched.

tests/test_gemma4_compose.py

test_gemma4_dockerfile.pyAdd static Dockerfile lint/tests for Dockerfile.vllm-gemma4 +163/-0

Add static Dockerfile lint/tests for Dockerfile.vllm-gemma4

• Adds content-based tests verifying the Dockerfile base image, uv-based Transformers install, gemma4_unified verification, and a guard that prevents invalid multi-line RUN bodies.

tests/test_gemma4_dockerfile.py

test_init.pyEnsure fleet init scaffolds Dockerfile.vllm-gemma4 +1/-0

Ensure fleet init scaffolds Dockerfile.vllm-gemma4

• Extends the init dry-run expectations so the new Dockerfile is included in generated fleet outputs.

tests/test_init.py

Documentation (9) +811 / -57

currentPoint devague “current” pointer to the new custom-image frame +1/-1

Point devague “current” pointer to the new custom-image frame

• Updates the active devague frame slug to the Gemma4 custom vLLM image work.

.devague/current

current_planPoint devague “current_plan” pointer to the new custom-image plan +1/-1

Point devague “current_plan” pointer to the new custom-image plan

• Updates the active devague plan slug to match the Gemma4 custom image plan.

.devague/current_plan

lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.jsonAdd devague frame for Gemma4 custom vLLM image spec +282/-0

Add devague frame for Gemma4 custom vLLM image spec

• Introduces an exported devague frame capturing goals, requirements, risks, and scope boundaries for issue #71.

.devague/frames/lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.json

lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.jsonAdd devague plan JSON for the Gemma4 custom image work +281/-0

Add devague plan JSON for the Gemma4 custom image work

• Adds an exported devague plan with task breakdown (Dockerfile, compose wiring, live validation, promotion rules).

.devague/plans/lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.json

lobes__public.jsonlRecord Gemma4 custom-image live-validation findings in eidetic memory +9/-5

Record Gemma4 custom-image live-validation findings in eidetic memory

• Appends/updates reference entries documenting the validated Transformers pin, quantization mismatch, speculative decode constraints, and attention backend root cause.

.eidetic/memory/lobes__public.jsonl

CHANGELOG.mdAdd 0.33.0 release notes for Gemma4 custom image + config corrections +17/-0

Add 0.33.0 release notes for Gemma4 custom image + config corrections

• Documents the new custom Dockerfile, MULTIMODAL_IMAGE override, and corrected multimodal quantization/spec-decode/attention settings.

CHANGELOG.md

gemma-4-12b-nvfp4.mdUpdate Gemma4 multimodal doc with validated runtime constraints and status +74/-50

Update Gemma4 multimodal doc with validated runtime constraints and status

• Corrects quantization to compressed-tensors, removes native MTP claims, documents TRITON_ATTN requirement and current serve blocker, and expands the live-validation matrix.

docs/gemma-4-12b-nvfp4.md

2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.mdAdd exported build plan markdown for issue #71 +71/-0

Add exported build plan markdown for issue #71

• Adds the human-readable plan and acceptance criteria for the custom image, compose wiring, and conditional promotion workflow.

docs/plans/2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.md

2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.mdAdd exported spec markdown for issue #71 +75/-0

Add exported spec markdown for issue #71

• Adds the human-readable spec capturing before/after, requirements, boundaries, and accepted risks for the Gemma4 custom image effort.

docs/specs/2026-06-30-lobes-ships-a-custom-vllm-image-from-nvcr-io-nvidi.md

Other (3) +71 / -13

docker-compose.ymlWire vllm-multimodal to custom build + fix quantization/spec-decode/attention env +32/-6

Wire vllm-multimodal to custom build + fix quantization/spec-decode/attention env

• Switches vllm-multimodal to a build+image pattern with MULTIMODAL_IMAGE override, sets VLLM_ATTENTION_BACKEND default to TRITON_ATTN, corrects quantization default to compressed-tensors, and removes speculative-config for gemma4_mtp.

lobes/templates/fleet/docker-compose.yml

env.exampleDocument MULTIMODAL_IMAGE override and corrected multimodal env defaults +38/-6

Document MULTIMODAL_IMAGE override and corrected multimodal env defaults

• Adds documentation and examples for MULTIMODAL_IMAGE usage, updates MULTIMODAL_QUANTIZATION to compressed-tensors, and introduces MULTIMODAL_ATTENTION_BACKEND with TRITON_ATTN default; clarifies DSpark draft usage requirements.

lobes/templates/fleet/env.example

pyproject.tomlBump project version to 0.33.0 +1/-1

Bump project version to 0.33.0

• Increments the package version from 0.32.1 to 0.33.0 for this release.

pyproject.toml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (4) 📜 Skill insights (0)

Context used

✅ Tickets: 🎫 Custom vLLM image for the Gemma 4 12B multimodal gear (gemma4_unified) — start with nightly Transformers 🎫 Gemma4 native MTP: source/build a gemma4_assistant draft model for speculative decoding 🎫 Validate coolthor/gemma-4-12B-it-NVFP4A16 (blog-proven) and consider it as the multimodal default

✅ Compliance rules (platform): 18 rules

✅ Skills: doc-test-alignment, cicd

1. vllm-multimodal not on registry tag 📎 Requirement gap ☼ Reliability

Description

vllm-multimodal is currently defaulted to a local build tag (lobes/vllm-gemma4:local) and,
because lobes fleet up --apply always runs docker compose up -d --build, setting
MULTIMODAL_IMAGE does not reliably select a pushed fleet-registry image and instead
rebuilds/tag-shadows it. This violates the requirement to publish the custom image to the fleet
registry and configure the service to use that pushed tag via the primary operator workflow.

Code

lobes/templates/fleet/docker-compose.yml[R394-398]

+    build:
+      context: .
+      dockerfile: Dockerfile.vllm-gemma4
+    image: ${MULTIMODAL_IMAGE:-lobes/vllm-gemma4:local}
    container_name: model-gear-vllm-multimodal

Relevance

⭐ Low

Spec pattern: local build default, registry optional override; not a required workflow (PR72).

PR-#72

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1516644 requires pushing the custom image to the fleet registry and configuring
vllm-multimodal to use the new tag, but the updated compose configuration defines
vllm-multimodal with a local-build default (lobes/vllm-gemma4:local) and only treats registry
usage as an optional override. Additionally, the compose defines both build: and image: for
vllm-multimodal, while the fleet CLI path always invokes the build flow (compose_up_build) with
docker compose up -d --build, so even when MULTIMODAL_IMAGE is set, the CLI rebuilds and tags
the resulting image as the configured image: name, preventing a true “use prebuilt registry image”
flow through lobes fleet up.

Publish custom image to fleet registry and point vllm-multimodal service to the new tag
lobes/templates/fleet/docker-compose.yml[394-398]
lobes/templates/fleet/env.example[118-140]
lobes/templates/fleet/docker-compose.yml[378-399]
lobes/cli/_commands/fleet.py[46-66]
lobes/runtime/_compose.py[335-344]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`vllm-multimodal` should be configured to run from a pushed fleet-registry image tag (per compliance), but the current fleet templates and CLI workflow default to local builds (`lobes/vllm-gemma4:local`) and `lobes fleet up --apply` always forces `docker compose up -d --build`, which rebuilds/tag-shadows any `MULTIMODAL_IMAGE` override.

## Issue Context
- Compliance (PR Compliance ID 1516644) requires publishing the custom image to the fleet registry and updating `vllm-multimodal` to use the new tag.
- The compose setup currently includes both `build:` and `image:` for `vllm-multimodal`, and the templates/documentation suggest `MULTIMODAL_IMAGE` can override to a prebuilt image.
- The fleet CLI currently has no “no-build” execution path; it always uses the build path, making a practical “override to a pushed tag and skip local build” workflow impossible unless operators bypass the CLI and run `docker compose` manually.
- Any corrected no-build path must still respect the audio overlay compose files (if scaffolded); note that the existing detached helper does not currently include `_compose_files(deploy_dir)`.

## Fix Focus Areas
- lobes/templates/fleet/docker-compose.yml[383-393]
- lobes/templates/fleet/docker-compose.yml[394-398]
- lobes/templates/fleet/env.example[118-140]
- lobes/templates/fleet/env.example[119-139]
- lobes/cli/_commands/fleet.py[46-66]
- lobes/runtime/_compose.py[331-344]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Multimodal serve not validated 📎 Requirement gap ☼ Reliability

Description

The documentation explicitly states the multimodal gear loads but does not yet serve, meaning
image+text and audio+text serving has not been achieved. This violates the requirement to
successfully serve multimodal requests with vision+audio active.

Code

docs/gemma-4-12b-nvfp4.md[R10-12]

+**Status:** `configured` — the custom image **loads** it, but it does not yet
+**serve** (issue #71 serve-enablement; see ["Live-validation status"](#live-validation-status-71) below)

Relevance

⭐ Low

Team policy: merge recipe even if not serving; keep status configured with documented residual
(PR72).

PR-#72

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1516646 requires successfully serving both image+text and audio+text requests with
vision+audio active. The updated doc states the gear does not yet serve, indicating this requirement
is not met.

Serve multimodal requests with vision+audio active (no --language-model-only)
docs/gemma-4-12b-nvfp4.md[10-12]
lobes/templates/fleet/docker-compose.yml[412-450]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Gemma 4 12B multimodal service is documented as not yet serving, so the required live validation (image+text and audio+text) is not satisfied.

## Issue Context
Compliance requires the deployed service to successfully handle both image+text and audio+text requests while not running in language-only mode.

## Fix Focus Areas
- docs/gemma-4-12b-nvfp4.md[10-12]
- lobes/templates/fleet/docker-compose.yml[412-450]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Catalog not promoted load-tested 📎 Requirement gap ≡ Correctness

Description

The Gemma 4 multimodal catalog entry remains status="configured" rather than being promoted to
load-tested. This violates the requirement to promote the status and record validation numbers
after live validation.

Code

lobes/catalog.py[R232-244]

+        shape="unified multimodal (text+image+audio)",
+        # Native context confirmed 128K (text_config.max_position_embeddings=131072,
+        # read from the checkpoint config during #71 live validation).
        context="128K native",
        native_max_model_len=131072,
        tool_parser="pythonic",
-        quantization="modelopt_fp4",
+        # This checkpoint is NVFP4 in compressed-tensors format (config.json
+        # quant_method="compressed-tensors", format "nvfp4-pack-quantized") — NOT
+        # nvidia modelopt. vLLM must be told --quantization=compressed-tensors;
+        # passing modelopt_fp4 fails with a quant-method-mismatch (verified #71).
+        quantization="compressed-tensors",
        status="configured",
        doc="gemma-4-12b-nvfp4.md",

Relevance

⭐ Low

Promotion to load-tested is explicitly conditional; blockers mean status stays configured (PR72).

PR-#72

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1516650 requires promoting the catalog status from configured to load-tested and
recording validation numbers in docs/gemma-4-12b-nvfp4.md. The catalog entry in this PR remains
status="configured", and the doc section describes unresolved serve-enablement blockers rather
than load-test promotion.

Promote catalog status configured → load-tested and record validation numbers in docs/gemma-4-12b-nvfp4.md
lobes/catalog.py[232-244]
docs/gemma-4-12b-nvfp4.md[143-186]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The multimodal catalog entry is not promoted from `configured` to `load-tested`, and the required validation numbers are not recorded as part of the promotion step.

## Issue Context
Compliance expects promotion to `load-tested` with validation numbers recorded in the referenced doc.

## Fix Focus Areas
- lobes/catalog.py[232-244]
- docs/gemma-4-12b-nvfp4.md[143-186]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View more (1)

4. Dockerfile.vllm-gemma4 wrong base 📎 Requirement gap ≡ Correctness

Description

The custom multimodal vLLM image is built from nvcr.io/nvidia/vllm:26.06-py3 instead of the
compliance-required nvcr.io/nvidia/vllm:26.05.post1-py3, while some operator-facing compose
comments still refer to 26.05.post1-py3. This both violates the mandated base-image requirement
and risks misleading operators about the vLLM/torch baseline when debugging or reproducing issues.

Code

lobes/templates/fleet/Dockerfile.vllm-gemma4[R20-22]

+FROM nvcr.io/nvidia/vllm:26.06-py3
+
+ARG TRANSFORMERS_REF=git+https://github.com/huggingface/transformers.git@181beb3ba4c47098ed8cbc97ee250d1d45ae0107

Relevance

⭐ Low

Similar “required base tag” feedback was rejected; team allows differing image tags when justified
(PR72).

PR-#72

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance ID 1516642 mandates that the custom image be built FROM
nvcr.io/nvidia/vllm:26.05.post1-py3, but the Dockerfile added/updated in this PR explicitly pins
nvcr.io/nvidia/vllm:26.06-py3, demonstrating non-compliance with the required base tag.
Separately, the fleet compose/comment guidance still references 26.05.post1 while the Dockerfile
uses 26.06, which creates an immediate mismatch between repository documentation and the actual
build source of truth.

Build custom vLLM image from nvcr.io/nvidia/vllm:26.05.post1-py3 with Transformers that registers gemma4_unified
lobes/templates/fleet/Dockerfile.vllm-gemma4[20-22]
lobes/templates/fleet/docker-compose.yml[379-382]
lobes/templates/fleet/Dockerfile.vllm-gemma4[20-23]
lobes/runtime/_compose.py[71-80]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The custom vLLM multimodal Dockerfile is required (per compliance) to use the NGC base image tag `nvcr.io/nvidia/vllm:26.05.post1-py3`, but it currently uses `nvcr.io/nvidia/vllm:26.06-py3`, and related operator-facing comments/docs still mention `26.05.post1-py3`, creating both a compliance violation and documentation drift.

## Issue Context
- Compliance (PR Compliance ID 1516642) requires the custom vLLM image to be built from the specific NGC base tag `26.05.post1-py3`.
- The Dockerfile is the source of truth for the image base; any compose/runtime comments or documentation should match the shipped base image tag to avoid misleading ops/debug/repro workflows.

## Fix Focus Areas
- lobes/templates/fleet/Dockerfile.vllm-gemma4[20-23]
- lobes/templates/fleet/docker-compose.yml[379-382]
- lobes/runtime/_compose.py[75-78]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[OriNachum]

Triage of Qodo's 4 requirement gaps

Thanks Qodo. 0 bugs / 0 rule violations; the 4 "requirement gaps" are compliance rules minted from #71's literal acceptance text. One sub-point was a genuine doc-drift bug (fixed); the rest are deliberate, spec-converged decisions or work explicitly split into follow-ups. Also fixed the CI lint failure (markdownlint MD049 in docs/gemma-4-12b-nvfp4.md:62).

Gap 1 — base 26.06 vs required 26.05.post1 → partly fixed, mostly pushback.

• Fixed: the real bug — a stale vllm-multimodal compose comment still said FROM …26.05.post1. Corrected to 26.06 (commit 48506b1), now consistent with the Dockerfile/catalog/env.
• Pushback on the base tag: 26.05.post1 was the issue's starting base; instruction Scaffold lepenseur as a full CLI/PyPI AgentCulture sibling (#1) #3 explicitly allowed "wait for a vLLM/NGC release that bundles gemma4_unified." 26.06-py3 (released 2026-06-29) ships vLLM 0.22.1 — the version proven to serve Gemma 4 Omni — on NGC's Blackwell torch 2.13.0a0, and the transformers overlay keeps that torch. Live validation drove this choice; pinning the older tag would be strictly worse.

Gap 2 — not on a registry tag → pushback (deliberate design). Local compose build: default + an optional MULTIMODAL_IMAGE override is a confirmed decision (the converged spec/plan; Qodo even notes "registry optional override; not a required workflow"). A registry is not required for a single-host Spark. The real sub-point — lobes fleet up forcing --build shadows a registry tag — is documented in env.example (use plain docker compose for a registry tag) and a lobes fleet up --no-build enhancement is noted; tracked alongside #73.

Gap 3 — serve not validated & Gap 4 — not promoted load-tested → pushback (deferred by design). #71's stated goal — "a custom vLLM image whose Transformers registers gemma4_unified" — is met: the image builds and the model loads (verified live). Promotion was always conditional in the converged spec ("any fail → stays configured, residual parked, recipe still merges"). Serving is blocked by Gemma 4's non-square attention (global_head_dim 512 ≠ head_dim 256 → o_proj 4096≠8192), needing TRITON_ATTN to engage on vLLM's transformers-modeling backend — root-caused and split into #73 (serve-enablement), #74 (validate/switch to the blog-proven coolthor checkpoint), #75 (gemma4_assistant MTP draft). status stays configured until #73 lands.

— lobes (Claude)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud