Fallback model + single front OpenAI gateway (3-container fleet)

CHANGELOG.md

@@ -4,6 +4,48 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.0] - 2026-05-28
+
+### Added
+
+- **Fallback model + single front OpenAI gateway ("fleet").** A new
+  scaffold-based deployment runs **two always-warm vLLM backends behind one
+  stdlib gateway** that model-gear manages as three containers
+  (`model-gear-gateway`, `model-gear-vllm-primary`, `model-gear-vllm-fallback`).
+  The gateway routes each request by its `model` field, defaults an
+  unknown/missing name to the primary, and fails over to the other backend when
+  the chosen one refuses the connection or returns a 5xx **before** the response
+  body (4xx is returned verbatim; no mid-stream retry). SSE streams are relayed
+  chunk-by-chunk. Default fallback: the MoE `mmangkad/Qwen3.6-35B-A3B-NVFP4`.
+- **New gateway package `model_gear/gateway/`** — a pure-stdlib
+  (`http.server` + `http.client`, no runtime deps) reverse proxy: `_routing.py`
+  (pure name/alias/default routing + failover ordering), `_config.py` (env →
+  routing table + server config), `server.py` (the `handle_post` failover seam,
+  upstream client, and `ThreadingHTTPServer` handler), run as
+  `python -m model_gear.gateway`.
+- **`model init --fleet`** scaffolds the fleet templates
+  (`docker-compose.yml` + `.env` + `Dockerfile.gateway`) and pins
+  `MODEL_GEAR_VERSION` to the running release; **`model fleet up | down |
+  status`** drives the deployment (`up`/`down` dry-run by default, `--apply` to
+  commit; `status` is read-only and reports all three containers + the gateway
+  `/health` + `/v1/models`).
+- **Docs:** `docs/gateway-fleet.md` (topology, routing/failover, memory,
+  verbs), `docs/qwen3.6-35b-a3b-nvfp4.md` (the MoE fallback), a README "fleet"
+  section, and `model explain fleet` / `model explain gateway` entries.
+
+### Changed
+
+- `model_gear/runtime/_compose.py` gained a template registry
+  (`SINGLE_TEMPLATES` / `FLEET_TEMPLATES`), a `templates=` argument on
+  `scaffold_plan` / `write_scaffold` (single-model stays the default — existing
+  callers unchanged), a `compose_up_build` helper, and `FLEET_CONTAINERS`.
+- The fleet `.env` mirrors `VLLM_MODEL` / `VLLM_SERVED_NAME` /
+  `VLLM_TOOL_CALL_PARSER` (= the primary) so the read-only single-model verbs
+  (`status` / `whoami` / `doctor`) stay coherent on a fleet deployment.
+  `model switch` remains single-model only.
+
+### Fixed
+
 ## [0.8.1] - 2026-05-27
 
 ### Changed

README.md

@@ -77,6 +77,36 @@ model whose repo ships custom modeling code. If vLLM rejects the `nvidia/`
 ModelOpt checkpoint, set `VLLM_MODEL` to the vLLM-native `RedHatAI/Qwen3-32B-NVFP4`
 and drop `--quantization` from the compose `command`.
 
+## Running two models behind one gateway (fleet)
+
+`model init --fleet` scaffolds a **three-container** deployment instead of one:
+two always-warm vLLM backends (a primary + an MoE fallback) and a single stdlib
+**gateway** that fronts them on the host port the acp `vllm-local` provider
+already expects. The gateway routes each request by its `model` field, defaults an
+unknown/missing name to the primary, and fails over to the other backend if the
+chosen one is down — so existing single-model clients keep working unchanged while
+a second model becomes addressable by name.
+
+```bash
+model init --fleet --apply        # ~/.model-gear/{docker-compose.yml,.env,Dockerfile.gateway}
+docker login nvcr.io              # NGC API key for the vLLM image
+model fleet up --apply            # builds the gateway image + starts all three
+model fleet status                # container states + gateway /health + /v1/models
+```
+
+```bash
+curl -s http://localhost:8000/v1/models       # lists BOTH served models
+# route explicitly by name; an unknown/missing model falls back to the primary
+curl -s http://localhost:8000/v1/chat/completions -d '{"model":"mmangkad/Qwen3.6-35B-A3B-NVFP4","messages":[...]}'
+```
+
+Both models stay loaded, so set `PRIMARY_GPU_MEM_UTIL` + `FALLBACK_GPU_MEM_UTIL`
+in the fleet `.env` to sum well under 1.0 (they share the 128 GB unified memory).
+`model switch` is single-model only — change fleet models by editing the fleet
+`.env` and re-running `model fleet up --apply`. See `model explain fleet` /
+`model explain gateway` for the routing and failover semantics, and
+[`docs/gateway-fleet.md`](docs/gateway-fleet.md) for the full topology.
+
 ### Per-model notes
 
 Each runtime model has a doc under `docs/` recording how to run it, live test
@@ -87,6 +117,9 @@ results, and caveats:
 - [`docs/qwen3.6-27b-nvfp4.md`](docs/qwen3.6-27b-nvfp4.md) — a **candidate**
   (`mmangkad/Qwen3.6-27B-NVFP4`), load-tested on DGX Spark; loads under the
   current vLLM image but is slower on decode, so the 32B stays.
+- [`docs/qwen3.6-35b-a3b-nvfp4.md`](docs/qwen3.6-35b-a3b-nvfp4.md) — the **MoE
+  fallback** (`mmangkad/Qwen3.6-35B-A3B-NVFP4`) the gateway fleet pairs with the
+  32B; ~3B active params decode much faster on this box.
 
 The numbers in each doc come from `model switch <model> --apply` then `model
 assess` (correctness) and `model benchmark` (throughput). `model overview --list`

docs/gateway-fleet.md

@@ -0,0 +1,107 @@
+# Fleet: two models behind one OpenAI gateway
+
+The **fleet** runs two always-warm vLLM models behind a single stdlib
+OpenAI-compatible gateway, managed by model-gear as three Docker containers. It is
+an alternative to the single-model deployment — scaffold it with
+`model init --fleet` (the single-model `model init` is unchanged and remains the
+default).
+
+## Why
+
+The single-model deployment serves one model on `:8000` and `model switch` swaps
+it (freeing the prior model). The fleet instead keeps **both** models loaded and
+puts one OpenAI endpoint in front of them, so:
+
+- existing clients (the acp `vllm-local` provider, `curl`, …) keep pointing at
+  `:8000` and keep working — an unknown/missing `model` defaults to the primary;
+- a second model is addressable by name in the same `/v1/...` calls;
+- if the chosen backend is down, the gateway fails over to the other one.
+
+On the DGX Spark (GB10, 128 GB unified memory) both ~30B-class NVFP4 models fit at
+once; the fleet pairs the dense primary with an **MoE** fallback (`A3B` ≈ 3B active
+params) that decodes much faster, so the fast model stays fast.
+
+## Topology
+
+```text
+client / acp ──:8000──▶ model-gear-gateway   (python -m model_gear.gateway)
+                          │  route by `model` → default → failover
+                          ├──▶ model-gear-vllm-primary   :8000 (internal)
+                          └──▶ model-gear-vllm-fallback  :8000 (internal)
+```
+
+Three containers, all `restart: unless-stopped`:
+
+| Container | Role | Host port |
+|---|---|---|
+| `model-gear-gateway` | stdlib reverse proxy (the single OpenAI front) | `${VLLM_PORT:-8000}` |
+| `model-gear-vllm-primary` | primary model (default: `nvidia/Qwen3-32B-NVFP4`) | internal only |
+| `model-gear-vllm-fallback` | MoE fallback (default: `mmangkad/Qwen3.6-35B-A3B-NVFP4`) | internal only |
+
+The backends are reachable only on the compose network
+(`http://vllm-primary:8000`, `http://vllm-fallback:8000`); only the gateway is
+published to the host. The gateway needs no Docker socket access — compose owns
+the lifecycle; the gateway only routes.
+
+## The gateway
+
+A pure-stdlib (`http.server` + `http.client`, no third-party deps) reverse proxy:
+
+- **Name routing** — a request's `model` routes to the backend that serves it,
+  plus any `GATEWAY_ALIASES`. The forwarded body's `model` is rewritten to the
+  backend's `--served-model-name` so the backend accepts aliased/default routes.
+- **Default model** — a missing or unknown `model` routes to
+  `GATEWAY_DEFAULT_MODEL` (the primary).
+- **Failover** — if the chosen backend refuses the connection or returns a 5xx
+  **before any response body**, the request is retried against the other backend.
+  A 4xx is a client error (returned verbatim, no failover). Once a 2xx body starts
+  streaming there is no retry — the client already has bytes.
+- **Streaming** — `"stream": true` (SSE) is relayed chunk-by-chunk with per-chunk
+  flushing; normal JSON is buffered with `Content-Length`.
+- **Endpoints** — `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`
+  (proxied), `/v1/models` (lists both backends), `/health` (gateway liveness).
+
+The gateway image is built from the scaffolded `Dockerfile.gateway`
+(`pip install model-gear==${MODEL_GEAR_VERSION}`); `model init --fleet` pins
+`MODEL_GEAR_VERSION` to the running model-gear release. From-source/dev boxes that
+run ahead of a PyPI release can point `MODEL_GEAR_VERSION` at a TestPyPI `.devN`
+build (or leave it empty to install the latest).
+
+## Verbs
+
+```bash
+model init --fleet --apply        # scaffold compose + .env + Dockerfile.gateway
+model fleet up --apply            # docker compose up -d --build, wait for gateway /health
+model fleet status                # each container's state + gateway /health + /v1/models
+model fleet down --apply          # docker compose down
+```
+
+`model fleet up` / `down` are **dry-run by default**; pass `--apply` to commit.
+`--compose-dir` overrides the deployment dir (default `$MODEL_GEAR_DIR` or
+`~/.model-gear`). `model fleet status` is read-only.
+
+**`model switch` does not drive the fleet** — it rewrites the single-model
+`VLLM_*` keys. Change fleet models by editing the fleet `.env`
+(`PRIMARY_MODEL` / `FALLBACK_MODEL` and their `*_SERVED_NAME` / `*_GPU_MEM_UTIL`
+/ `*_TOOL_CALL_PARSER` / `*_QUANTIZATION`) and re-running `model fleet up --apply`.
+
+## Memory (both warm)
+
+Both models stay resident, so `PRIMARY_GPU_MEM_UTIL` + `FALLBACK_GPU_MEM_UTIL`
+must sum well under 1.0 of the 128 GB. The scaffolded defaults are **0.40** +
+**0.35** (≈ 96 GB reserved, leaving headroom for the OS and KV growth). These are
+estimates for a dense 32B + a 35B-A3B MoE — **validate live** (watch `nvidia-smi`
+at `model fleet up`; OOM is the top operational risk) and tune the two values.
+
+Note the throughput trade-off: decode is memory-bandwidth bound and the bandwidth
+(~273 GB/s) is **shared**. The MoE reads only its active experts per token, so it
+stays fast; two backends decoding *simultaneously* split the bandwidth. The
+gateway routes one request to one backend, so a single client sees full speed.
+
+## Coherence with the single-model verbs
+
+The fleet `.env` mirrors `VLLM_MODEL` / `VLLM_SERVED_NAME` / `VLLM_TOOL_CALL_PARSER`
+(= the primary's) so the read-only single-model verbs (`model status`,
+`model whoami`, `model doctor`'s `env_coherence` check) stay sensible on a fleet
+deployment. `culture.yaml` needs no change: its `model: vllm-local/nvidia/Qwen3-32B-NVFP4`
+resolves through the gateway on `:8000` as the default.

docs/qwen3.6-35b-a3b-nvfp4.md

@@ -0,0 +1,82 @@
+# Fallback model: `mmangkad/Qwen3.6-35B-A3B-NVFP4`
+
+The **MoE fallback** the gateway fleet pairs with the dense primary
+(`nvidia/Qwen3-32B-NVFP4`). See [`docs/gateway-fleet.md`](gateway-fleet.md) for the
+fleet topology; this doc records what the model is and how it is configured in the
+fleet.
+
+Source: <https://huggingface.co/mmangkad/Qwen3.6-35B-A3B-NVFP4>.
+
+> **Status: configured, not yet load-tested on this hardware.** The numbers below
+> are *expectations* from the architecture, not measured values. Fill in the
+> Benchmark table from a live `model fleet up` → `model assess` / `model benchmark`
+> run (and confirm the quantization/parser caveats) before relying on them.
+
+## What it is
+
+- An **NVFP4 (Mixture-of-Experts)** checkpoint: ~35B total parameters, **~3B
+  active per token** (`A3B`). vLLM loads *all* experts into memory; the small
+  active set only reduces per-token compute.
+- Decode is memory-bandwidth bound on the GB10 (~273 GB/s shared). Reading only
+  ~3B active params per token (≈1.5 GB at 4-bit) gives an **expected decode
+  ceiling far above the dense 32B** (which reads ~18 GB/token) — the reason it is
+  the fast fallback. *Confirm live.*
+
+## How it runs in the fleet
+
+Configured via the `FALLBACK_*` keys in the fleet `.env` (scaffolded by
+`model init --fleet`); served by the `model-gear-vllm-fallback` container:
+
+```dotenv
+FALLBACK_MODEL=mmangkad/Qwen3.6-35B-A3B-NVFP4
+FALLBACK_SERVED_NAME=mmangkad/Qwen3.6-35B-A3B-NVFP4
+FALLBACK_MAX_MODEL_LEN=32768
+FALLBACK_GPU_MEM_UTIL=0.35          # both models warm: keep primary+fallback well under 1.0
+FALLBACK_TOOL_CALL_PARSER=qwen3_coder
+FALLBACK_QUANTIZATION=modelopt_fp4
+```
+
+Address it through the gateway by name (or set `GATEWAY_ALIASES` for a short
+alias):
+
+```bash
+curl -s http://localhost:8000/v1/chat/completions \
+  -d '{"model":"mmangkad/Qwen3.6-35B-A3B-NVFP4","messages":[{"role":"user","content":"hi"}]}'
+```
+
+## Caveats to confirm on first load
+
+1. **Tool-call format.** Qwen3.6 emits the Qwen3-Coder **XML** function format, so
+   the backend is served with `--tool-call-parser=qwen3_coder` (not the `hermes`
+   parser the dense Qwen3-32B uses). `model_gear.runtime._parser.infer_parser`
+   already maps `qwen3.6` → `qwen3_coder`. Verify a `tool_choice:"auto"` probe
+   returns a `finish` tool call.
+2. **Quantization format.** The fleet defaults `FALLBACK_QUANTIZATION=modelopt_fp4`
+   (as for the `nvidia/` checkpoints). This community (`mmangkad`) checkpoint may
+   instead be a compressed-tensors NVFP4 — if vLLM rejects `modelopt_fp4`, drop or
+   change `FALLBACK_QUANTIZATION`.
+3. **`--trust-remote-code`.** The fleet compose omits it (as the single-model
+   template does). If this checkpoint ships custom modeling code, vLLM will say so
+   on load; add it back deliberately (it lets repo code run in-container alongside
+   `HF_TOKEN` and the mounted cache).
+4. **Architecture support.** Confirm the engine registers the checkpoint's
+   architecture, as done for the 27B sibling:
+   `docker exec model-gear-vllm-fallback python3 -c "from
+   vllm.model_executor.models.registry import ModelRegistry;
+   print(ModelRegistry.get_supported_archs())"`.
+
+## Benchmark — pending
+
+Fill from a live run (`model fleet up --apply`, then `model assess` / `model
+benchmark` against `:8000` with this model's name):
+
+| Property | Value |
+|---|---|
+| Health / `max_model_len` | *pending* |
+| Correctness (`17×23`, train 14:45→17:10) | *pending* |
+| Reasoning trace field | *pending* |
+| Tool calling (`tool_choice:auto`, `qwen3_coder`) | *pending* |
+| **Decode throughput** | *pending* (expected ≫ the dense 32B's ~9.7 tok/s) |
+| Prefill (~2K tokens) | *pending* |
+| GPU memory reserved (at `FALLBACK_GPU_MEM_UTIL`) | *pending* |
+| Co-resident total (primary + fallback) | *pending* — watch for OOM |

model_gear/cli/__init__.py

@@ -66,6 +66,7 @@ def _build_parser() -> argparse.ArgumentParser:
     from model_gear.cli._commands import cli as _cli_group
     from model_gear.cli._commands import doctor as _doctor_cmd
     from model_gear.cli._commands import explain as _explain_cmd
+    from model_gear.cli._commands import fleet as _fleet_cmd
     from model_gear.cli._commands import init as _init_cmd
     from model_gear.cli._commands import learn as _learn_cmd
     from model_gear.cli._commands import overview as _overview_cmd
@@ -96,6 +97,7 @@ def _build_parser() -> argparse.ArgumentParser:
     _assess_cmd.register(sub)
     _benchmark_cmd.register(sub)
     _init_cmd.register(sub)
+    _fleet_cmd.register(sub)
 
     # Agent-first / introspection verbs (sibling rubric).
     _whoami_cmd.register(sub)