|
| 1 | +# Fleet: two models behind one OpenAI gateway |
| 2 | + |
| 3 | +The **fleet** runs two always-warm vLLM models behind a single stdlib |
| 4 | +OpenAI-compatible gateway, managed by model-gear as three Docker containers. It is |
| 5 | +an alternative to the single-model deployment — scaffold it with |
| 6 | +`model init --fleet` (the single-model `model init` is unchanged and remains the |
| 7 | +default). |
| 8 | + |
| 9 | +## Why |
| 10 | + |
| 11 | +The single-model deployment serves one model on `:8000` and `model switch` swaps |
| 12 | +it (freeing the prior model). The fleet instead keeps **both** models loaded and |
| 13 | +puts one OpenAI endpoint in front of them, so: |
| 14 | + |
| 15 | +- existing clients (the acp `vllm-local` provider, `curl`, …) keep pointing at |
| 16 | + `:8000` and keep working — an unknown/missing `model` defaults to the primary; |
| 17 | +- a second model is addressable by name in the same `/v1/...` calls; |
| 18 | +- if the chosen backend is down, the gateway fails over to the other one. |
| 19 | + |
| 20 | +On the DGX Spark (GB10, 128 GB unified memory) both ~30B-class NVFP4 models fit at |
| 21 | +once; the fleet pairs the dense primary with an **MoE** fallback (`A3B` ≈ 3B active |
| 22 | +params) that decodes much faster, so the fast model stays fast. |
| 23 | + |
| 24 | +## Topology |
| 25 | + |
| 26 | +```text |
| 27 | +client / acp ──:8000──▶ model-gear-gateway (python -m model_gear.gateway) |
| 28 | + │ route by `model` → default → failover |
| 29 | + ├──▶ model-gear-vllm-primary :8000 (internal) |
| 30 | + └──▶ model-gear-vllm-fallback :8000 (internal) |
| 31 | +``` |
| 32 | + |
| 33 | +Three containers, all `restart: unless-stopped`: |
| 34 | + |
| 35 | +| Container | Role | Host port | |
| 36 | +|---|---|---| |
| 37 | +| `model-gear-gateway` | stdlib reverse proxy (the single OpenAI front) | `${VLLM_PORT:-8000}` | |
| 38 | +| `model-gear-vllm-primary` | primary model (default: `nvidia/Qwen3-32B-NVFP4`) | internal only | |
| 39 | +| `model-gear-vllm-fallback` | MoE fallback (default: `mmangkad/Qwen3.6-35B-A3B-NVFP4`) | internal only | |
| 40 | + |
| 41 | +The backends are reachable only on the compose network |
| 42 | +(`http://vllm-primary:8000`, `http://vllm-fallback:8000`); only the gateway is |
| 43 | +published to the host. The gateway needs no Docker socket access — compose owns |
| 44 | +the lifecycle; the gateway only routes. |
| 45 | + |
| 46 | +## The gateway |
| 47 | + |
| 48 | +A pure-stdlib (`http.server` + `http.client`, no third-party deps) reverse proxy: |
| 49 | + |
| 50 | +- **Name routing** — a request's `model` routes to the backend that serves it, |
| 51 | + plus any `GATEWAY_ALIASES`. The forwarded body's `model` is rewritten to the |
| 52 | + backend's `--served-model-name` so the backend accepts aliased/default routes. |
| 53 | +- **Default model** — a missing or unknown `model` routes to |
| 54 | + `GATEWAY_DEFAULT_MODEL` (the primary). |
| 55 | +- **Failover** — if the chosen backend refuses the connection or returns a 5xx |
| 56 | + **before any response body**, the request is retried against the other backend. |
| 57 | + A 4xx is a client error (returned verbatim, no failover). Once a 2xx body starts |
| 58 | + streaming there is no retry — the client already has bytes. |
| 59 | +- **Streaming** — `"stream": true` (SSE) is relayed chunk-by-chunk with per-chunk |
| 60 | + flushing; normal JSON is buffered with `Content-Length`. |
| 61 | +- **Endpoints** — `/v1/chat/completions`, `/v1/completions`, `/v1/embeddings` |
| 62 | + (proxied), `/v1/models` (lists both backends), `/health` (gateway liveness). |
| 63 | + |
| 64 | +The gateway image is built from the scaffolded `Dockerfile.gateway` |
| 65 | +(`pip install model-gear==${MODEL_GEAR_VERSION}`, as a non-root user); `model init |
| 66 | +--fleet` pins `MODEL_GEAR_VERSION` to the running model-gear release. The version |
| 67 | +is required (pinning keeps the image reproducible); from-source/dev boxes that run |
| 68 | +ahead of a PyPI release point `MODEL_GEAR_VERSION` at a published TestPyPI `.devN` |
| 69 | +build. |
| 70 | + |
| 71 | +## Verbs |
| 72 | + |
| 73 | +```bash |
| 74 | +model init --fleet --apply # scaffold compose + .env + Dockerfile.gateway |
| 75 | +model fleet up --apply # docker compose up -d --build, wait for gateway /health |
| 76 | +model fleet status # each container's state + gateway /health + /v1/models |
| 77 | +model fleet down --apply # docker compose down |
| 78 | +``` |
| 79 | + |
| 80 | +`model fleet up` / `down` are **dry-run by default**; pass `--apply` to commit. |
| 81 | +`--compose-dir` overrides the deployment dir (default `$MODEL_GEAR_DIR` or |
| 82 | +`~/.model-gear`). `model fleet status` is read-only. |
| 83 | + |
| 84 | +**`model switch` does not drive the fleet** — it rewrites the single-model |
| 85 | +`VLLM_*` keys. Change fleet models by editing the fleet `.env` |
| 86 | +(`PRIMARY_MODEL` / `FALLBACK_MODEL` and their `*_SERVED_NAME` / `*_GPU_MEM_UTIL` |
| 87 | +/ `*_TOOL_CALL_PARSER` / `*_QUANTIZATION`) and re-running `model fleet up --apply`. |
| 88 | + |
| 89 | +## Memory (both warm) |
| 90 | + |
| 91 | +Both models stay resident, so `PRIMARY_GPU_MEM_UTIL` + `FALLBACK_GPU_MEM_UTIL` |
| 92 | +must sum well under 1.0 of the 128 GB. The scaffolded defaults are **0.40** + |
| 93 | +**0.35** (≈ 96 GB reserved, leaving headroom for the OS and KV growth). These are |
| 94 | +estimates for a dense 32B + a 35B-A3B MoE — **validate live** (watch `nvidia-smi` |
| 95 | +at `model fleet up`; OOM is the top operational risk) and tune the two values. |
| 96 | + |
| 97 | +Note the throughput trade-off: decode is memory-bandwidth bound and the bandwidth |
| 98 | +(~273 GB/s) is **shared**. The MoE reads only its active experts per token, so it |
| 99 | +stays fast; two backends decoding *simultaneously* split the bandwidth. The |
| 100 | +gateway routes one request to one backend, so a single client sees full speed. |
| 101 | + |
| 102 | +## Coherence with the single-model verbs |
| 103 | + |
| 104 | +The fleet `.env` mirrors `VLLM_MODEL` / `VLLM_SERVED_NAME` / `VLLM_TOOL_CALL_PARSER` |
| 105 | +(= the primary's) so the read-only single-model verbs (`model status`, |
| 106 | +`model whoami`, `model doctor`'s `env_coherence` check) stay sensible on a fleet |
| 107 | +deployment. `culture.yaml` needs no change: its `model: vllm-local/nvidia/Qwen3-32B-NVFP4` |
| 108 | +resolves through the gateway on `:8000` as the default. |
0 commit comments