Fallback model + single front OpenAI gateway (3-container fleet) - #16
Conversation
`model init --fleet` scaffolds a three-container deployment — two always-warm vLLM backends (a dense primary + an MoE fallback) behind one stdlib gateway — managed by model-gear as `model-gear-gateway` / `model-gear-vllm-primary` / `model-gear-vllm-fallback`. The gateway fronts both models on the host port acp already expects (8000): it routes each request by its `model` field, defaults an unknown/missing name to the primary, and fails over to the other backend when the chosen one refuses the connection or returns a 5xx before the response body (4xx verbatim; no mid-stream retry). SSE streams are relayed chunk-by-chunk. - New `model_gear/gateway/` package (pure stdlib http.server + http.client, no runtime deps): `_routing.py` (routing/failover order), `_config.py` (env → config), `server.py` (`handle_post` failover seam + upstream client + handler). - New verbs: `model init --fleet` (pins MODEL_GEAR_VERSION to the running release) and `model fleet up | down | status` (up/down dry-run by default). - `_compose.py`: template registry (SINGLE/FLEET) + `templates=` arg (single-model stays default), `compose_up_build`, `FLEET_CONTAINERS`. - Fleet `.env` mirrors VLLM_* (= primary) so status/whoami/doctor stay coherent; `model switch` remains single-model only. - Docs: docs/gateway-fleet.md, docs/qwen3.6-35b-a3b-nvfp4.md, README fleet section, `model explain fleet` / `gateway` entries. - Tests: 36 new (routing/config/body helpers, handle_post failover matrix, loopback relay + chunked streaming, open_upstream, fleet verbs, init --fleet). Full suite 125 passed; coverage 90%. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Review Summary by QodoFallback model + single front OpenAI gateway (3-container fleet)
WalkthroughsDescription• Add fallback model + single front OpenAI gateway (3-container fleet deployment) • New model fleet up|down|status verbs to manage the gateway deployment • New model init --fleet scaffolds 3-container templates with gateway Dockerfile • Pure-stdlib gateway package with routing, failover, and streaming support • Comprehensive docs and 36 new tests covering routing, failover, and integration Diagramflowchart LR
A["Client requests<br/>on :8000"] -->|"model field<br/>routing"| B["model-gear-gateway<br/>stdlib reverse proxy"]
B -->|"primary or<br/>failover"| C["vllm-primary<br/>Qwen3-32B"]
B -->|"fallback or<br/>failover"| D["vllm-fallback<br/>Qwen3.6-35B-A3B"]
E["model init --fleet"] -->|"scaffolds"| F["docker-compose.yml<br/>.env<br/>Dockerfile.gateway"]
G["model fleet up/down/status"] -->|"manages"| B
File Changes1. model_gear/gateway/__init__.py
|
Code Review by Qodo
1. Docs reference ~/.model-gear
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
- Dockerfile.gateway: pin the model-gear install unconditionally
(`==${MODEL_GEAR_VERSION}`, no unpinned "latest" fallback) so the image is
reproducible (docker:S8544), and run as a non-root `gateway` user (clears the
root-user security hotspot). MODEL_GEAR_VERSION is now required; init --fleet
fills it and dev boxes set a TestPyPI .devN.
- fleet.py: restructure cmd_fleet_up / cmd_fleet_down to a single return path
(python:S3516) and hoist the repeated --json / --port help strings into
_JSON_HELP / _PORT_HELP constants (python:S1192).
- Docs/env.example updated for the required pinned version.
The remaining new-code hotspots are the internal-network `http://vllm-*` URLs
(no TLS between sibling compose containers, by design) — reviewed as safe.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_read_body` honored only Content-Length, so a valid HTTP/1.1 `Transfer-Encoding: chunked` request body was forwarded empty (misrouted to the default + invalid JSON upstream). Add a stdlib `read_chunked_body` decoder (ignores chunk extensions, caps total size) and use it when Content-Length is absent but the request is chunked. Adds a unit test for the decoder and a loopback integration test posting a chunked body through the gateway. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`open_upstream` caught only OSError, so a malformed `base_url` — e.g. a non-numeric port (`parts.port` raises ValueError) or a bad host/path (`http.client.InvalidURL`) — propagated as an uncaught 500 instead of failing over. Wrap the parse + connect + request in one guard that maps (OSError, http.client.HTTPException, ValueError) to UpstreamError, so a misconfigured backend fails over (or 502s) cleanly. Adds a test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Thanks for the review — addressed the rest of the findings:
|
The pip-install and useradd both run as root before the USER switch, so there's no reason for two image layers. Combine them with `&&`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
1 participant
What
Adds the two joined features requested: a fallback model and a single front OpenAI API that fronts both, with model-gear managing all three containers.
model init --fleetscaffolds a 3-container deployment — two always-warm vLLM backends (a dense primary + an MoE fallback) behind one stdlib gateway — asmodel-gear-gateway/model-gear-vllm-primary/model-gear-vllm-fallback. The gateway listens on the host port acp already expects (8000) and:modelfield (plusGATEWAY_ALIASES),modelto the primary (so existing single-model clients keep working),/v1/models, and exposes/health.Both models stay loaded (no swap, no queue) — the design follows the confirmed decisions: custom stdlib gateway (not llama-swap/LiteLLM), both warm, default + failover + name routing.
Design decisions (confirmed up front)
http.server+http.client, zero runtime deps)*_GPU_MEM_UTIL)mmangkad/Qwen3.6-35B-A3B-NVFP4(MoE, ~3B active → fast decode)Changes
model_gear/gateway/—_routing.py(pure name/alias/default + failover ordering),_config.py(env → routing table + server config),server.py(thehandle_postfailover seam,open_upstreamhttp.client,ThreadingHTTPServerhandler), run aspython -m model_gear.gateway.model init --fleet(pinsMODEL_GEAR_VERSIONto the running release) andmodel fleet up | down | status(up/downdry-run by default;statusread-only, reports all 3 containers + gateway/health+/v1/models).model_gear/templates/fleet/{docker-compose.yml,env.example,Dockerfile.gateway}. Single-model deployment is unchanged and remains the default (added, not replaced)._compose.py— template registry (SINGLE_TEMPLATES/FLEET_TEMPLATES) +templates=arg (existing callers unchanged),compose_up_build,FLEET_CONTAINERS..envmirrorsVLLM_*(= primary) sostatus/whoami/doctorstay coherent.model switchstays single-model only (documented).docs/gateway-fleet.md,docs/qwen3.6-35b-a3b-nvfp4.md, README fleet section,model explain fleet/model explain gateway.Testing
handle_postfailover matrix (refused → fallback, 5xx → fallback, 4xx → no failover, both down → 502, default/alias routing + body rewrite), a loopback integration covering the handler relay + chunked SSE framing +open_upstream, and the fleet CLI verbs +init --fleet.black/isort/flake8/banditclean;afi cli doctor . --strictpasses; markdownlint clean. Version bumped 0.8.1 → 0.9.0.Not done here (out of band)
Live validation on the DGX Spark —
model fleet up --apply, confirm both backends co-resident without OOM (the0.40/0.35GPU_MEM_UTILsplit is an estimate), and confirm themmangkadcheckpoint's--quantization/--tool-call-parser. The fallback per-model doc marks its benchmark table pending rather than fabricating numbers.🤖 Generated with Claude Code