feat(overview): live fleet dashboard — online / offered / busy + usage + endpoints

[OriNachum]

What

model overview was a static description of the tool. model overview --live now
probes the running deployment and answers the five "what is the fleet doing
right now" questions:

• online — per-backend health
• offered — served + candidate models, task families, the endpoint list
• busy — in-flight / queued requests (vLLM num_requests_running/waiting)
• usage — cumulative prompt/generation tokens + finished requests by reason
• endpoints

It is read-only and HTTP-only, so it works against a local deployment or a
model tunnel hostname alike, and degrades gracefully when a backend or its
metrics is unreachable.

How

The fleet's backends are internal-only (only the gateway port is published), so
the host CLI can't reach them directly. The gateway therefore grows a
model-gear-native GET /status that fans out to each backend's /health +
/metrics and returns one JSON aggregate:

{"object":"model-gear.fleet_status","default_model":"…","busy":{"running":1,"waiting":0},
 "backends":[{"name":"primary","task":"generate","served_name":"…","health":"ok","metrics":{…}}],
 "endpoints":[…]}

A bare single-model server has no /status, so --live reads it directly from its
/metrics + /health. New stdlib model_gear._metrics parses vLLM's
Prometheus exposition (running/waiting, prompt/generation tokens,
request_success_total by finish reason, KV-cache usage) with best-effort HTTP
probes that never raise.

Pure section builders + an injected probe seam → fully unit-tested without sockets.

# model-gear (live)
## Online (live)
- sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP on :8001 — ok
## Busy
- running: 0    waiting: 0
## Usage
- prompt tokens: 715,833    generation tokens: 52,413
- requests succeeded: 42  (stop=42)

Tests / validation

• 341 tests pass; new tests/test_overview_live.py (metrics parsing, section
  builders, live_sections fleet/single/nothing discrimination, the CLI verb) +
  tests/test_gateway_status.py (the /status fan-out + endpoint-by-task-family,
  unreachable backends).
• Verified live against the running :8001 backend — real token/usage counts
  above came straight from it (read-only /metrics + /health; no tool calls, no
  disruption to the mesh).
• black / isort / flake8 / bandit / markdownlint / rubric-gate (26/26) all clean.

Builds on #51 (durable logs), now merged.

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Add live fleet dashboard to model overview via gateway /status
✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Description

• Add model overview --live to probe a running deployment and render live sections.
• Add gateway GET /status that fans out to backend /health + /metrics.
• Introduce stdlib vLLM Prometheus parsing + full unit-test coverage for live views.

Diagram

graph TD
  A(["model overview --live"]) --> B["model_gear.cli._live"] --> H["Rendered sections"]
  B --> C["model_gear._metrics"] --> D["GET /status"] --> E["Gateway server"] --> F(("Backends: /health + /metrics"))
  C --> G["GET /health + /metrics"] --> F

  subgraph Legend
    direction LR
    _cli(["CLI entrypoint"]) ~~~ _mod["Python module"] ~~~ _svc["HTTP service"] ~~~ _ext(("Backend service"))
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Gateway proxy for backend `/metrics` (per-backend passthrough)

• ➕ Avoids defining a new aggregate schema; reuse raw Prometheus output
• ➕ CLI could query per-backend metrics on demand
• ➖ More surface area (multiple proxy endpoints) and more parsing burden on the CLI
• ➖ Harder to keep output stable and high-level across different backends/versions

2. Use a Prometheus text parsing library

• ➕ More robust parsing for edge cases and escaping rules
• ➕ Less custom parsing code to maintain
• ➖ Introduces a third-party dependency (PR explicitly prefers stdlib-only)
• ➖ More packaging/compat risk for a small, targeted metric subset

3. Local-only Docker/Compose introspection for fleet backends

• ➕ No new HTTP endpoint; can directly reach internal backend networks locally
• ➕ Potentially richer information (container status, logs, etc.)
• ➖ Does not work over model tunnel/remote hostnames (breaks stated goal)
• ➖ Adds coupling to Docker availability and compose layouts

Recommendation: Keep the PR’s approach: a single gateway-owned GET /status aggregate plus a stdlib-only, best-effort metrics reducer. It’s the smallest API surface that works both locally and over tunnels, avoids adding dependencies, and centralizes fleet-only visibility (internal backends) where it belongs. The injected probe seam and unit tests mitigate the primary risk (partial/unreachable scrapes).

Files changed (9) +677 / -5

Enhancement (4) +379 / -3

_metrics.pyAdd stdlib vLLM metrics parser and best-effort HTTP probes +128/-0

Add stdlib vLLM metrics parser and best-effort HTTP probes

• Introduces a small Prometheus exposition reducer for the vLLM series used by the live dashboard (running/waiting, token totals, success by finish reason, KV cache). Adds HTTP GET helpers and a 'probe_backend()' function that returns structured results and never raises.

model_gear/_metrics.py

overview.pyAdd '--live' mode to overview CLI with port/compose-dir support +39/-2

Add '--live' mode to overview CLI with port/compose-dir support

• Adds '--live' (plus '--port' and '--compose-dir') to run a live probe flow instead of static sections. Includes best-effort served-name resolution from deployment '.env' to label single-server output consistently.

model_gear/cli/_commands/overview.py

_live.pyImplement live section builders for fleet and single-server modes +148/-0

Implement live section builders for fleet and single-server modes

• Adds pure section builders for the five live views (Online/Offered/Busy/Usage/Endpoints) and a thin 'live_sections()' probe wrapper. Detects fleet via gateway '/status', falls back to direct '/health' + '/metrics', and degrades to a friendly 'nothing serving' section when unreachable.

model_gear/cli/_live.py

server.pyExpose 'GET /status' fleet aggregate and endpoint enumeration +64/-1

Expose 'GET /status' fleet aggregate and endpoint enumeration

• Adds 'GET /status' to the gateway handler and implements an aggregator that probes each backend’s '/health' and '/metrics', sums busy counts, and returns a single JSON payload including an endpoint list based on task families and audio enablement. Probe function is injectable for socket-free tests.

model_gear/gateway/server.py

Tests (2) +253 / -0

test_gateway_status.pyAdd unit tests for gateway '/status' aggregation behavior +88/-0

Add unit tests for gateway '/status' aggregation behavior

• Validates endpoint enumeration by task family/audio toggle and verifies busy/health aggregation when some or all backends are unreachable, using an injected fake probe.

tests/test_gateway_status.py

test_overview_live.pyAdd unit tests for metrics parsing and live overview CLI output +165/-0

Add unit tests for metrics parsing and live overview CLI output

• Covers vLLM metrics parsing edge cases, fleet/single/nothing discrimination in 'live_sections()', and CLI rendering in both text and JSON modes via monkeypatched HTTP helpers.

tests/test_overview_live.py

Documentation (2) +44 / -1

CHANGELOG.mdDocument v0.24.0 live overview and gateway '/status' +27/-0

Document v0.24.0 live overview and gateway '/status'

• Adds a 0.24.0 changelog entry describing the new 'model overview --live' dashboard, the gateway 'GET /status' aggregate, and the new '_metrics' helper.

CHANGELOG.md

gateway-fleet.mdDescribe gateway '/status' and 'model overview --live' usage +17/-1

Describe gateway '/status' and 'model overview --live' usage

• Extends gateway docs to include the new '/status' endpoint and explains how 'model overview --live' derives its output for fleet vs single-server deployments.

docs/gateway-fleet.md

Other (1) +1 / -1

pyproject.tomlBump package version to 0.24.0 +1/-1

Bump package version to 0.24.0

• Updates the project version to reflect the new live overview feature release.

pyproject.toml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 35 rules

✅ Skills: 4 invoked

  sonarclaude

  version-bump

  cicd

  doc-test-alignment

1. Non-finite metrics can crash ✓ Resolved 🐞 Bug ☼ Reliability

Description

parse_metrics() can parse non-finite float values (e.g., NaN/inf) via float(value) and then raises
when converting accumulated values to int(), which can break gateway /status and overview --live
unexpectedly. This contradicts the intended “best-effort / degrades gracefully” behavior of the live
dashboard pipeline.

Code

model_gear/_metrics.py[R52-83]

+        try:
+            left, value = line.rsplit(" ", 1)
+            val = float(value)
+        except ValueError:
+            continue
+        brace = left.find("{")
+        name = left[:brace] if brace >= 0 else left
+        labels = left[brace:] if brace >= 0 else ""
+        if name == _RUNNING:
+            running += val
+        elif name == _WAITING:
+            waiting += val
+        elif name == _KV:
+            kv = val if kv is None else max(kv, val)
+        elif name == _PROMPT_TOK:
+            prompt_tok += val
+        elif name == _GEN_TOK:
+            gen_tok += val
+        elif name == _SUCCESS:
+            reason = _label(labels, "finished_reason") or "?"
+            by_reason[reason] = by_reason.get(reason, 0.0) + val
+    out = {
+        "running": int(running),
+        "waiting": int(waiting),
+        "prompt_tokens": int(prompt_tok),
+        "generation_tokens": int(gen_tok),
+        "requests_succeeded": int(sum(by_reason.values())),
+        "by_finish_reason": {k: int(v) for k, v in by_reason.items() if v},
+    }
+    if kv is not None:
+        out["kv_cache_usage"] = round(kv, 3)
+    return out

Relevance

⭐⭐⭐ High

Team enforces “never raises” best-effort probes; similar defensive parsing accepted in PR #14.

PR-#14

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The code parses values as floats and then converts them to ints without guarding against
ValueError/OverflowError on the final casts, so certain float inputs can raise and break the
live path.

model_gear/_metrics.py[48-83]
PR-#14

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`parse_metrics()` can throw during the final `int(...)` conversions if any parsed value becomes non-finite (`nan`/`inf`). That exception can bubble up through `probe_backend()` and break `/status` or the CLI live view, defeating the “best-effort” contract.

## Issue Context
The parser accepts `val = float(value)` and later does `int(running)`, `int(prompt_tok)`, etc. Non-finite floats are not filtered and will cause `int()` to raise.

## Fix Focus Areas
- model_gear/_metrics.py[38-83]

## Suggested fix
- Import `math` and skip values where `not math.isfinite(val)`.
- Alternatively (or additionally), wrap the final `int(...)` conversions in a `try/except (ValueError, OverflowError)` and fall back to 0 for that field.
- Ensure `kv_cache_usage` is only emitted when finite, and avoid propagating `nan` into JSON.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Unbounded metrics read ✓ Resolved 🐞 Bug ☼ Reliability

Description

http_get_text() reads the entire response body with r.read() and has no maximum size, so a large
/metrics response can cause high memory usage and slow the gateway/CLI live probes. This affects
both gateway /status fan-out and single-server overview --live.

Code

model_gear/_metrics.py[R86-95]

+def http_get_text(url: str, *, timeout: float = 3.0) -> str | None:
+    """Best-effort GET → body text, or ``None`` if unreachable / non-2xx. Never raises."""
+    try:
+        with urllib.request.urlopen(
+            url, timeout=timeout
+        ) as r:  # nosec B310 - http(s) only, fixed scheme
+            if 200 <= r.status < 300:
+                return r.read().decode("utf-8", errors="replace")
+    except (urllib.error.URLError, OSError, ValueError):
+        return None

Relevance

⭐⭐⭐ High

Repo previously accepted bounding/streaming to avoid buffering large HTTP bodies (PR #24).

PR-#24

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The probe helper reads the full body into memory, and it is directly used for /metrics in both the
backend prober and live overview paths.

model_gear/_metrics.py[86-97]
model_gear/_metrics.py[116-128]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`http_get_text()` performs an unbounded `r.read()` into memory. Prometheus `/metrics` payloads can become large, and a misbehaving backend could return an excessively large body that stresses memory/latency.

## Issue Context
This helper is used by both the gateway `/status` probe fan-out and the CLI live overview probes.

## Fix Focus Areas
- model_gear/_metrics.py[86-97]
- model_gear/_metrics.py[116-128]

## Suggested fix
- Add a `max_bytes` parameter (separately for metrics vs health) and read at most that many bytes (e.g., `r.read(max_bytes + 1)` then treat overflow as failure).
- Optionally special-case `/health` to allow a tiny cap and `/metrics` to allow a larger but still bounded cap (e.g., 1–5 MiB).
- Ensure the function still never raises and returns `None` when the cap is exceeded.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Sequential status fan-out ✓ Resolved 🐞 Bug ➹ Performance

Description

fleet_status_payload() probes backends sequentially, and each probe performs two HTTP GETs (/health
then /metrics), so one slow/unreachable backend can add ~2×timeout of latency per backend per
request. This makes /status a latency and load amplifier and can degrade the gateway under repeated
polling.

Code

model_gear/gateway/server.py[R365-375]

+def fleet_status_payload(
+    table: RoutingTable, cfg: ServerConfig, probe=_metrics.probe_backend
+) -> dict:
+    """Live status for every backend + an aggregate busy count + the endpoint list."""
+    backends: list[dict] = []
+    running = waiting = 0
+    for b in table.backends:
+        st = probe(b.base_url, timeout=cfg.connect_timeout)
+        metrics = st.get("metrics") or {}
+        running += int(metrics.get("running", 0) or 0)
+        waiting += int(metrics.get("waiting", 0) or 0)

Relevance

⭐⭐ Medium

No clear precedent for parallelizing gateway fan-out; perf optimizations sometimes rejected (e.g.,
caching in PR #19).

PR-#19

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The gateway iterates backends and probes each synchronously; the prober itself does two network
calls per backend, multiplying worst-case delay and load.

model_gear/gateway/server.py[365-385]
model_gear/_metrics.py[111-128]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`GET /status` currently performs a sequential fan-out across all backends and performs multiple HTTP requests per backend. This can make the endpoint slow in proportion to fleet size and backend reachability, and increases load on both the gateway and backends when polled.

## Issue Context
`fleet_status_payload()` loops over `table.backends` and calls `probe_backend()`. `probe_backend()` does `health_ok()` (GET /health) and then GET /metrics.

## Fix Focus Areas
- model_gear/gateway/server.py[365-392]
- model_gear/_metrics.py[111-128]

## Suggested fix
- Probe backends concurrently (e.g., `concurrent.futures.ThreadPoolExecutor`) with a per-backend timeout and an overall request deadline.
- Consider skipping `/metrics` when `/health` is unreachable to halve worst-case timeouts.
- Consider a small in-memory cache (e.g., 0.5–2s TTL) so rapid polling doesn’t re-fan-out every time.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

4. Status leaks backend URLs ✓ Resolved 🐞 Bug ⛨ Security

Description

The gateway’s new GET /status response includes each backend’s internal base_url, exposing internal
hostnames/ports and contradicting the documented payload shape. This is an information disclosure
surface for anyone who can reach the gateway.

Code

model_gear/gateway/server.py[R376-384]

+        backends.append(
+            {
+                "name": b.name,
+                "task": b.task,
+                "served_name": b.served_name,
+                "base_url": b.base_url,
+                "health": st.get("health", "unreachable"),
+                "metrics": st.get("metrics"),
+            }

Relevance

⭐⭐ Medium

Security-hardening changes are often accepted (PR #5), but no direct precedent on hiding backend
base_url in /status.

PR-#5

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The /status payload explicitly serializes base_url, while the gateway docs specify a backends
shape that does not include it.

model_gear/gateway/server.py[365-385]
docs/gateway-fleet.md[96-101]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`GET /status` currently includes each backend’s `base_url` in the public JSON payload. This leaks internal-only routing details (hostnames/ports) and also diverges from the documented response schema.

## Issue Context
Docs describe `backends: [{name, task, served_name, health, metrics}]` (no `base_url`). The CLI live view also doesn’t need `base_url` to render.

## Fix Focus Areas
- model_gear/gateway/server.py[365-385]
- docs/gateway-fleet.md[96-101]

## Suggested fix
- Remove `base_url` from each backend entry in `fleet_status_payload()`.
- If you still want it for debugging, gate it behind an explicit config/env flag (default off) and document it clearly.
- Update docs to match the actual schema (whichever direction you choose).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[OriNachum]

Addressed all 4 Qodo findings + the colleague review in 9968d81:

1. Non-finite metrics crash (Reliability) → parse_metrics skips NaN/inf via math.isfinite before the int() conversions, preserving the best-effort contract.
2. Unbounded metrics read (Reliability) → http_get_text caps the body at 5 MiB (reads max_bytes+1, treats overflow as unavailable) so a misbehaving backend can't stress memory.
3. /status leaks backend URLs (Security) → dropped base_url from the payload (it's internal-only routing detail and /status can be reached over a public tunnel). Now matches the documented {name, task, served_name, health, metrics} schema; the CLI never needed it.
4. Sequential fan-out (Performance) → backends are now probed in parallel (ThreadPoolExecutor) with a bounded 3 s timeout, so /status can't hang for timeout × N; order preserved. Plus (colleague review) probe_backend short-circuits /metrics when /health fails, halving the cost for a dead backend.

Tests added for each; 345 pass; black/isort/flake8/bandit/markdownlint/rubric-gate clean.

• model-gear (Claude)

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud