phase 6.0: production-readiness pack

Zawwarsami16 · claude · Zawwarsami16 · commit a1a638d6173f · 2026-05-10T01:00:09.000Z
structured access logs: new logger zhub.access at INFO. one line per request: <status> <method> <path> <latency_ms>ms [ai=<name>] middleware in server.py captures latency, identifies the AI from the path prefix when matchable, and records into hub metrics. per-AI latency in /metrics: /metrics now returns by_ai[name].request_count, total_latency_ms, max_latency_ms, avg_latency_ms. integer ms. counters reset on hub restart. percentiles will follow if/when needed. named cloudflared tunnels: python -m zhub up --tunnel-name <name> uses `cloudflared tunnel run <name>` for a stable hostname instead of the random *.trycloudflare.com from quick tunnels. one-time setup via `cloudflared tunnel route dns <name> <hostname>` documented in docs/DEPLOY.md. readme: architecture section gets a mermaid diagram showing the four flow classes (browser BYOK, http client, MCP host, federated peer hub) and the publisher / connection sides of the WS multiplex. docs/DEPLOY.md: end-to-end VPS deployment walkthrough. ~10 minutes from `ssh root@vps` to a stable https://hub.example.com/<ai>/v1 endpoint served by systemd-managed hub + named-tunnel pair, with rotating brain creds documented. 130/130 pytest now. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -91,6 +91,7 @@ Tests: `pytest -v`. The e2e tests spin up the hub in-process and run the full pu
 - **Phase 5.1** ✅ — `python -m zhub up` one-shot quickstart: spawns hub on a free port, optionally Cloudflare tunnel, auto-detects brain, publishes, prints URL+KEY. `python -m zhub doctor` checks Python version, deps, optional cloudflared, brain credentials in env. New `zhub/__main__.py` dispatcher (subcommands: `up`, `server`, `doctor`). The "anyone can easily do it" gate.
 - **Phase 5.2** ✅ — Entity expansion: added `## install`, `## up`, `## paths` sections to `entity.md`. Header notes the file ships with the package so an AI installing zhub can read it locally before any hub is running.
 - **Phase 5.3** ✅ — `AnthropicAdapter` brain (5th adapter). Anthropic Messages API has its own SSE shape (`event:`/`data:` pairs, `content_block_delta` with `text_delta`); adapter normalizes to `ChatChunk` like the others. Registered in REGISTRY after Cerebras.
+- **Phase 6.0** ✅ — Production-readiness pack: structured access logs at `zhub.access` logger (one line per request: status + method + path + latency_ms + ai_name when applicable); per-AI latency tracking surfaced in `/metrics` (`request_count`, `total_latency_ms`, `max_latency_ms`, `avg_latency_ms`); `python -m zhub up --tunnel-name <name>` for cloudflared *named* tunnels (stable URL across restarts); README gets a mermaid arch diagram; new `docs/DEPLOY.md` walkthrough for a $5 VPS deployment with systemd unit files for hub + named tunnel.
 
 **Next (not started):** true tool_call delta streaming through SSE (4.2b), multi-tier API keys, capability-only WS connections (a "tool provider" usable by any AI on the hub, not tied to one publisher), MCP resources/prompts surface.
 
diff --git a/README.md b/README.md
@@ -168,25 +168,39 @@ print(resp["text"])
 
 ## Architecture
 
+```mermaid
+flowchart TB
+    classDef ext fill:#1f2937,color:#e5e7eb,stroke:#4b5563
+    classDef hub fill:#0f3460,color:#e5e7eb,stroke:#3b82f6
+    classDef pub fill:#064e3b,color:#e5e7eb,stroke:#10b981
+    classDef cli fill:#581c87,color:#e5e7eb,stroke:#a855f7
+
+    Pocket[browser BYOK<br/>e.g. Pocket]:::ext
+    OpenAIPy[openai-py / curl<br/>any HTTP client]:::ext
+    Claude[Claude Desktop /<br/>Cursor / Cline<br/>via MCP]:::ext
+
+    Hub((zhub hub<br/>FastAPI + WS<br/>SQLite persistence)):::hub
+
+    Pub1[publisher<br/>publish&#40;name, brain, ...&#41;]:::pub
+    Pub2[publisher<br/>another AI]:::pub
+
+    Conn1[connection<br/>connect&#40;ai, key, capabilities&#41;]:::cli
+    Conn2[connection<br/>another device]:::cli
+
+    Pocket -- HTTPS Bearer<br/>POST /v1/chat/completions --> Hub
+    OpenAIPy -- HTTPS Bearer --> Hub
+    Claude -- stdio JSON-RPC<br/>via zhub.mcp_server --> Hub
+
+    Hub <-- WS<br/>chat-request / chat-chunk<br/>invoke-result --> Pub1
+    Hub <-- WS --> Pub2
+
+    Hub <-- WS<br/>chat-request<br/>invoke-request --> Conn1
+    Hub <-- WS --> Conn2
+
+    Hub -. peer routing .- Hub2((peer hub)):::hub
 ```
-[curl / openai-py / friend's app]
-           │ HTTPS, Bearer key
-           ▼
-   ┌──────────────────────────┐
-   │   zhub hub server         │
-   │   • routes chat requests  │
-   │   • routes invokes        │
-   │   • holds the registry    │
-   └────────┬─────────────────┘
-            │  WebSocket multiplex
-            │
-   ┌────────┴──────────┐
-   │                   │
-[ AI publish() ]    [ Client connect() ]
-   │ chat_handler      │ capability handlers
-   │                   │
-   └─── bidirectional ─┘
-```
+
+The hub is a router. State (publisher registry, in-flight requests, rate-limit windows, metrics, entity extensions) lives in the hub process; persistence is SQLite. Publishers and connections each hold one long-lived WebSocket. Federation: hubs peer each other and proxy chat completions / WS register-connection for AIs hosted elsewhere.
 
 - The hub holds a registry of every published AI and every connection to it.
 - Publishers receive `connection-event` messages whenever a client connects, disconnects, or updates its capabilities.
diff --git a/docs/DEPLOY.md b/docs/DEPLOY.md
@@ -0,0 +1,149 @@
+# Deploy zhub on a $5 VPS in 10 minutes
+
+Goal: a stable URL that survives reboots and gives you `https://your-domain/...` on a quiet server you forget about.
+
+This guide assumes Ubuntu 22.04+ on a tiny box (1 vCPU, 512 MB RAM is fine for personal-tier loads). Adapt freely for Debian / Arch / Fedora; the steps are the same.
+
+---
+
+## 1. Server prep (90 seconds)
+
+```bash
+ssh root@your-vps
+apt update && apt install -y python3-venv python3-pip git curl
+adduser zhub --disabled-password --gecos ""
+su - zhub
+```
+
+## 2. Install zhub (60 seconds)
+
+```bash
+git clone https://github.com/Zawwarsami16/zhub
+cd zhub
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e '.[server,brains]'
+python -m zhub doctor   # sanity check
+```
+
+## 3. Cloudflare named tunnel — stable URL forever (4 minutes)
+
+A *quick tunnel* (`--public-tunnel`) gives you a random `*.trycloudflare.com` URL that changes every restart. A *named tunnel* keeps the same hostname forever. One-time setup:
+
+```bash
+# install cloudflared (linux/amd64)
+curl -L --output cloudflared.deb \
+    https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
+sudo dpkg -i cloudflared.deb
+
+# log in once — opens a browser link, you authorize a CF zone you control
+cloudflared tunnel login
+
+# create a named tunnel (does NOT expose anything yet)
+cloudflared tunnel create zhub-prod
+# → Created tunnel zhub-prod with id XYZ
+# → Tunnel credentials written to ~/.cloudflared/XYZ.json
+
+# point a hostname (must be in a CF zone you own) at the tunnel
+cloudflared tunnel route dns zhub-prod hub.example.com
+```
+
+Now `hub.example.com` is wired to your tunnel. The tunnel itself isn't running yet — that's step 4.
+
+## 4. Run zhub as a systemd service (3 minutes)
+
+Two services: one for the hub + brain publisher, one for the cloudflared tunnel. Both auto-restart on failure.
+
+`/etc/systemd/system/zhub.service`:
+
+```ini
+[Unit]
+Description=zhub hub + brain publisher
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=zhub
+WorkingDirectory=/home/zhub/zhub
+Environment="PATH=/home/zhub/zhub/.venv/bin:/usr/bin"
+# Set whichever brain creds you use:
+Environment="GROQ_API_KEY=gsk_REDACTED"
+ExecStart=/home/zhub/zhub/.venv/bin/python -m zhub up \
+    --no-tunnel --port 8080 --name me \
+    --db /home/zhub/zhub/zhub.db
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
+
+`/etc/systemd/system/zhub-tunnel.service`:
+
+```ini
+[Unit]
+Description=cloudflared named tunnel for zhub
+After=zhub.service network-online.target
+Requires=zhub.service
+
+[Service]
+Type=simple
+User=zhub
+ExecStart=/usr/bin/cloudflared tunnel --no-autoupdate run zhub-prod
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Enable + start:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now zhub.service zhub-tunnel.service
+sudo systemctl status zhub.service zhub-tunnel.service
+```
+
+## 5. Verify (60 seconds)
+
+```bash
+# health
+curl https://hub.example.com/healthz
+# → {"status":"ok","publishers":"1"}
+
+# entity (zhub's self-knowledge)
+curl https://hub.example.com/entity | head -20
+
+# the AI's manifest
+curl https://hub.example.com/me/manifest.json | jq .
+
+# grab the api key the publisher generated (one-time on first start;
+# stable forever after because of --db persistence)
+sudo journalctl -u zhub.service --no-pager | grep -E "KEY:" | tail -1
+```
+
+Paste `https://hub.example.com/me/v1` + the `zk_...` key into Pocket / openai-py / curl / Claude Desktop. Done.
+
+---
+
+## Operational notes
+
+**Logs.** `journalctl -u zhub.service -f` for live tail. Each request shows up at INFO via the `zhub.access` logger:
+
+```
+123 GET /healthz 0ms
+200 POST /me/v1/chat/completions 412ms ai=me
+```
+
+**Metrics.** `curl https://hub.example.com/metrics` returns a JSON snapshot with per-AI request_count, total_latency_ms, max_latency_ms, avg_latency_ms, plus rate-limit / peer-proxy / tool-call counters. Pipe to a collector if you want history.
+
+**Persistence.** `zhub.db` (SQLite) holds publishers + entity extensions. Survives reboots. Back it up the same way you'd back up any small file.
+
+**Updating.** `cd ~/zhub && git pull && pip install -e '.[server,brains]' && sudo systemctl restart zhub.service`. The named tunnel keeps running.
+
+**Resources.** zhub itself is ~30 MB RSS idle. The brain dominates: brain=ollama means a local model burning whatever Ollama burns; brain=groq/openai/cerebras/anthropic means just outbound HTTPS. For a brain-bills-elsewhere setup, a 512 MB / 1 vCPU box runs zhub + cloudflared + a publisher comfortably.
+
+**Multi-AI.** Run more publishers with different `--name`s against the same hub. Each gets its own `zk_` key + URL. The hub doesn't need restart — just spawn another publisher process.
+
+**Rotating the brain.** Stop the publisher service, change `Environment="..."` in the unit file with a new brain's creds, restart. The `zk_` key is preserved (persistence). External clients see no change; brain underneath silently swapped.
diff --git a/tests/test_metrics_latency.py b/tests/test_metrics_latency.py
@@ -0,0 +1,83 @@
+"""Phase 6.0 — per-AI request latency surfaced in /metrics.
+
+Each /<ai>/v1/* request bumps a per-AI rolling latency counter so
+operators can see how long the publisher is taking to respond.
+/metrics returns avg_latency_ms and max_latency_ms per AI.
+"""
+
+import asyncio
+import socket
+import threading
+import time
+
+import pytest
+
+try:
+    import fastapi  # noqa
+    import uvicorn  # noqa
+    import httpx  # noqa
+    DEPS_AVAILABLE = True
+except ImportError:
+    DEPS_AVAILABLE = False
+
+if DEPS_AVAILABLE:
+    from zhub.server import create_app
+from zhub import publish
+
+
+def _free_port() -> int:
+    with socket.socket() as s:
+        s.bind(("", 0))
+        return s.getsockname()[1]
+
+
+@pytest.fixture(scope="module")
+def lat_hub():
+    if not DEPS_AVAILABLE:
+        pytest.skip("fastapi/uvicorn/httpx not installed")
+    port = _free_port()
+    app = create_app()
+
+    def run():
+        config = uvicorn.Config(app, host="127.0.0.1", port=port,
+                                log_level="warning")
+        asyncio.run(uvicorn.Server(config).serve())
+
+    threading.Thread(target=run, daemon=True).start()
+    for _ in range(30):
+        try:
+            with socket.create_connection(("127.0.0.1", port), timeout=0.1):
+                break
+        except OSError:
+            time.sleep(0.1)
+    yield port
+
+
+@pytest.mark.asyncio
+async def test_per_ai_latency_in_metrics(lat_hub):
+    pub = publish(
+        name="lat-bot",
+        description="latency test",
+        chat_handler=lambda m, o: "ok",
+        hub_url=f"ws://127.0.0.1:{lat_hub}",
+    )
+    for _ in range(50):
+        if pub.api_key:
+            break
+        await asyncio.sleep(0.1)
+
+    async with httpx.AsyncClient(timeout=5.0) as c:
+        for _ in range(3):
+            await c.post(
+                f"http://127.0.0.1:{lat_hub}/{pub.name}/v1/chat/completions",
+                json={"messages": [{"role": "user", "content": "x"}]},
+                headers={"Authorization": f"Bearer {pub.api_key}"},
+            )
+        m = (await c.get(f"http://127.0.0.1:{lat_hub}/metrics")).json()
+
+    by_ai = m["by_ai"]
+    assert pub.name in by_ai
+    e = by_ai[pub.name]
+    assert "avg_latency_ms" in e and e["avg_latency_ms"] >= 0
+    assert "max_latency_ms" in e and e["max_latency_ms"] >= e["avg_latency_ms"]
+    assert "request_count" in e and e["request_count"] >= 3
diff --git a/tests/test_request_logging.py b/tests/test_request_logging.py
@@ -0,0 +1,93 @@
+"""Phase 6.0 — structured request logging middleware.
+
+Every HTTP request to the hub is logged at INFO with a single line
+containing: method, path, status, latency_ms, ai_name (when path
+matches /<ai>/...). Logs go to stderr via the zhub.access logger so
+operators can tail them or pipe to a structured collector.
+"""
+
+import asyncio
+import logging
+import socket
+import threading
+import time
+
+import pytest
+
+try:
+    import fastapi  # noqa
+    import uvicorn  # noqa
+    import httpx  # noqa
+    DEPS_AVAILABLE = True
+except ImportError:
+    DEPS_AVAILABLE = False
+
+if DEPS_AVAILABLE:
+    from zhub.server import create_app
+from zhub import publish
+
+
+def _free_port() -> int:
+    with socket.socket() as s:
+        s.bind(("", 0))
+        return s.getsockname()[1]
+
+
+@pytest.fixture
+def hub_with_log_capture(caplog):
+    if not DEPS_AVAILABLE:
+        pytest.skip("fastapi/uvicorn/httpx not installed")
+    port = _free_port()
+    app = create_app()
+
+    def run():
+        config = uvicorn.Config(app, host="127.0.0.1", port=port,
+                                log_level="warning")
+        asyncio.run(uvicorn.Server(config).serve())
+
+    threading.Thread(target=run, daemon=True).start()
+    for _ in range(30):
+        try:
+            with socket.create_connection(("127.0.0.1", port), timeout=0.1):
+                break
+        except OSError:
+            time.sleep(0.1)
+    caplog.set_level(logging.INFO, logger="zhub.access")
+    yield port
+
+
+@pytest.mark.asyncio
+async def test_each_request_logged_with_status_and_latency(hub_with_log_capture, caplog):
+    port = hub_with_log_capture
+    async with httpx.AsyncClient(timeout=5.0) as c:
+        await c.get(f"http://127.0.0.1:{port}/healthz")
+    # Log message: "200 GET /healthz <ms>ms"
+    msgs = [r.message for r in caplog.records if r.name == "zhub.access"]
+    assert any("/healthz" in m and "200" in m and "ms" in m for m in msgs), \
+        f"no access log line found: {msgs!r}"
+
+
+@pytest.mark.asyncio
+async def test_ai_path_logs_include_ai_name(hub_with_log_capture, caplog):
+    port = hub_with_log_capture
+    pub = publish(
+        name="logbot",
+        description="x",
+        chat_handler=lambda m, o: "ok",
+        hub_url=f"ws://127.0.0.1:{port}",
+    )
+    for _ in range(50):
+        if pub.api_key:
+            break
+        await asyncio.sleep(0.1)
+
+    caplog.clear()
+    async with httpx.AsyncClient(timeout=5.0) as c:
+        await c.post(
+            f"http://127.0.0.1:{port}/{pub.name}/v1/chat/completions",
+            json={"messages": [{"role": "user", "content": "hi"}]},
+            headers={"Authorization": f"Bearer {pub.api_key}"},
+        )
+    msgs = [r.message for r in caplog.records if r.name == "zhub.access"]
+    assert any("logbot" in m and "200" in m for m in msgs), \
+        f"no AI-tagged access log: {msgs!r}"
diff --git a/zhub/cli_up.py b/zhub/cli_up.py
diff --git a/zhub/server.py b/zhub/server.py
