phase 18.0: prometheus metrics format + tutorial doc + examples README

Zawwarsami16 · claude · Zawwarsami16 · commit 5a355e89f528 · 2026-05-10T04:17:04.000Z
GET /metrics?format=prometheus emits OpenMetrics text exposition:

  # HELP zhub_uptime_seconds Hub process uptime in seconds.
  # TYPE zhub_uptime_seconds gauge
  zhub_uptime_seconds 12345
  # HELP zhub_chat_requests_total Total chat completions handled per AI.
  # TYPE zhub_chat_requests_total counter
  zhub_chat_requests_total{ai="my-ai"} 1247
  zhub_p95_latency_ms{ai="my-ai"} 1820
  ...

per-AI metrics carry the `ai` label; counters use `_total` suffix per
OpenMetrics convention. consumable by prometheus, victoriametrics,
datadog openmetrics agents, grafana cloud agents.

default response (no format param) stays JSON — backwards compat.

new docs:
  docs/TUTORIAL.md — 10-minute hands-on walkthrough. install → bring
    up hub + brain → curl/openai-py/Pocket/Claude Desktop integration
    → expose a tool. table-of-pitfalls at the end with common
    fix-ups.
  examples/README.md — index of all runnable examples with one-line
    descriptions, what each needs, and pointers for plugging the
    output into Pocket / Claude Desktop / curl.

readme: tutorial linked under TL;DR.

entity.md: /metrics section notes the ?format=prometheus query param.

178/178 pytest in isolation; same handful of cross-test flakes in
the long suite resolve fine in CI's faster runners.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/README.md b/README.md
@@ -23,6 +23,8 @@ pip install -e '.[server,brains]'
 GROQ_API_KEY=gsk_... zhub up
 ```
 
+> **First time?** A 10-minute hands-on walkthrough lives at [`docs/TUTORIAL.md`](docs/TUTORIAL.md) — clone to "Pocket talks to your AI" in a single sitting.
+
 (`zhub up` is the installed CLI; `python -m zhub up` is the equivalent module form when the script isn't on PATH.)
 
 Output:
diff --git a/docs/TUTORIAL.md b/docs/TUTORIAL.md
@@ -0,0 +1,185 @@
+# zhub in 10 minutes — hands-on
+
+Goal: by the end of this, you have an OpenAI-compatible AI endpoint reachable from your phone. We'll go from `git clone` to "Pocket talks to your AI" in a single sitting.
+
+## Minute 0–2: install
+
+```bash
+git clone https://github.com/Zawwarsami16/zhub
+cd zhub
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e '.[server,brains]'
+```
+
+Sanity check:
+
+```bash
+python -m zhub doctor
+```
+
+You should see green checks for `import zhub`, the server deps, and `cloudflared` (if installed). Brain creds will all show ✗ unless you've already set env vars — that's fine, we'll fix it next.
+
+## Minute 2–3: pick a brain
+
+You need credentials for **one** of these — any of the eight will work. Pick by what you have or want to use:
+
+| Brain | Why pick it | Get a key |
+|---|---|---|
+| **Ollama** | Free, runs on your machine, no signup | `ollama serve` then `ollama pull llama3.2` |
+| **Groq** | Free tier, **700 tok/s**, fastest perceived | https://console.groq.com |
+| **OpenAI** | Familiar, gpt-4o-mini is cheap | https://platform.openai.com |
+| **Anthropic** | Best reasoning (Claude) | https://console.anthropic.com |
+| **Together / Mistral / Cohere / Cerebras** | More options | their respective consoles |
+
+For this tutorial we'll use **Groq** because the free tier is generous and replies are instant:
+
+```bash
+export GROQ_API_KEY=gsk_your_key_here
+```
+
+## Minute 3–4: bring it up
+
+```bash
+python -m zhub up
+```
+
+You'll see something like:
+
+```
+================================================================
+  brain:    Groq Llama 3.3 70B
+  URL:      https://stuck-bonus-eight-spider.trycloudflare.com/me/v1
+  KEY:      zk_BWOuFb8-Fiw8JpVjWO3hNwCaTfASE_To
+  paste both into Pocket / openai-py / curl / Claude Desktop
+================================================================
+```
+
+That's it. `python -m zhub up` started:
+- A hub server on port 8080
+- A Cloudflare quick-tunnel (the `*.trycloudflare.com` URL is reachable from anywhere)
+- A publisher that proxies chat to Groq
+- Persistent SQLite at `./zhub.db` so the `zk_` key survives restarts
+
+## Minute 4–6: try it from the command line
+
+In another terminal:
+
+```bash
+curl -X POST https://stuck-bonus-eight-spider.trycloudflare.com/me/v1/chat/completions \
+  -H "Authorization: Bearer zk_BWO..." \
+  -H "Content-Type: application/json" \
+  -d '{"messages":[{"role":"user","content":"explain zhub in one sentence"}]}'
+```
+
+If that works, your AI is reachable from anywhere on the internet.
+
+## Minute 6–7: open the dashboard
+
+Open `http://localhost:8080/` in a browser. You'll see a live operator console — particles flow through the SVG every time a request lands. Stat tiles show publishers / connections / requests / p95 latency. The recent-requests feed scrolls in real time.
+
+## Minute 7–9: plug it into a chat app
+
+### Pocket (browser-based BYOK)
+
+Pocket auto-detects zhub providers. Open Pocket → settings → add provider → paste:
+- Base URL: `https://stuck-bonus-eight-spider.trycloudflare.com/me/v1`
+- API Key: `zk_BWO...`
+
+Save. The model `me` appears in the picker. Send a message — it round-trips through your hub.
+
+### Claude Desktop / Cursor / Cline
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json` (or your client's MCP config):
+
+```json
+{
+  "mcpServers": {
+    "me": {
+      "command": "python",
+      "args": [
+        "-m", "zhub.mcp_server",
+        "--hub", "https://stuck-bonus-eight-spider.trycloudflare.com",
+        "--ai", "me",
+        "--key", "zk_BWO..."
+      ]
+    }
+  }
+}
+```
+
+Restart Claude Desktop. Your AI shows up as a `chat` tool. Anything connected to your hub via `expose()` (next section) shows up as additional MCP tools.
+
+### openai-py
+
+```python
+from openai import OpenAI
+client = OpenAI(
+    base_url="https://stuck-bonus-eight-spider.trycloudflare.com/me/v1",
+    api_key="zk_BWO...",
+)
+print(client.chat.completions.create(
+    model="me",
+    messages=[{"role": "user", "content": "hi"}],
+).choices[0].message.content)
+```
+
+## Minute 9–10: add a tool
+
+Your AI doesn't have a `weather_lookup` tool, but you can give it one in 30 lines. In a third terminal:
+
+```python
+# weather_sensor.py
+import asyncio
+from zhub import expose
+
+e = expose(
+    name="weather-sensor",
+    capabilities={
+        "weather_lookup": (
+            {"type": "object", "required": ["city"],
+             "properties": {"city": {"type": "string"}}},
+            lambda args: {"city": args["city"], "temp_c": 14, "condition": "cloudy"},
+        ),
+    },
+    hub_url="ws://127.0.0.1:8080",
+    public=True,
+)
+
+async def main():
+    while not e.exposure_id:
+        await asyncio.sleep(0.05)
+    print(f"weather sensor exposed as {e.exposure_id}")
+    await asyncio.Event().wait()
+
+asyncio.run(main())
+```
+
+Run it: `python weather_sensor.py`. Now the dashboard shows an "exposure" tile. Any AI on the hub can invoke this capability via `POST /exposures/<id>/invoke` with a publisher's bearer key. After Phase 1.9 capability injection lands for exposures, the AI will see it as a tool automatically.
+
+## What you have now
+
+A reusable, production-ready AI endpoint that:
+- Speaks OpenAI Chat Completions wire format → works in any BYOK client
+- Streams via SSE (Cursor / Continue compatible)
+- Is reachable from anywhere via Cloudflare Tunnel
+- Survives restarts (SQLite persistence keeps the `zk_` key stable)
+- Has a live operator dashboard
+- Routes tool calls to connected device-only "exposures"
+- Bridges to Claude Desktop via MCP (chat + every connected capability becomes a tool)
+
+## Next steps
+
+- [`docs/DEPLOY.md`](DEPLOY.md) — deploy this on a $5 VPS so it stays up forever
+- [`examples/`](../examples/README.md) — more runnable demos (federation, council, multi-brain, full-stack)
+- `curl <hub>/entity` — zhub's self-knowledge: routes, errors, patterns, debug recipes
+- `python -m zhub status <hub-url>` — pretty-print any remote hub's state
+
+## Common pitfalls
+
+| Symptom | Fix |
+|---|---|
+| `python -m zhub up` says "no brains available" | Set one of the env vars in step 2 |
+| Cloudflared tunnel didn't start | `apt install cloudflared` (or skip with `--no-tunnel`) |
+| Pocket says "couldn't auto-detect" | Make sure your URL ends with `/v1` (zhub provider matches that pattern) |
+| `zk_` key changed after restart | You're missing `--db ./zhub.db` flag on the hub. `zhub up` sets it by default |
+| Browser dashboard is blank | Check `http://localhost:8080/api/dashboard` returns JSON. If not, check `python -m zhub doctor` |
diff --git a/examples/README.md b/examples/README.md
@@ -0,0 +1,64 @@
+# zhub examples
+
+Runnable scripts that show off each primitive in isolation, plus a few that combine them. Each is self-contained — no setup beyond a running hub (or, in some cases, the script spins one up itself).
+
+## Index
+
+| File | What it shows | Needs |
+|---|---|---|
+| [`publish_demo.py`](publish_demo.py) | Minimal `publish()` of a stateless echo AI | hub running |
+| [`connect_demo.py`](connect_demo.py) | `connect()` a generic client + expose two capabilities back to a paired AI | hub + a published AI's name + key |
+| [`orchestrate_demo.py`](orchestrate_demo.py) | All-in-one: AI publishes itself, client connects with capabilities, AI invokes them on demand | hub running |
+| [`council_demo.py`](council_demo.py) | A "council" pattern — multiple AIs deliberate on a question | hub running |
+| [`tool_demo.py`](tool_demo.py) | OpenAI tool-calling end-to-end (Phase 1.8): publisher emits `tool_calls`, hub auto-resolves against connected capabilities | hub running |
+| [`federation_demo.py`](federation_demo.py) | Two hubs in one process; AI on hub B reachable through hub A via HTTP and WS | none (in-process) |
+| [`mcp_bridge.py`](mcp_bridge.py) | Wrap an existing MCP server as a zhub publisher (subprocess + JSON-RPC over stdio) | hub + an MCP server binary (`MCP_COMMAND` env) |
+| [`multi_brain_publisher.py`](multi_brain_publisher.py) | Pick a brain (auto-detect or explicit `--brain ollama|groq|openai|cerebras|anthropic|together|mistral|cohere`), publish, stream responses through hub | hub + brain credentials in env |
+| [`full_stack_demo.py`](full_stack_demo.py) | The whole thing in one file: hub + publisher + exposure + tool resolution + dashboard pointer | none (in-process) |
+
+## Running
+
+Most examples assume a hub is reachable at `ws://localhost:8080`. Start one in a separate terminal:
+
+```bash
+python -m zhub.server --port 8080
+# or
+python -m zhub up --no-tunnel --brain ollama
+```
+
+Then run the example:
+
+```bash
+python examples/publish_demo.py
+```
+
+## Wiring an example into Pocket / Claude Desktop / curl
+
+Once an example publishes, the printed `URL:` and `KEY:` are usable from anywhere that speaks OpenAI Chat Completions:
+
+```bash
+# curl
+curl -X POST http://localhost:8080/echo/v1/chat/completions \
+  -H "Authorization: Bearer zk_..." \
+  -d '{"messages":[{"role":"user","content":"hi"}]}'
+
+# openai-py
+from openai import OpenAI
+client = OpenAI(base_url="http://localhost:8080/echo/v1", api_key="zk_...")
+client.chat.completions.create(model="echo", messages=[{"role":"user","content":"hi"}])
+```
+
+For Claude Desktop / Cursor / Cline, add to MCP config:
+
+```json
+{"mcpServers": {"echo": {
+  "command": "python", "args": ["-m", "zhub.mcp_server",
+    "--hub", "http://localhost:8080", "--ai", "echo", "--key", "zk_..."]
+}}}
+```
+
+## Writing your own
+
+Take `publish_demo.py` as a starting template — replace the `chat_handler` with whatever your brain is (a real LLM API, a local model, a stub). The rest of the surface (auth, tunnel, persistence, MCP, dashboard, federation) is the hub's job, not yours.
+
+For wrapping existing services as publishers, `mcp_bridge.py` shows the subprocess + adapter pattern; `multi_brain_publisher.py` shows the env-driven brain selection pattern.
diff --git a/tests/test_metrics_prometheus.py b/tests/test_metrics_prometheus.py
@@ -0,0 +1,100 @@
+"""Phase 18.0 — Prometheus exposition format for /metrics."""
+
+import asyncio
+import socket
+import threading
+import time
+
+import pytest
+
+try:
+    import fastapi  # noqa
+    import uvicorn  # noqa
+    import httpx  # noqa
+    DEPS_AVAILABLE = True
+except ImportError:
+    DEPS_AVAILABLE = False
+
+if DEPS_AVAILABLE:
+    from zhub.server import create_app
+from zhub import publish
+
+
+def _free_port() -> int:
+    with socket.socket() as s:
+        s.bind(("", 0))
+        return s.getsockname()[1]
+
+
+@pytest.fixture
+def hub():
+    if not DEPS_AVAILABLE:
+        pytest.skip("fastapi/uvicorn/httpx not installed")
+    port = _free_port()
+    app = create_app()
+
+    def run():
+        config = uvicorn.Config(app, host="127.0.0.1", port=port,
+                                log_level="warning")
+        asyncio.run(uvicorn.Server(config).serve())
+
+    threading.Thread(target=run, daemon=True).start()
+    for _ in range(30):
+        try:
+            with socket.create_connection(("127.0.0.1", port), timeout=0.1):
+                break
+        except OSError:
+            time.sleep(0.1)
+    yield port
+
+
+@pytest.mark.asyncio
+async def test_prometheus_format_returns_text_exposition(hub):
+    pub = publish(
+        name="prom-bot", description="x",
+        chat_handler=lambda m, o: "ok",
+        hub_url=f"ws://127.0.0.1:{hub}",
+    )
+    for _ in range(50):
+        if pub.api_key:
+            break
+        await asyncio.sleep(0.1)
+
+    async with httpx.AsyncClient(timeout=5.0) as c:
+        for _ in range(3):
+            await c.post(
+                f"http://127.0.0.1:{hub}/{pub.name}/v1/chat/completions",
+                json={"messages": [{"role": "user", "content": "x"}]},
+                headers={"Authorization": f"Bearer {pub.api_key}"},
+            )
+        r = await c.get(f"http://127.0.0.1:{hub}/metrics?format=prometheus")
+
+    assert r.status_code == 200
+    assert "text/plain" in r.headers["content-type"].lower()
+    body = r.text
+
+    # Hub-level counters
+    assert "# HELP zhub_uptime_seconds" in body
+    assert "# TYPE zhub_uptime_seconds gauge" in body
+    assert "zhub_uptime_seconds " in body
+    assert "zhub_publishers " in body
+
+    # Per-AI metrics with labels
+    assert "# HELP zhub_chat_requests_total" in body
+    assert "# TYPE zhub_chat_requests_total counter" in body
+    # Label format: zhub_chat_requests_total{ai="prom-bot"} N
+    assert 'zhub_chat_requests_total{ai="prom-bot"}' in body
+    assert 'zhub_avg_latency_ms{ai="prom-bot"}' in body
+    assert 'zhub_p95_latency_ms{ai="prom-bot"}' in body
+
+
+@pytest.mark.asyncio
+async def test_metrics_default_still_json(hub):
+    """Backwards compat: no `format` param returns the existing JSON shape."""
+    async with httpx.AsyncClient(timeout=5.0) as c:
+        r = await c.get(f"http://127.0.0.1:{hub}/metrics")
+    assert r.status_code == 200
+    assert "application/json" in r.headers["content-type"].lower()
+    d = r.json()
+    assert "hub_id" in d
+    assert "by_ai" in d
diff --git a/zhub/entity.md b/zhub/entity.md
@@ -103,14 +103,19 @@ hosted elsewhere.
 ### `GET /healthz`
 Liveness probe. Returns `{status, publishers}`. No auth.
 
-### `GET /metrics`
+### `GET /metrics` (`?format=prometheus` for text exposition)
 Hub-wide counters (per-AI: chat_requests, rate_limited, peer_proxied,
 tool_calls_resolved, http_invoke, request_count, total_latency_ms,
 max_latency_ms, avg_latency_ms, **p50/p95/p99_latency_ms**). The
 percentiles come from a per-AI ring buffer of the last 200 latencies —
 recent behavior, not lifetime history. Use to track who's hot, who's
 failing, and where the tail is. No auth (snapshot only, no secrets).
 
+Append `?format=prometheus` for OpenMetrics text exposition consumable
+by Prometheus, VictoriaMetrics, Datadog OpenMetrics agents, etc.
+Counter metrics use the `_total` suffix; per-AI metrics carry an `ai`
+label.
+
 ### `GET /` and `GET /api/dashboard`
 `/` serves an HTML operator dashboard (auto-refresh 3s). `/api/dashboard`
 returns the JSON snapshot it polls — same data shape as `/metrics` plus
diff --git a/zhub/server.py b/zhub/server.py
