feat(basilica): mint scoped operator key for tenant runtime traffic (#239)

Splits per-tenant proxy auth into two roles so a compromised tenant app
no longer holds admin scope on its own LLMTrace pod:

- admin_key (bootstrap): retained by the caller. Used by the lifecycle
  layer to mint per-tenant keys and by the self-service / admin portal.
- api_key (operator): minted on the live proxy via POST /api/v1/auth/keys
  after readiness, returned in TenantInstances.api_key. This is the
  bearer the tenant's runtime apps use.

provision() now bootstraps the per-pod tenant row via POST /api/v1/tenants,
mints the operator key, and injects LLMTRACE_AUTH_RUNTIME_KEY into the
dashboard env (informational; dashboard wiring is a follow-up).

update(strategy="restart") rediscovers the tenant by label, lists keys,
and re-mints only when the operator record is missing. update(strategy=
"recreate") always re-mints since the DB volume is destroyed.

cli.py emits admin_key alongside api_key. The tenant-lifecycle workflow
masks BOTH keys via ::add-mask:: before any cat result.json and exposes
both as step outputs.

Adds deployments/basilica/tests/ with 19 unit tests that exercise the
real urllib admin-API client against an in-process http.server, plus a
provision() integration test using a fake Basilica client.

Author: epappas

.github/workflows/tenant-lifecycle.yml

@@ -259,16 +259,18 @@ jobs:
           code=$?
           set -e
 
-          # Mask the api_key BEFORE any log line (cat / step summary) touches
-          # result.json. ::add-mask:: is per-job and applies to subsequent log
-          # lines, so registering it here covers the cat below + the summary
-          # step + any downstream consumers that print step outputs.
+          # Mask BOTH api_key and admin_key BEFORE any log line (cat /
+          # step summary) touches result.json. ::add-mask:: is per-job
+          # and applies to subsequent log lines, so registering them
+          # here covers the cat below + the summary step + any
+          # downstream consumers that print step outputs.
           python3 - <<'PY'
           import json, pathlib
           data = json.loads(pathlib.Path("result.json").read_text() or "{}")
-          key = data.get("api_key")
-          if key:
-              print(f"::add-mask::{key}")
+          for field in ("api_key", "admin_key"):
+              value = data.get(field)
+              if value:
+                  print(f"::add-mask::{value}")
           PY
 
           cat result.json
@@ -280,10 +282,14 @@ jobs:
           print(f"dashboard_instance_id={data.get('dashboard_instance_id') or ''}")
           print(f"proxy_url={data.get('proxy_url') or ''}")
           print(f"dashboard_url={data.get('dashboard_url') or ''}")
-          # api_key is masked; emitting it as a step output is fine because
-          # downstream consumers (the caller's app via the Actions API) need it
-          # to give the tenant their bearer token. The value is opaque to logs.
+          # api_key (operator-scoped, given to the tenant) and admin_key
+          # (bootstrap-scoped, retained by the caller for self-service /
+          # admin pages) are masked above. Emitting them as step outputs
+          # is fine because downstream consumers (the caller's app via
+          # the Actions API) need both: api_key for the tenant's bearer,
+          # admin_key for the platform's own admin pages.
           print(f"api_key={data.get('api_key') or ''}")
+          print(f"admin_key={data.get('admin_key') or ''}")
           PY
           if [[ "${code}" -ne 0 ]]; then
             exit "${code}"

deployments/basilica/README.md

@@ -298,69 +298,100 @@ wins for that tenant.
 
 Both demonstrate `${VAR}` substitution and the optional override pattern.
 
-## Per-tenant API key auth (secure by default)
+## Per-tenant API key auth (two-tier, scoped runtime key)
 
 LLMTrace's proxy has built-in API-key auth (`crates/llmtrace-proxy/src/auth.rs`).
 Without it, the public Basilica URL accepts any request — anyone with the
 URL can burn the tenant's upstream quota and pollute their traces. With it
 on, every non-`/health` request must carry `Authorization: Bearer llmt_<key>`
 or get a 401.
 
-The lifecycle library enables this **by default** at provision time:
+The lifecycle library enables auth **by default** at provision time, and
+issues TWO keys with distinct scopes:
 
-1. Resolves a key, in priority order:
+| Key | Role | Who holds it | What it can do |
+|---|---|---|---|
+| `admin_key` | bootstrap admin | the **caller** (your app / portal) | Mint / list / revoke per-tenant keys, manage tenants, read audit logs, change feature flags. Used by the lifecycle layer itself and by your self-service / admin portal pages |
+| `api_key`   | operator | the **tenant**'s runtime apps | Proxy LLM calls, write traces, report agent actions. **No** key management, **no** audit-log access, **no** tenant CRUD |
+
+### Bootstrap sequence (what `provision()` does internally)
+
+1. Resolves a bootstrap admin key, priority order:
    - Explicit `spec.api_key` from the caller (or `api_key:` field in the
-     YAML config), used for plan-recreates that must preserve the tenant's
-     existing key
-   - Existing `LLMTRACE_AUTH_ADMIN_KEY` in `proxy.env` (rare — caller wrote
-     it themselves)
-   - Auto-generated `llmt_<64-hex>` via `generate_api_key()` matching the
-     format produced by the Rust proxy at `auth.rs:44`
-2. Injects `LLMTRACE_AUTH_ENABLED=true` + `LLMTRACE_AUTH_ADMIN_KEY=<key>`
-   into the proxy's env, and the same `LLMTRACE_AUTH_ADMIN_KEY` into the
-   dashboard's env (so the dashboard can authenticate to the proxy's admin
-   endpoints)
-3. Returns the plaintext key in `TenantInstances.api_key` — **this is the
-   only time it's exposed**. Persist it in your app DB and ship it to the
-   tenant.
+     YAML config) — pin this if you need the admin key stable across
+     recreates.
+   - Existing `LLMTRACE_AUTH_ADMIN_KEY` in `proxy.env` (rare — caller
+     wrote it themselves).
+   - Auto-generated `llmt_<64-hex>` via `generate_api_key()`.
+2. Injects `LLMTRACE_AUTH_ENABLED=true` + `LLMTRACE_AUTH_ADMIN_KEY=<admin_key>`
+   into BOTH the proxy and dashboard envs. The dashboard keeps the admin
+   key so its server-side handlers can call the proxy's admin endpoints
+   on behalf of the operator/portal UI.
+3. Creates the proxy deployment and waits for it to become ready.
+4. Calls `POST /api/v1/tenants` on the live proxy URL (auth: admin key)
+   to materialise the per-pod tenant row. Captures the assigned tenant
+   UUID.
+5. Calls `POST /api/v1/auth/keys` (auth: admin key, scoped to that
+   tenant UUID) with body `{name: "tenant-runtime", role: "operator",
+   tenant_id: <uuid>}` and captures the plaintext operator key.
+6. Adds `LLMTRACE_AUTH_RUNTIME_KEY=<operator_key>` to the dashboard env
+   (informational today; consumed by a follow-up dashboard wiring
+   change) and creates the dashboard deployment.
+7. Returns `TenantInstances(api_key=<operator>, admin_key=<admin>)`.
+
+Both plaintext keys are exposed only at this moment. Persist them
+immediately in your app's secret store.
 
 ```python
 result = lifecycle.provision(spec)
-tenant_record.api_key = result.api_key      # llmt_xxxxxxxxxxxx... — only exposed here
+tenant_record.api_key   = result.api_key     # operator-scoped — ship to the tenant
+tenant_record.admin_key = result.admin_key   # bootstrap-scoped — retain in your app
 tenant_record.proxy_url = result.proxy.url
 db.session.commit()
 ```
 
-The CLI emits it in the result JSON:
+The CLI emits both in the result JSON:
 
 ```json
 {
   "tenant_id": "acme",
   "proxy_url": "https://...basilica.ai",
   "dashboard_url": "https://...basilica.ai",
-  "api_key": "llmt_d34c8a..."
+  "api_key": "llmt_op...",
+  "admin_key": "llmt_ad..."
 }
 ```
 
-The workflow registers `::add-mask::` for the key before any `cat`
-operation, so it never appears in run logs — only in step outputs (which
+The workflow registers `::add-mask::` for BOTH keys before any `cat`
+operation, so neither appears in run logs — only in step outputs (which
 the calling app fetches via the Actions API).
 
 ### Tenant-side usage
 
-The tenant programs their downstream apps to send their API key on every
-request to their proxy URL:
+The tenant programs their downstream apps to send the OPERATOR key on
+every request to their proxy URL:
 
 ```bash
 curl -X POST https://<proxy_uuid>.deployments.basilica.ai/v1/chat/completions \
-  -H "Authorization: Bearer llmt_d34c8a..." \
+  -H "Authorization: Bearer llmt_op..." \
   -H "Content-Type: application/json" \
   -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'
 ```
 
-A 401 means they used the wrong key. A 200 means LLMTrace authenticated
-them, then forwarded upstream (with whatever upstream auth was configured
-in `tenant_secrets`).
+A 401 means they used the wrong key. A 403 with "Insufficient permissions"
+means they tried an admin-only endpoint (`/api/v1/auth/keys`,
+`/api/v1/tenants`, etc.) with the operator key — by design.
+
+### Update semantics
+
+| Strategy | DB persistence | Operator key behaviour |
+|---|---|---|
+| `recreate` | DB volume is destroyed alongside the pods | Always re-minted. `result.api_key` carries the new operator key; caller must overwrite the stored value |
+| `restart`  | Same DB volume — rows survive | The library lists `GET /api/v1/tenants` to rediscover the tenant by label, then lists `GET /api/v1/auth/keys` for that tenant. If a non-revoked `tenant-runtime` operator key is found, `result.api_key` is `None` (carry forward the previously-stored plaintext from your DB — the proxy stores only a hash, so re-fetching the plaintext is impossible). If the record is missing (e.g. DB wipe), a fresh operator key is minted and returned |
+
+The admin key follows the same recreate / restart split: pin it via
+`spec.api_key` (or `api_key:` in YAML) if you need it stable across
+recreates; on restart it is simply re-derived from spec and not re-minted.
 
 ### Disabling auth
 
@@ -371,43 +402,41 @@ mesh, a dev sandbox, etc.), turn auth off:
 enable_proxy_auth: false
 ```
 
-The library skips key generation, doesn't inject `LLMTRACE_AUTH_*`, and
-returns `api_key: null` in the result. The proxy URL is then wide open;
-own the consequences.
-
-### Preserving the key across recreates
-
-`update(..., strategy="recreate")` calls `provision()` under the hood,
-which will auto-generate a fresh key unless you pass the existing one.
-For plan upgrades where the tenant's apps shouldn't break, fetch the
-current key from your DB and put it in the spec:
-
-```python
-spec = build_spec_from_tenant_record(tenant_record)
-spec = dataclasses.replace(spec, api_key=tenant_record.api_key)  # preserve
-new_instances = lifecycle.update(spec, proxy_id=..., dashboard_id=..., strategy="recreate")
-# new_instances.api_key == tenant_record.api_key  (carried forward)
-```
-
-### Caveats
-
-- **Admin-key-as-runtime-key is overpowered.** The auto-generated key
-  takes the `auth.admin_key` slot (bootstrap admin role), meaning the
-  tenant's apps technically have admin scope on their own instance.
-  Hardening: after provision, POST a scoped non-admin key via the
-  proxy's `/admin/keys` endpoint and hand THAT to the tenant. Tracked
-  as a follow-up.
-- **Defence-in-depth** still worth doing separately: per-tenant rate
-  limits (LLMTrace likely has them — configure), and DoS protection
-  against CPU burn from ML detectors running before the upstream call.
-  The proxy now bounds intra-pod ML detection concurrency via a tokio
-  semaphore — tune the cap per-pod with the
-  `LLMTRACE_ML_MAX_CONCURRENT` env var (default `8`). Excess requests
-  receive `503 Service Unavailable` with `Retry-After: 1` instead of
-  every concurrent request stalling on contended CPU; the counter
-  `llmtrace_ml_rejected_total` and the gauge
-  `llmtrace_ml_inflight_requests` are exposed on `/metrics` for
-  alerting on sustained saturation.
+The library skips both the admin-key injection AND the operator-key
+bootstrap; `api_key` and `admin_key` are both `null` in the result. The
+proxy URL is then wide open; own the consequences.
+
+### Why two keys
+
+Before this PR the lifecycle layer handed the bootstrap admin key
+straight to the tenant. That key can mint more keys, view audit logs,
+manage feature flags, and create/delete tenants. A compromised tenant
+app would have taken the whole proxy with it. The operator role exists
+exactly to bound runtime traffic to "proxy LLM calls + report actions"
+without any control-plane scope. See
+`crates/llmtrace-core/src/lib.rs::ApiKeyRole` for the role definitions.
+
+### Caveats and follow-ups
+
+- **Dashboard wiring follow-up**: the Next.js dashboard
+  (`dashboard/src/lib/api.ts`, `dashboard/src/lib/proxy-helpers.ts`)
+  currently only reads `LLMTRACE_AUTH_ADMIN_KEY`. The lifecycle layer
+  injects `LLMTRACE_AUTH_RUNTIME_KEY` informationally; a separate PR
+  should switch the dashboard to use the runtime key for tenant-facing
+  traffic and the admin key only for admin pages.
+- **Admin key rotation** is opt-in via `rotate_admin_after_bootstrap:
+  true` in the tenant config — see the "Admin key rotation" section
+  below. Without it the bootstrap admin key lives in the proxy env for
+  the life of the deployment.
+- **Per-tenant rate limits** ship via the `rate_limit:` block in the
+  tenant config (see "Tenant config format"); when set, the proxy
+  honours `LLMTRACE_RATE_LIMIT_RPS` / `LLMTRACE_RATE_LIMIT_BURST`.
+- **DoS protection against CPU burn from ML detectors** is enforced by
+  an intra-pod tokio semaphore — tune with `LLMTRACE_ML_MAX_CONCURRENT`
+  (default `8`). Excess requests receive `503 Service Unavailable` with
+  `Retry-After: 1` instead of stalling on contended CPU; the counter
+  `llmtrace_ml_rejected_total` and gauge `llmtrace_ml_inflight_requests`
+  are exposed on `/metrics` for alerting on sustained saturation.
 
 ## Per-tenant secret injection
 
@@ -919,13 +948,14 @@ win).
 
 | File | Purpose |
 |---|---|
-| `lifecycle.py:66` | `ComponentSpec` dataclass — one component's full deployment shape |
-| `lifecycle.py:91` | `TenantSpec` dataclass — tenant's pair (incl. `enable_proxy_auth` + `api_key`) |
-| `lifecycle.py:55` | `generate_api_key()` — `llmt_<64-hex>` matching the Rust proxy |
-| `lifecycle.py:343` | `provision(spec)` |
-| `lifecycle.py:387` | `update(spec, proxy_id, dashboard_id, strategy)` |
-| `lifecycle.py:438` | `deprovision(tenant_id, proxy_id?, dashboard_id?)` |
-| `lifecycle.py:452` | `status(tenant_id, proxy_id?, dashboard_id?)` |
+| `lifecycle.py` | `ComponentSpec`, `TenantSpec`, `TenantInstances` (`api_key` + `admin_key`) |
+| `lifecycle.py` | `generate_api_key()` — `llmt_<64-hex>` matching the Rust proxy |
+| `lifecycle.py` | `_bootstrap_tenant_in_proxy`, `_mint_operator_key`, `_find_tenant_by_label`, `_find_operator_key_record`, `_verify_or_remint_operator_key` — admin HTTP boundary against `/api/v1/tenants` + `/api/v1/auth/keys` |
+| `lifecycle.py` | `provision(spec)` — bootstrap admin key, deploy proxy, mint operator key, deploy dashboard |
+| `lifecycle.py` | `update(spec, proxy_id, dashboard_id, strategy)` — recreate re-mints operator; restart verifies + re-mints only if missing |
+| `lifecycle.py` | `deprovision(tenant_id, proxy_id?, dashboard_id?)` |
+| `lifecycle.py` | `status(tenant_id, proxy_id?, dashboard_id?)` |
+| `deployments/basilica/tests/test_operator_key_minting.py` | Unit tests for the admin HTTP boundary, `provision()` integration, CLI serialisation |
 | `cli.py:40` | `_substitute_env` — `${VAR}` resolver |
 | `cli.py:60` | `_load_config` — YAML/JSON loader |
 | `cli.py:113` | `_tenant_spec_from_config` — dict → `TenantSpec` |

deployments/basilica/cli.py

@@ -168,6 +168,7 @@ def view(info: lifecycle.InstanceInfo | None) -> dict[str, Any] | None:
         "proxy_url": instances.proxy.url if instances.proxy else None,
         "dashboard_url": instances.dashboard.url if instances.dashboard else None,
         "api_key": instances.api_key,
+        "admin_key": instances.admin_key,
     }
 
 

deployments/basilica/configs/examples/pro.yaml

@@ -1,9 +1,13 @@
 # LLMTrace tenant config — pro tier example.
 # Multi-replica proxy, persistent dashboard, debug logging, datamarking on.
 #
-# Auth: per-tenant API key is auto-generated at provision time (or supply
-# `api_key:` at the top level to force a specific value). See starter.yaml
-# for the full auth contract.
+# Auth: two-tier key model. The lifecycle library bootstraps a tenant on
+# the live proxy after readiness, then mints a scoped Operator-role key
+# via POST /api/v1/auth/keys. That operator key is what the tenant uses
+# for runtime traffic. The bootstrap admin key (auto-generated or pinned
+# via `api_key:` below) is retained by the caller for self-service /
+# admin pages and NEVER handed to the tenant. See starter.yaml + the
+# README "Per-tenant API key auth" section for the full contract.
 
 proxy:
   image: "ghcr.io/techlab-innov/llmtrace-proxy:${LLMTRACE_VERSION:-latest}"

deployments/basilica/configs/examples/starter.yaml

@@ -8,13 +8,22 @@
 # from the process environment at deploy time. Use this to keep secrets out
 # of the config file.
 #
-# Auth: `enable_proxy_auth` defaults to true. On provision, the lifecycle
-# library either uses a caller-supplied `api_key` (top-level) or auto-generates
-# an `llmt_<64-hex>` key, injects it into both proxy + dashboard envs as
-# LLMTRACE_AUTH_ADMIN_KEY, and returns it in the result JSON. The plaintext
-# key is exposed once on provision — your app must persist it and ship it to
-# the tenant as their `Authorization: Bearer llmt_<key>` header. Set
-# `enable_proxy_auth: false` to deploy an open proxy.
+# Auth: `enable_proxy_auth` defaults to true. The lifecycle library issues
+# TWO keys at provision time (see README "Per-tenant API key auth"):
+#
+#   - admin_key  — bootstrap admin scope. Used by the lifecycle layer to
+#                  mint per-tenant keys + retained by the caller for the
+#                  self-service / admin portal. NEVER given to the tenant.
+#                  Sourced from `api_key:` below (if set) or `LLMTRACE_AUTH_ADMIN_KEY`
+#                  in proxy.env, or auto-generated `llmt_<64-hex>`.
+#   - api_key    — operator scope. Minted on the live proxy via
+#                  POST /api/v1/auth/keys after the proxy is ready. This is
+#                  the key handed to the tenant for runtime traffic; it
+#                  cannot manage tenants/keys/audit logs.
+#
+# Both plaintext keys are exposed once in the result JSON (`api_key` and
+# `admin_key`). Persist them in your secret store immediately. Set
+# `enable_proxy_auth: false` to deploy an open proxy with no auth at all.
 
 proxy:
   image: ghcr.io/techlab-innov/llmtrace-proxy:latest
@@ -48,18 +57,23 @@ dashboard:
     HOSTNAME: 0.0.0.0
     NODE_ENV: production
     # LLMTRACE_AUTH_ADMIN_KEY is set by the lifecycle library at provision
-    # time to match the proxy's resolved key. If you set it here explicitly,
-    # that value wins for the dashboard env (but the proxy's resolved key is
-    # what the tenant must actually send — keep them consistent).
+    # time to match the proxy's resolved admin key (used by dashboard
+    # server-side handlers for admin endpoints). LLMTRACE_AUTH_RUNTIME_KEY
+    # is set after the operator key is minted — dashboard wiring to
+    # consume the runtime key for tenant-facing traffic is a follow-up.
+    # If you set either var here explicitly, that value wins for the
+    # dashboard env.
 
 # Optional — override if your tenant naming scheme differs.
 # proxy_name_template: "llmtrace-proxy-{tenant_id}"
 # dashboard_name_template: "llmtrace-dashboard-{tenant_id}"
 # inject_proxy_url_into_dashboard: true
 # proxy_url_env_var: LLMTRACE_PROXY_URL
 
-# Auth controls (defaults shown). Set explicit `api_key` to force a specific
-# value (e.g. when recreating to preserve the existing tenant key).
+# Auth controls (defaults shown). `api_key` here pins the BOOTSTRAP ADMIN
+# key (so the admin_key stays stable across recreates). The runtime
+# operator key is always re-minted on recreate (the proxy's DB is fresh)
+# and verified-or-re-minted on restart (same volume).
 # enable_proxy_auth: true
 # api_key: null