feat(basilica): opt-in admin key rotation after bootstrap (#238)

Add `rotate_admin_key()` primitive + `rotate_admin_after_bootstrap`
TenantSpec field (default off) so callers can invalidate the bootstrap
admin key as soon as `provision()` returns. Wire it through the CLI as
the `rotate-admin-key` subcommand and the tenant-lifecycle workflow as a
new `action` choice. The workflow `::add-mask::`s `admin_key` before any
`cat result.json` line, same pattern as the existing `api_key` mask.

Mechanism: the Basilica SDK exposes no env-patch primitive
(create/delete/restart only). Rotation therefore deletes the existing
proxy UUID and creates a fresh one with the rotated env. As a
consequence, `proxy_instance_id` and `proxy.url` change; the result
JSON returns the post-rotation values which the caller must persist.

Trade-off: opt-in because the rotation adds one proxy re-roll (~30s)
to provisioning time. Recommended for production tenants where the
bootstrap admin key must be invalidated; safe to leave off for dev /
sandbox tenants.

Tests: 9 unit tests under deployments/basilica/tests/ exercising the
rotation logic with the Basilica SDK HTTP boundary mocked
(create_deployment / delete_deployment / get_deployment). Rotation
logic itself runs unmodified.

Author: epappas

.github/workflows/tenant-lifecycle.yml

@@ -16,6 +16,7 @@ on:
           - update
           - status
           - deprovision
+          - rotate-admin-key
       config_path:
         description: "Path in repo to a YAML/JSON tenant config (provision/update only)"
         required: false
@@ -44,6 +45,11 @@ on:
         options:
           - recreate
           - restart
+      new_admin_key:
+        description: "Optional explicit new admin key for rotate-admin-key (else auto-generated)"
+        required: false
+        type: string
+        default: ""
       tenant_secrets_json:
         description: >-
           JSON object of per-tenant secrets to inject as env vars before the
@@ -106,6 +112,7 @@ jobs:
           WF_PROXY_ID: ${{ inputs.proxy_instance_id }}
           WF_DASHBOARD_ID: ${{ inputs.dashboard_instance_id }}
           WF_STRATEGY: ${{ inputs.strategy }}
+          WF_NEW_ADMIN_KEY: ${{ inputs.new_admin_key }}
           WF_SECRETS: ${{ inputs.tenant_secrets_json }}
           RD_PAYLOAD: ${{ toJson(github.event.client_payload) }}
         run: |
@@ -121,6 +128,7 @@ jobs:
               proxy_id  = os.environ.get("WF_PROXY_ID", "")
               dash_id   = os.environ.get("WF_DASHBOARD_ID", "")
               strategy  = os.environ.get("WF_STRATEGY", "")
+              new_admin_key = os.environ.get("WF_NEW_ADMIN_KEY", "")
               secrets_json = os.environ.get("WF_SECRETS", "{}") or "{}"
               try:
                   secrets = json.loads(secrets_json)
@@ -137,13 +145,20 @@ jobs:
               proxy_id  = payload.get("proxy_instance_id", "")
               dash_id   = payload.get("dashboard_instance_id", "")
               strategy  = payload.get("strategy", "recreate")
+              new_admin_key = payload.get("new_admin_key", "") or ""
               secrets   = payload.get("tenant_secrets", {}) or {}
           else:
               raise SystemExit(f"::error::unsupported event {event!r}")
 
           if not isinstance(secrets, dict):
               raise SystemExit("::error::tenant_secrets must be a JSON object")
 
+          # Mask the optional explicit new admin key from the workflow input
+          # surface — it ends up as a CLI arg downstream and must not show up
+          # in subsequent log lines.
+          if new_admin_key:
+              print(f"::add-mask::{new_admin_key}")
+
           env_path = pathlib.Path(os.environ["GITHUB_ENV"])
           with env_path.open("a") as fh:
               fh.write(f"TENANT_ID={tenant_id}\n")
@@ -152,6 +167,7 @@ jobs:
               fh.write(f"PROXY_ID={proxy_id}\n")
               fh.write(f"DASHBOARD_ID={dash_id}\n")
               fh.write(f"STRATEGY={strategy}\n")
+              fh.write(f"NEW_ADMIN_KEY={new_admin_key}\n")
               # config_json may contain newlines; use heredoc form
               if cfg_json:
                   fh.write(f"CFG_JSON<<__LIFECYCLE_EOF__\n{cfg_json}\n__LIFECYCLE_EOF__\n")
@@ -219,6 +235,16 @@ jobs:
             status|deprovision)
               :
               ;;
+            rotate-admin-key)
+              if [[ -z "${CFG_PATH:-}" && -z "${CFG_JSON:-}" ]]; then
+                echo "::error::action=rotate-admin-key requires config_path or config_json"
+                exit 2
+              fi
+              if [[ -z "${PROXY_ID:-}" ]]; then
+                echo "::error::action=rotate-admin-key requires proxy_instance_id"
+                exit 2
+              fi
+              ;;
             *)
               echo "::error::unknown or missing action: ${ACTION:-<empty>}"
               exit 2
@@ -252,6 +278,17 @@ jobs:
               [[ -n "${PROXY_ID:-}" ]] && args+=("--proxy-instance-id" "${PROXY_ID}")
               [[ -n "${DASHBOARD_ID:-}" ]] && args+=("--dashboard-instance-id" "${DASHBOARD_ID}")
               ;;
+            rotate-admin-key)
+              args+=("--proxy-instance-id" "${PROXY_ID}")
+              if [[ -n "${CFG_JSON:-}" ]]; then
+                args+=("--config-json" "${CFG_JSON}")
+              else
+                args+=("--config" "${CFG_PATH}")
+              fi
+              if [[ -n "${NEW_ADMIN_KEY:-}" ]]; then
+                args+=("--new-key" "${NEW_ADMIN_KEY}")
+              fi
+              ;;
           esac
 
           set +e

deployments/basilica/README.md

@@ -438,6 +438,84 @@ without any control-plane scope. See
   `llmtrace_ml_rejected_total` and gauge `llmtrace_ml_inflight_requests`
   are exposed on `/metrics` for alerting on sustained saturation.
 
+### Admin key rotation
+
+After `provision`, the bootstrap admin key lives in the proxy pod env
+indefinitely. The key was returned once on the workflow run output and
+flows through the calling app's secret-handling path before reaching the
+tenant. If you treat the workflow output / app log surface as a higher
+exposure boundary than the live Basilica pod env, you can **rotate the
+admin key after bootstrap** so the value returned at provision time is
+no longer the live key.
+
+**When to enable.** Production tenants where you want the bootstrap admin
+key invalidated as soon as the tenant has it (or you don't trust the
+single-channel return path). Skip for dev / sandbox tenants — the
+rotation is opt-in because it costs an extra proxy re-roll.
+
+**Trade-off.** Rotation deletes the bootstrap proxy and creates a fresh
+one with the rotated `LLMTRACE_AUTH_ADMIN_KEY`. That adds ~30s to
+provisioning time, and **the proxy's `instance_id` and `url` change** as
+a result (Basilica's SDK has no env-patch primitive — see *Lifecycle
+operations in detail* below). The caller MUST persist the post-rotation
+`proxy_instance_id` and `proxy_url` over the bootstrap values; the
+result JSON returns the post-rotation UUID/URL.
+
+**Enable via YAML config:**
+
+```yaml
+rotate_admin_after_bootstrap: true
+```
+
+With the flag set, `provision` runs as usual, then internally calls
+`rotate_admin_key(proxy_instance_id=...)` against the just-created proxy,
+regenerates a fresh `llmt_<64-hex>` key, and returns that as the
+`api_key` field in the result JSON. The bootstrap key never leaves the
+library boundary.
+
+**Invoke the rotation independently** (e.g. for an already-running
+tenant whose admin key you want to roll on a schedule):
+
+```bash
+python -m deployments.basilica.cli rotate-admin-key \
+  --tenant-id acme \
+  --config deployments/basilica/configs/examples/starter.yaml \
+  --proxy-instance-id <current_proxy_uuid>
+# optional: --new-key llmt_...   to force a specific value
+```
+
+Result JSON shape:
+
+```json
+{
+  "tenant_id": "acme",
+  "proxy_instance_id": "<new_uuid>",
+  "proxy_url": "https://<new_uuid>.deployments.basilica.ai",
+  "admin_key": "llmt_..."
+}
+```
+
+**Dispatch via the workflow:**
+
+```bash
+gh api -X POST /repos/techlab-innov/llmtrace/dispatches \
+  --raw-field event_type=tenant-lifecycle \
+  --raw-field 'client_payload={
+    "tenant_id":"acme","action":"rotate-admin-key",
+    "config_path":"deployments/basilica/configs/examples/starter.yaml",
+    "proxy_instance_id":"<current_proxy_uuid>"
+  }'
+```
+
+The workflow `::add-mask::`s the returned `admin_key` before any
+`cat result.json` line, same as it does for `api_key` on provision. The
+key is then available as a step output (`steps.run.outputs.admin_key`)
+for the dispatching app to consume via the Actions API.
+
+**Idempotency.** Re-running rotation on an already-rotated tenant
+generates a fresh key (it's not a no-op). The caller must serialise
+rotation per tenant to avoid concurrent delete+create races.
+
 ## Per-tenant secret injection
 
 Two trigger paths, pick by intended audience:

deployments/basilica/cli.py

@@ -142,6 +142,7 @@ def _tenant_spec_from_config(tenant_id: str, cfg: dict[str, Any]) -> lifecycle.T
         "proxy_url_env_var",
         "enable_proxy_auth",
         "api_key",
+        "rotate_admin_after_bootstrap",
     ):
         if key in cfg:
             kwargs[key] = cfg[key]
@@ -172,6 +173,16 @@ def view(info: lifecycle.InstanceInfo | None) -> dict[str, Any] | None:
     }
 
 
+def _serialise_rotation(result: lifecycle.RotationResult) -> dict[str, Any]:
+    return {
+        "tenant_id": result.tenant_id,
+        "proxy": dataclasses.asdict(result.proxy),
+        "proxy_instance_id": result.proxy.instance_id,
+        "proxy_url": result.proxy.url,
+        "admin_key": result.admin_key,
+    }
+
+
 def _build_parser() -> argparse.ArgumentParser:
     parser = argparse.ArgumentParser(
         prog="deployments.basilica.cli",
@@ -229,10 +240,33 @@ def add_ids(p: argparse.ArgumentParser, required: bool) -> None:
     add_tenant(p_dep)
     add_ids(p_dep, required=False)
 
+    p_rot = sub.add_parser(
+        "rotate-admin-key",
+        help=(
+            "Rotate the proxy admin key on a live deployment. "
+            "Delete+recreate under the hood — proxy_instance_id and URL change."
+        ),
+    )
+    add_tenant(p_rot)
+    add_config(p_rot)
+    p_rot.add_argument(
+        "--proxy-instance-id",
+        required=True,
+        help="Basilica UUID for the existing proxy deployment",
+    )
+    p_rot.add_argument(
+        "--new-key",
+        required=False,
+        default=None,
+        help="Optional explicit new admin key (else a fresh llmt_<64-hex> is generated)",
+    )
+
     return parser
 
 
-def _dispatch(args: argparse.Namespace) -> lifecycle.TenantInstances:
+def _dispatch(
+    args: argparse.Namespace,
+) -> lifecycle.TenantInstances | lifecycle.RotationResult:
     if args.action == "provision":
         cfg = _load_config(args.config, args.config_json)
         spec = _tenant_spec_from_config(args.tenant_id, cfg)
@@ -258,6 +292,16 @@ def _dispatch(args: argparse.Namespace) -> lifecycle.TenantInstances:
             proxy_instance_id=args.proxy_instance_id,
             dashboard_instance_id=args.dashboard_instance_id,
         )
+    if args.action == "rotate-admin-key":
+        cfg = _load_config(args.config, args.config_json)
+        spec = _tenant_spec_from_config(args.tenant_id, cfg)
+        return lifecycle.rotate_admin_key(
+            tenant_id=args.tenant_id,
+            proxy_instance_id=args.proxy_instance_id,
+            proxy_spec=spec.proxy,
+            new_key=args.new_key,
+            proxy_name_template=spec.proxy_name_template,
+        )
     raise SystemExit(f"unknown action {args.action!r}")
 
 
@@ -276,7 +320,11 @@ def main(argv: list[str] | None = None) -> int:
         json.dump({"error": str(exc), "action": args.action}, sys.stdout)
         sys.stdout.write("\n")
         return 3
-    json.dump(_serialise(result), sys.stdout, indent=2, sort_keys=True)
+    if isinstance(result, lifecycle.RotationResult):
+        payload = _serialise_rotation(result)
+    else:
+        payload = _serialise(result)
+    json.dump(payload, sys.stdout, indent=2, sort_keys=True)
     sys.stdout.write("\n")
     return 0
 

deployments/basilica/configs/examples/pro.yaml

@@ -52,3 +52,7 @@ dashboard:
 # rate_limit:
 #   requests_per_second: 500
 #   burst_size: 1000
+
+# Rotate the admin key once the proxy is up. Adds ~30s and changes the
+# proxy instance_id + url; recommended for production. Opt-in.
+# rotate_admin_after_bootstrap: true

deployments/basilica/configs/examples/starter.yaml

@@ -84,3 +84,10 @@ dashboard:
 # rate_limit:
 #   requests_per_second: 50
 #   burst_size: 100
+
+# rotate_admin_after_bootstrap: false
+# ↑ Set to `true` for production tenants where the bootstrap admin key
+#   must be invalidated immediately after provision. Adds ~30s to
+#   provisioning (extra proxy re-roll) and changes the proxy
+#   instance_id + url — caller must persist the post-rotation values.
+#   See README.md → "Admin key rotation".