Multi-tenant Cloudflare subrouter with tenant registry and account upload API (#52)

* Multi-tenant Cloudflare subrouter: tenant registry, hashed srt_ keys, account upload API

- TenantRegistryDurableObject: create/list/rotate/revoke with sha256-hashed
  srt_* keys, admin-token gated
- Data-plane auth requires a tenant key; per-tenant DO state namespaced as
  tenant:<id> so registry ids can never collide with the bare legacy/demo-org DOs
- POST/GET/DELETE /tenant/accounts for Claude OAuth, Anthropic API key,
  Codex OAuth, OpenAI API key; responses sanitized, refresh alarms reused
- Legacy shared PROXY_TOKEN disabled unless ALLOW_LEGACY_PROXY_TOKEN=1,
  routes only to tenant 'legacy'
- Tenant-key headers stripped from upstream + transcript paths
- Tests: lifecycle, upload, isolation (incl. hostile X-Subrouter-Org-ID and
  body orgId spoofing), legacy-token default-off, reserved-name collision

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address PR 52 review findings: provider-aware routing, quota pool seeding, validation timeout, shared test helpers

- Seed bootstrap quota pools on tenant account upload (anthropic: opus+sonnet, codex: spark) so uploaded accounts serve pooled models immediately
- Filter account selection by upstream provider family on every proxy, /route, and WS route path (incl. sticky/preferred) so mixed-provider tenants never cross upstreams
- Bounded, injectable timeout on ?validate=1 upstream usage fetch
- Shared worker test helper with process-tree teardown + temp dir cleanup; isolation tests use default-pool models so the provider filter itself is exercised

* Restore drain-aware readiness per tenant

/_subrouter/ready?tenant=<id> consults the tenant DO's drain state (503 while draining, boolean only, no tenant data); bare /_subrouter/ready stays service-level

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: lawrencecchen

cloudflare/packages/worker/README.md

@@ -1,9 +1,9 @@
 # @subrouter/cloudflare-worker
 
-Cloudflare **Durable Objects** subrouter: a per-org subrouter service with
-sticky session affinity, per-model-family quota pools, Go-compatible
-`/_subrouter/*` admin endpoints, and upstream proxying for Codex/OpenAI and
-Claude/Anthropic accounts.
+Cloudflare **Durable Objects** subrouter: a multi-tenant routing service with
+hashed tenant keys, per-tenant Durable Object state, sticky session affinity,
+OAuth refresh alarms, Go-compatible `/_subrouter/*` admin endpoints, and
+upstream proxying for Codex/OpenAI and Claude/Anthropic accounts.
 
 ## Environments
 
@@ -13,8 +13,155 @@ Claude/Anthropic accounts.
 | staging    | `regatta-subrouter-do-staging`    | https://subrouter-staging.cmux.dev   |
 | production | `regatta-subrouter-do-production` | https://subrouter.cmux.dev           |
 
-Each env is a distinct Worker with its own Durable Object namespace (isolated
-SQLite state), its own admin/proxy tokens, and its own Durable Object alarms.
+Each env is a distinct Worker with its own Durable Object namespaces and
+SQLite state. Registry tenants store tenant-scoped accounts, sticky sessions,
+usage, transcripts, quotas, lifecycle, and alarms under
+`SUBROUTER_DO.getByName("tenant:" + <tenant id>)`; historical bare names such
+as `legacy` and `demo-org` remain reserved for compatibility state.
+`TENANT_REGISTRY_DO.getByName("registry")` stores only tenant metadata and
+SHA-256 hashes of tenant keys.
+
+## Auth Model
+
+`ADMIN_TOKEN` gates `/admin/*`, `/_subrouter/*`, and `/websocket-status`.
+
+Data-plane requests require a tenant key in either:
+
+```sh
+Authorization: Bearer srt_<32 lowercase hex chars>
+X-Subrouter-Tenant-Key: srt_<32 lowercase hex chars>
+```
+
+Tenant keys are returned only when created or rotated. The registry stores
+`sha256(key)` hex, never the plaintext key. Revoked keys return `403`; missing,
+malformed, unknown, and rotated-away keys return `401`.
+
+Legacy shared-token access is disabled by default. To temporarily allow the old
+`PROXY_TOKEN` path, set `ALLOW_LEGACY_PROXY_TOKEN=1` and `PROXY_TOKEN`; those
+requests route only to tenant id `legacy`.
+
+## Tenant Lifecycle
+
+Create a tenant:
+
+```sh
+curl -sS https://subrouter.cmux.dev/admin/tenants \
+  -H "Authorization: Bearer $ADMIN_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"Acme"}'
+```
+
+Response:
+
+```json
+{"id":"acme","name":"Acme","key":"srt_0123456789abcdef0123456789abcdef","created_at":1783123200000,"revoked_at":null}
+```
+
+List tenants, without keys or hashes:
+
+```sh
+curl -sS https://subrouter.cmux.dev/admin/tenants \
+  -H "Authorization: Bearer $ADMIN_TOKEN"
+```
+
+Rotate a tenant key:
+
+```sh
+curl -sS -X POST https://subrouter.cmux.dev/admin/tenants/acme/rotate \
+  -H "Authorization: Bearer $ADMIN_TOKEN"
+```
+
+Revoke a tenant:
+
+```sh
+curl -sS -X POST https://subrouter.cmux.dev/admin/tenants/acme/revoke \
+  -H "Authorization: Bearer $ADMIN_TOKEN"
+```
+
+## Account Upload API
+
+Upload Claude OAuth credentials:
+
+```sh
+curl -sS https://subrouter.cmux.dev/tenant/accounts \
+  -H "Authorization: Bearer $TENANT_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"provider":"claude","label":"alice@example.com","claudeAiOauth":{"accessToken":"...","refreshToken":"...","expiresAt":1735689600000,"subscriptionType":"max","rateLimitTier":"..."}}'
+```
+
+Upload an Anthropic API key:
+
+```sh
+curl -sS https://subrouter.cmux.dev/tenant/accounts \
+  -H "Authorization: Bearer $TENANT_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"provider":"anthropic-apikey","label":"anthropic prod","apiKey":"sk-ant-..."}'
+```
+
+Upload Codex OAuth tokens:
+
+```sh
+curl -sS https://subrouter.cmux.dev/tenant/accounts \
+  -H "Authorization: Bearer $TENANT_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"provider":"codex","label":"codex@example.com","tokens":{"accessToken":"...","refreshToken":"...","idToken":"...","accountID":"..."}}'
+```
+
+Upload an OpenAI API key:
+
+```sh
+curl -sS https://subrouter.cmux.dev/tenant/accounts \
+  -H "Authorization: Bearer $TENANT_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{"provider":"openai-apikey","label":"openai prod","apiKey":"sk-..."}'
+```
+
+Responses are sanitized:
+
+```json
+{"id":"claude-...","provider":"claude","label":"alice@example.com","auth_mode":"oauth","created_at":"2026-07-03T00:00:00.000Z","email":"alice@example.com"}
+```
+
+List and delete accounts:
+
+```sh
+curl -sS https://subrouter.cmux.dev/tenant/accounts \
+  -H "Authorization: Bearer $TENANT_KEY"
+
+curl -sS -X DELETE https://subrouter.cmux.dev/tenant/accounts/$ACCOUNT_ID \
+  -H "Authorization: Bearer $TENANT_KEY"
+```
+
+OAuth uploads schedule the existing Durable Object alarm refresh path. Rotated
+refresh tokens are persisted in the tenant DO.
+
+## Admin Tenant Data
+
+Tenant-scoped admin reads require an explicit `?tenant=<id>`:
+
+```sh
+curl -sS "https://subrouter.cmux.dev/_subrouter/accounts?tenant=acme" \
+  -H "Authorization: Bearer $ADMIN_TOKEN"
+
+curl -sS "https://subrouter.cmux.dev/_subrouter/transcripts?tenant=acme" \
+  -H "Authorization: Bearer $ADMIN_TOKEN"
+```
+
+`GET /_subrouter/health` and bare `GET /_subrouter/ready` are
+unauthenticated and do not return tenant data. `GET /_subrouter/ready?tenant=<id>`
+is also unauthenticated and returns only `{ ok, draining }` for load-balancer
+and rollout probes; malformed tenant ids return `400`, while well-formed unknown
+ids return ready because no tenant state exists yet.
+
+## cmux CLI Call Sequence
+
+The follow-up cmux integration should call:
+
+1. Admin side: `POST /admin/tenants` to create the tenant and show the key once.
+2. Tenant side: `POST /tenant/accounts` to upload Claude/Codex OAuth or API-key credentials.
+3. Tenant side: `GET /tenant/accounts` to show sanitized configured accounts.
+4. Tenant side: `DELETE /tenant/accounts/:id` to remove an account.
+5. Data plane: proxy requests with `Authorization: Bearer $TENANT_KEY`.
 
 ## Deploy
 
@@ -23,76 +170,56 @@ bun run deploy:staging
 bun run deploy:production
 ```
 
-GitHub Actions runs `Cloudflare Durable Object` for changes under
-`cloudflare/**`. Pull requests run install, typecheck, tests, and a Wrangler
-dry-run bundle validation. Pushes to `main` deploy staging first, smoke
-`https://subrouter-staging.cmux.dev/healthz`, then deploy production and smoke
-`https://subrouter.cmux.dev/healthz`.
-
 Required Actions secrets:
 
 - `CLOUDFLARE_ACCOUNT_ID` - Cloudflare account ID for Wrangler.
 - `CLOUDFLARE_API_TOKEN` - long-lived Cloudflare API token with Workers deploy
   permissions for both `regatta-subrouter-do-staging` and
   `regatta-subrouter-do-production`.
 
-## Secrets
+## Local Dev
 
-`ADMIN_TOKEN` gates the `/admin/*`, `/_subrouter/*`, and `/websocket-status`
-endpoints. `PROXY_TOKEN` gates upstream proxy requests when present; if it is
-unset, proxy requests use `ADMIN_TOKEN`. Set a separate proxy token when clients
-should not share the admin credential:
+Create `.dev.vars` (gitignored) with at least:
 
 ```sh
-bun run secret:admin:staging
-bun run secret:admin:production
-bun run secret:proxy:staging
-bun run secret:proxy:production
+ADMIN_TOKEN=<anything>
 ```
 
-## Local dev
-
-Create `.dev.vars` (gitignored) with `ADMIN_TOKEN=<anything>` and
-`PROXY_TOKEN=<anything>` then:
+Optional legacy compatibility:
 
 ```sh
-bun run dev                  # local workerd + DO SQLite on :8787
+PROXY_TOKEN=<anything>
+ALLOW_LEGACY_PROXY_TOKEN=1
 ```
 
-## OAuth refresh
-
-OAuth account credentials can include `accessToken`, `refreshToken`,
-`expiresAt`, `tokenEndpoint`, and `clientId`. The Durable Object refreshes
-expired Codex and Claude OAuth credentials on use, persists rotated refresh
-tokens atomically in SQLite, and schedules the next refresh with the Durable
-Object Alarms API. Alarms wake the object in the future; long sleeps are not
-used.
-
-## Logs
+Then run:
 
 ```sh
-bun run tail:staging         # wrangler tail --env staging
-bun run tail:production
+bun run dev
 ```
 
 ## Endpoints
 
 - `GET /healthz`
 - `GET /_subrouter/health`
-- `GET /_subrouter/ready`
-- `POST /_subrouter/drain`
-- `GET /_subrouter/drain-status`
-- `GET /_subrouter/accounts`
-- `GET|POST /_subrouter/account-status`
-- `GET /_subrouter/usage-status`
-- `POST /_subrouter/reload-accounts`
-- `GET /_subrouter/sessions`
-- `GET /_subrouter/dashboard`
-- `GET /_subrouter/transcripts`
-- `GET /_subrouter/transcripts/:agent/:session`
-- `GET /_subrouter/transcripts/:agent/:session/raw`
-- `POST /route` — `{ orgId, sessionId, model?, preferAccountId?, quotaKey? }` → selected account
-- `POST /usage` — `{ orgId, sessionId, accountId }`
-- `GET /status?orgId=` — DO route counters
-- `GET /ws?orgId=` — WebSocket; messages `{ type: "ping"|"route"|"usage"|"status", ... }`
-- `GET /admin/accounts`, `POST /admin/accounts`, `POST /admin/model-probe`, `GET /websocket-status` — require `Authorization: Bearer $ADMIN_TOKEN`
+- `GET /_subrouter/ready` - service-level readiness, no tenant data.
+- `GET /_subrouter/ready?tenant=` - tenant drain readiness, returns 503 when draining; malformed tenant ids return 400.
+- `POST /admin/tenants`, `GET /admin/tenants`
+- `POST /admin/tenants/:id/rotate`, `POST /admin/tenants/:id/revoke`
+- `POST /tenant/accounts`, `GET /tenant/accounts`, `DELETE /tenant/accounts/:id`
+- `POST /_subrouter/drain?tenant=`
+- `GET /_subrouter/drain-status?tenant=`
+- `GET /_subrouter/accounts?tenant=`
+- `GET|POST /_subrouter/account-status?tenant=`
+- `GET /_subrouter/usage-status?tenant=`
+- `POST /_subrouter/reload-accounts?tenant=`
+- `GET /_subrouter/sessions?tenant=`
+- `GET /_subrouter/dashboard?tenant=`
+- `GET /_subrouter/transcripts?tenant=`
+- `GET /_subrouter/transcripts/:agent/:session?tenant=`
+- `GET /_subrouter/transcripts/:agent/:session/raw?tenant=`
+- `POST /route` - tenant-key authed route selection.
+- `POST /usage` - tenant-key authed usage recording.
+- `GET /status` - tenant-key authed DO route counters.
+- `GET /ws` - tenant-key authed WebSocket; messages `{ type: "ping"|"route"|"usage"|"status", ... }`.
+- `GET /websocket-status?tenant=`, `GET /admin/accounts?tenant=`, `POST /admin/accounts`, `POST /admin/model-probe` - require `Authorization: Bearer $ADMIN_TOKEN`.

cloudflare/packages/worker/src/contract.ts

@@ -117,6 +117,7 @@ const subrouterHeadersToStrip = [
   "X-Model",
   "X-Subrouter-Org-ID",
   "X-Regatta-Org-ID",
+  "X-Subrouter-Tenant-Key",
   "X-Subrouter-Proxy-Token",
 ] as const