docs(ops): stable-mode Phase 4 — operations runbook + uptime monitoring by cloakmaster · Pull Request #64 · cloakmaster/foxbook

cloakmaster · 2026-05-04T18:29:00Z

Phase 4 of the stable-mode plan. Master runbook + automated uptime monitoring so the substrate runs unattended for 12+ months without rebuilding mental context. No code changes; no behavior changes.

What's in

File	Status	Purpose
docs/OPERATIONS.md	NEW (~280 lines)	Master runbook: cost, redeploy, key rotation, incident response, backups, monitoring, future-you sanity check
.github/workflows/uptime.yml	NEW	15-min cron pinging 3 endpoints; opens / closes GitHub issues on failure / recovery
apps/api/fly.toml	MODIFIED	Cross-reference comment pointing at OPERATIONS.md
docs/operations/env-vars.md	REWRITTEN	Stripped Vercel / Upstash / KMS / MCP scaffolding that never shipped; pinned to actual production reality (4 Fly secrets + 1 Worker secret)

OPERATIONS.md sections

Future-you sanity check — 5 commands to run when something breaks, before changing anything
Monthly cost summary — target <$30/mo with line-by-line breakdown + spike triggers
Re-deploy each service — extracted from existing fly.toml + wrangler.toml + scripts; single source of truth
Key rotation — transparency-log signing key procedure + per-agent recovery / rotation pointers
Incident response — three concrete scenarios (log corruption, forged claim, leaked secret)
Database backups — Neon PITR by tier; restore procedure
Uptime monitoring — workflow + silencing alerts during planned maintenance

Uptime workflow

Triggers: cron `*/15 * * * *` (UTC) + manual `workflow_dispatch`
Endpoints: `transparency.foxbook.dev/root`, `api.foxbook.dev/healthz`, `foxbook.dev/`
On fail: opens issue tagged `uptime-incident` (or comments on existing); maintainer's GitHub email notifications page automatically
On recovery: comments + auto-closes the incident issue
Security: all matrix values passed via `env:` blocks rather than direct `${{ }}` interpolation in run: blocks (defense-in-depth against future user-derived matrix entries; matrix is static config today)
Free under GitHub Actions free tier (~3 hours runner time/month at 15-min interval)

env-vars.md cleanup

Production reality is 5 secrets total:

Fly.io: `DATABASE_URL`, `DATABASE_URL_DIRECT`, `RESEND_API_KEY`, `FOXBOOK_LOG_SIGNING_KEY_HEX`
Cloudflare Workers: `DATABASE_URL` (same Neon pooled URL as Fly)

Pre-stable-mode env-vars.md listed Vercel/Upstash/KMS/MeiliSearch/MCP servers that were planned but never shipped. Removed under stable-mode discipline (the doc should reflect reality, not aspirational scaffolding).

Verification

`pnpm check:generated` — 10 files match
`pnpm check:core-isolation` — clean
`pnpm -r typecheck` — clean
No code touched (docs + workflow only)
CI green
After merge: uptime.yml first run within 15 min should produce a green check (or open an incident if any endpoint is genuinely down — see OPERATIONS.md § Future-you sanity check)

What's NOT in this PR

Cloudflare Pages landing page at foxbook.dev — Phase 5
Final Discussion #1803 announcement — Phase 5, deferred until Phase 4.5 (eriknewton spec session → ADR 0009 cross-impl integration shape) lands first. Per Cowork's strategic guidance, the announcement materially upgrades from "stable, future TBD" to "stable substrate other systems compose against" once Concordia + Sanctuary integration is documented.

Refs

Plan: `/Users/tester/.claude/plans/focus-deeply-and-use-nifty-swan.md`
ADR 0008 — stable-mode posture (Phase 2)
Phase 1: shipped 2026-05-03 (feat(firehose): self-test wedge defence + alive sentinel + decode-error robustness #42, feat(schema): tl-leaf v1.2 — signing-key-registration (rotation) shape #60, feat(sdk-claim): v0.2.0 — verified_signing_key_hex + structured reason_code #61)
Phase 2: docs lockdown (docs: stable-mode Phase 2 — documentation lockdown #62)
Phase 3 brand: NOTICE + TRADEMARK in docs: stable-mode Phase 2 — documentation lockdown #62; identity-bound terminal steps confirmed
Phase 4: this PR
Phase 4.5: ADR 0009 after eriknewton spec session (gates Phase 5)
Phase 5: deferred until 4.5 lands

Phase 4 of the stable-mode plan. Master runbook for keeping transparency.foxbook.dev / api.foxbook.dev / foxbook.dev running unattended at low cost. No code changes; no behavior changes. ## What's in this PR - docs/OPERATIONS.md (NEW) — master runbook covering: - Future-you sanity check (5 commands to run when something breaks) - Monthly cost summary (target <$30/mo) - Re-deploy each service (Fly.io api, Cloudflare Worker, Pages landing) - Key rotation (transparency-log signing key + per-agent keys) - Incident response (log corruption, forged claim, leaked secret) - Database backups (Neon PITR by tier) - Uptime monitoring (this PR's workflow + silencing alerts) - Phase 5 stable-mode landing notes - Cross-references to ADRs + config files - .github/workflows/uptime.yml (NEW) — 15-min cron, three endpoints, opens / comments-on / closes GitHub issues tagged `uptime-incident`. All matrix values passed via env: rather than direct ${{ }} interpolation (defense-in-depth against future user-derived matrix entries; matrix is static config today). - apps/api/fly.toml — added cross-reference comment pointing at OPERATIONS.md for re-deploy / rollback / key-rotation procedures. Existing 72-line first-time-setup runbook preserved. - docs/operations/env-vars.md — rewritten to reflect production reality (Fly + Cloudflare Worker secrets, four actual variables). Removed Vercel / Upstash / KMS / MCP sections that were earlier scaffolding never shipped. Added pointer to OPERATIONS.md for the broader runbook. ## Verification - pnpm check:generated — 10 files match - pnpm check:core-isolation — clean - pnpm -r typecheck — clean - No code touched (docs + workflow only) ## What's NOT in this PR - Cloudflare Pages landing page itself (foxbook.dev) — Phase 5 - Final Discussion #1803 announcement — Phase 5, deferred until Phase 4.5 (eriknewton spec session + ADR 0009) lands first ## Refs - Plan: /Users/tester/.claude/plans/focus-deeply-and-use-nifty-swan.md - ADR 0008 — stable-mode posture (Phase 2) - Phase 1: shipped 2026-05-03 (#42, #60, #61) - Phase 2: docs lockdown (#62, #63) - Phase 3: brand protection (NOTICE + TRADEMARK in #62; identity-bound terminal steps confirmed by maintainer) - Phase 4: this PR - Phase 4.5: ADR 0009 after eriknewton spec session (gates Phase 5) - Phase 5: deferred until 4.5 lands

…ed 250ms wait (#67) Earlier 250ms wall-clock wait was too tight under CI runner load — flaked on PRs #62 and #64 (different commits, same assertion). Replaced with poll-until-condition up to 2s max: - Fast path: ~100ms on healthy runs (no change in test wall time) - Slow path: tolerates up to 2s on degraded CI before declaring failure - Same assertions retained — sentinel log + reconnect log + resubscribe Bug fix; allowed under stable mode (ADR 0008 freezes features, not test hardening). No production behavior change. Co-authored-by: Ben <ben@inkog.io>

@master

…93) (#94) * chore(lint): clear biome warnings + remove dead code, gate CI on warnings Drive the biome warning count to zero without changing behavior: - Export base64urlDecode from @foxbook/core and drop the byte-identical duplicate copy in apps/api/src/claim/handlers.ts (import from core). - Remove 2 dead biome-ignore comments in validators capability.test.ts and the dead eslint-disable in apps/api/src/firehose/listener.ts (the repo has no eslint, and the loop condition reads `stopped` so it was never a constant condition). - Scope lint/style/noNonNullAssertion off for TEST globs only (so test indexing bangs stop warning) — kept on for src, fixing the real src non-null assertions (sha256, jws, claim/repository, merkle-repository, smoke-test-revoke) with bounded `?? 0` / destructure-guard rewrites. - Apply the one safe useTemplate fix in claim.repository.test.ts. - Add --error-on-warnings to the biome step in CI so the count can't silently creep back up. Verification: `pnpm biome check --error-on-warnings` (clean, exit 0), `pnpm typecheck` (24/24), `pnpm --filter @foxbook/api test` (86 pass), `pnpm --filter @foxbook/core test` (59 pass), validators (72 pass), plus core-isolation / no-deskduck / generated pre-commit checks. * docs: fix doc inaccuracies (verifyAgentCard arity, firehose default, env vars, ADR count, 0007 header) - README hero + RFC + did:foxbook samples: add required asset_type to verifyAgentCard options - README: move firehose out of Live; it is disabled by default since #84 (FOXBOOK_FIREHOSE_ENABLED) - env-vars.md + .env.example: document FOXBOOK_FIREHOSE_ENABLED; .env.example adds DATABASE_URL_DIRECT + FOXBOOK_LOG_SIGNING_KEY_HEX, drops stale VERCEL_TOKEN/SENTRY_AUTH_TOKEN + Day-N prose - RATIONALE.md: ADR count eight->nine, add ADR 0009 line, fix misattributed 60s cache-policy sentence (API dynamic-state, not all read endpoints) - STATE-AT-STABLE-MODE-CLOSE.md: fix truncated Discussion #73 link - ADR 0007: add standard metadata block to match other ADRs * docs(spec): reconcile did:foxbook method spec to shipped code The did:foxbook method spec advertised behaviour the reference deployment does not implement. Filing to W3C as-is would describe a method that cannot be resolved. Downgrade the spec to describe what actually ships (no new endpoints — ADR 0008 freeze): - §1.1/§4.4/§6.2/§9: the signing path uses insertion-order canonical JSON (JSON.stringify semantics with caller-fixed key order; ADR 0005, core/src/crypto/canonical.ts, packages/db/src/merkle-repository.ts), NOT RFC 8785 JCS. The JCS canonicalizer in core/src/crypto/jcs.ts is a separate interop-only path, not used to sign leaves/STHs/JWS. Move RFC 8785 from Normative to Informative. Mirror the same fix in docs/specs/W3C-REGISTRY-SUBMISSION.md. - §3.2/§3.2.4/§7.1: there is no GET /agents/<did> resolver and no /.well-known/jwks.json. Rewrite resolution as a client-side projection over the real endpoints: /api/v1/claim/by-handle/..., /leaf/<index>, /inclusion/<index>, /root, and /.well-known/foxbook.json (where log_signing_public_key_hex lives). - §4.2: consistency proofs are GET /consistency?old=N&new=M (query params), not /consistency/<from>/<to> path segments. - §3.3/§3.4: rewrite the rotation and revocation leaf examples to match schemas/tl-leaf.v1.json exactly (signing-key-registration uses prior_ed25519_public_key_hex + new_ed25519_public_key_hex + recovery_key_signature + published_at, dropping previous_leaf_index; revocation uses revoked_key_hex + recovery_key_signature + revocation_timestamp + reason_code enum). - §2.1: relax the MSI ABNF/regex to ^did:foxbook:[0-9A-HJKMNP-TV-Z]{26}$ to match core/src/did.ts (the code does not enforce the [0-7] leading-character restriction). Fix the same regex in the W3C runbook. Docs only; no code change. * ops(uptime): split Neon-touching probes to */30 + document autosuspend as the #1 free-tier survival setting Reduce how often the uptime monitor wakes the free-tier Neon compute while keeping fast detection of a hard outage. uptime.yml: add a second cron so the static landing (no DB) keeps its */15 cadence while the two DB-backed probes (api/healthz, transparency/root) ride a slower */30 cron — roughly halving monitor-induced Neon wakeups. Each matrix entry is tagged with the cron it runs on; the probe step is gated via github.event.schedule so the off-schedule endpoints are skipped (and workflow_dispatch still probes everything). The 'Fail run if endpoint is down' behavior is unchanged: a skipped probe yields an empty outcome, so the fail step simply doesn't fire. OPERATIONS.md: in § 'Neon compute-hour exhaustion', make enabling Neon autosuspend / scale-to-zero the prominent #1 action (the Fly /health and uptime changes only reduce wakeups; they do nothing if the compute can never idle), add a one-line 'how to check current usage' pointer to the compute-hours graph, and restate that Launch ($19/mo) removes the cap entirely. Update the 'Uptime monitoring' runbook to describe the split schedule and the detection-latency tradeoff. * chore(ci): harden supply chain — dependabot, SHA-pinned actions, prod prune - add .github/dependabot.yml: weekly grouped minor/patch updates for npm (root), pip (pyproject), gomod (apps/log-daemon-go), and github-actions. - pin every workflow `uses:` to a 40-char commit SHA with a trailing # vX.Y.Z comment. superfly/flyctl-actions/setup-flyctl was on @master in the token-bearing deploy job — now pinned to ed8efb3 (v1.6). - add `permissions: { contents: read }` to deploy-api.yml (least privilege for the tag-gated Fly deploy). - bump vulnerable deps: hono ^4.12.23 (api + transparency), turbo ^2.9.16. - apps/api/Dockerfile: `pnpm prune --prod` after install to drop devDependencies from the prod image. tsx moved to @foxbook/api dependencies so it survives the prune — the container boots with `node --import tsx ./src/main.ts` and needs it at runtime. Guarded by a new prod-runtime-deps test. - ci.yml: non-blocking SCA step (`pnpm audit --audit-level=high || true`). * test(hardening): de-flake firehose sleeps + tier/consistency correctness fixes Mechanical test hardening plus two small correctness fixes (each TDD'd). Test hardening: - firehose.listener.test.ts: replace the three fixed setTimeout sleeps (the reconnect-sentinel and self-test-ping positive waits, and the alive-sentinel negative wait) with the existing poll-until pattern; the negative no-event assertion now uses fake timers so absence is deterministic. These fixed waits caused CI flakes in #62 and #64. - sdk-claim: add a drift guard (merkle-parity.test.ts) that runs every schemas/merkle-test-vectors.json inclusion vector through the inlined verifyInclusion (merkle-internal.ts), pinning it in lockstep with @foxbook/core (the vectors are core-generated ground truth). Correctness fixes: - core verifyConsistency: the m===0 branch accepted ANY oldRoot vacuously; now enforce bytesEqual(oldRoot, EMPTY_TREE_ROOT) so a forged zero-leaf root can't "prove" consistency. Added a negative vector. - claim/by-handle: tier2_pending (an unverified domain claim awaiting DNS/endpoint proof, app-state-only with no Merkle leaf) reported verification_tier 1; map it to 0. The tier>=1 leaf lookup now correctly skips it. Added handler + response-shape tests. Tooling: - add a .husky/pre-push hook running `pnpm test` (turbo-cached unit suite) so a red branch never leaves the machine. * fix(adapter-endpoint-challenge): SSRF guard on outbound fetch + per-IP claim rate limit The Tier-2 endpoint-challenge adapter fetched a claimant-supplied URL (apps/api/src/claim/body-schema.ts accepts endpoint_url) with no scheme or host validation. Reachable via two unauthenticated POSTs (/claim/start-domain then /claim/verify-endpoint), it was an SSRF primitive: point endpoint_url at cloud metadata (169.254.169.254), loopback, or RFC-1918/ULA internal services and read the JSON we round-trip. SSRF guard (two layers, sharing one isBlockedIp policy): 1. Pre-flight (guard.ts: assertOutboundAllowed) before any socket — https-only, then resolve the hostname and reject if ANY candidate IP is private/loopback/link-local/ULA/metadata (0.0.0.0, 127/8, 10/8, 172.16/12, 192.168/16, 169.254/16, ::1, ::, fc00::/7, fe80::/10, incl. IPv4-mapped v6). Bare IP literals are classified directly with no DNS. Fail-closed on resolver error. 2. Connect-time pin (node-transport.ts) — the default transport drives the request through node:https with a guarded dns.lookup that re-applies isBlockedIp to the exact address the socket is about to dial, closing the DNS-rebinding window between pre-flight and connect. Handles the all:true array shape so a private record in a multi-A response can't slip through. Redirects: redirect:"manual" + reject every 3xx (a Location can re-open the hole the pre-flight closed; a real challenge endpoint returns the JWS directly). The adapter's existing interface is unchanged; new optional EndpointVerifyOptions.resolveHostname lets tests drive the pre-flight deterministically, and the existing fetch seam is preserved. Rate limit (apps/api/src/claim/rate-limit.ts): per-IP in-memory token bucket on the mutating claim POST routes (keyed off Fly-Client-IP / X-Forwarded-For), 429 + Retry-After when empty. Blunts the amplification/abuse the outbound fetch enables. Per-instance by design (documented) — a coarse safety valve, not a distributed quota. The read-only GET /claim/by-handle is intentionally excluded. Tests: private/loopback/metadata + non-https targets rejected before any fetch; redirect to a private host blocked; connect-time guard blocks a loopback resolution (rebind defence); normal public https still round-trips (resolver/fetch mocked). Rate-limit: bucket math, per-IP isolation, refill, and the route 429 boundary. * docs(rationale): drop /api/v1/discover from Cache-Control list (sets none) * docs(ops): Neon autosuspend step references the 'main' branch (matches restore runbook) --------- Co-authored-by: Ben <ben@inkog.io>

cloakmaster merged commit 7572533 into main May 4, 2026
3 checks passed

cloakmaster deleted the day-10/stable-mode-phase-4-ops branch May 4, 2026 18:30

This was referenced May 4, 2026

ci(uptime): defer foxbook.dev landing probe until Phase 5 ships #66

Merged

fix(test): firehose listener wedge test — poll until condition #67

Merged

cloakmaster mentioned this pull request Jun 2, 2026

test(hardening): de-flake firehose sleeps + tier/consistency correctness fixes #92

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(ops): stable-mode Phase 4 — operations runbook + uptime monitoring - #64

docs(ops): stable-mode Phase 4 — operations runbook + uptime monitoring#64
cloakmaster merged 1 commit into
mainfrom
day-10/stable-mode-phase-4-ops

cloakmaster commented May 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cloakmaster commented May 4, 2026

What's in

OPERATIONS.md sections

Uptime workflow

env-vars.md cleanup

Verification

What's NOT in this PR

Refs

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants