Harden production control-plane deploys

Author: richiemcilroy

.env.production.example

@@ -30,7 +30,7 @@ REND_PLAYER_PLAYBACK_BASE_URL=https://ash-1.play.rend.so
 REND_PLAYER_PLAYBACK_COOKIE_DOMAIN=rend.so
 REND_API_CORS_ALLOWED_ORIGINS=https://rend.so,https://www.rend.so
 REND_API_BIND_ADDR=0.0.0.0:4000
-REND_API_AUTO_MIGRATE=true
+REND_API_AUTO_MIGRATE=false
 REND_API_INLINE_MEDIA_PROCESSING=false
 REND_SITE_INTERNAL_TOKEN=replace-me
 REND_BILLING_MODE=autumn

.github/workflows/release-deploy.yml

@@ -12,26 +12,68 @@ The GitHub Actions workflow is `.github/workflows/release-deploy.yml`.
 3. Write a release manifest with immutable `@sha256:` image refs.
 4. Wait on the GitHub `Production` environment approval if that environment is
    configured with required reviewers.
-5. SSH to the control-plane host and run:
+5. SSH to the control-plane host and run a host-side blue/green transaction:
    - `scripts/preflight-control-plane-host.sh --skip-bind-port-check`
    - `scripts/deploy-control-plane-host.sh --dry-run`
    - `scripts/deploy-control-plane-host.sh`
-   - local `rend-api` `/readyz` and `/healthz`
-   The deploy helper waits for `rend-api` `/readyz` before starting
-   `rend-media-worker`.
+   Before preflight, the deploy wrapper installs the current
+   `control-plane.compose.yml`, patches an existing concrete Caddyfile to use
+   the managed upstream snippet, creates that snippet if it is missing, and
+   leaves any existing upstream target intact. The bootstrap also removes
+   legacy `admin off` Caddy settings and reloads Caddy while the upstream still
+   points at the current slot; if that one-time reload fails, it restarts Caddy
+   before the deploy transaction begins so later promotions can use normal
+   reloads. The managed upstream snippet is written `0644` so the `caddy`
+   service user can import it during reload. The env sync step also enforces
+   `REND_API_AUTO_MIGRATE=false` for production serving containers.
+   `scripts/deploy-release-over-ssh.sh` launches these control-plane commands
+   through `sudo systemd-run --wait --collect --pipe`, so the transaction runs
+   as a host-side transient unit. If the GitHub runner or SSH session dies after
+   the unit starts, the host can still finish promotion rollback/cleanup. The
+   deploy helper takes a host lock, runs one-shot `rend-api migrate`, starts the
+   inactive API slot, probes private `/readyz` and `/healthz`, switches the
+   managed Caddy upstream snippet, reloads Caddy, and rolls back the snippet if
+   post-promotion checks fail. The previous API slot remains running.
 6. SSH to each configured edge host, serially, and run:
+   - sync deploy-managed edge env keys from `REND_READINESS_EDGES`:
+     `REND_EDGE_ID`, `REND_EDGE_REGION`, `REND_EDGE_BASE_URL`,
+     `REND_EXPECTED_EDGES`, plus `REND_EDGE_CORS_ALLOWED_ORIGINS`
    - `scripts/preflight-edge-host.sh --skip-bind-port-check`
    - `scripts/deploy-edge-host.sh --dry-run`
    - `scripts/deploy-edge-host.sh`
    - local `rend-edge` `/readyz` and `/healthz`
-7. Check public API `/readyz`.
-8. Run the synthetic playback readiness gate when the required readiness secrets
+7. Monitor public API `/readyz` during the control-plane deploy, store the
+   redacted JSONL monitor artifact, and check public API `/readyz`. Each monitor
+   sample records the attempt count and only fails the workflow after three
+   short failed attempts, so a single transport reset during Caddy's graceful
+   reload is visible but not treated as a sustained outage.
+8. When `run_first_host_verifier` is enabled, verify the edge registry from the
+   control-plane host with `scripts/verify-edge-registry-over-ssh.sh`, then run
+   `scripts/verify-first-host-deploy.sh` against production API and edge targets.
+   The workflow reuses the edge SSH tunnels for private `/readyz`,
+   `/internal/warm`, and `/metrics` checks, then verifies public deny rules,
+   warmed signed playback, telemetry analytics, and ClickHouse reachability.
+9. Run the synthetic playback readiness gate when the required readiness secrets
    and edge targets are configured.
-
-Preflight skips bind-port checks during automated updates because the existing
-service should already own the port. It still validates env files, registry
-pullability, image platform, host dependencies, managed dependency reachability,
-edge control-plane registration/heartbeat, and telemetry ingest.
+10. For an explicit production rollback drill, set
+    `verify_control_plane_rollback=true` on a manual workflow dispatch. The
+    workflow switches Caddy back to the previous control-plane slot without a
+    pull/build, checks public `/readyz`, then re-promotes the current digest and
+    checks public `/readyz` again.
+
+Control-plane preflight treats bound blue/green ports as expected because the
+old slot must keep serving. It still validates env files, Caddy upstream wiring,
+registry pullability, image platform, host dependencies, and managed dependency
+reachability. Edge preflight still validates edge control-plane
+registration/heartbeat and telemetry ingest.
+
+## Host Requirements
+
+Control-plane hosts must have `systemd-run` available to run deploy and rollback
+transactions as transient host-side units. If `systemd-run` is missing, the
+automated control-plane deploy fails before the transaction starts or traffic is
+changed. Edge deploys still run as direct SSH commands because the edge path is
+not yet transactional.
 
 ## Required GitHub Environment
 
@@ -99,17 +141,51 @@ The GitHub workflow opens SSH tunnels to the configured `rend-edge-ash-1` and
 `rend-edge-ams-1` hosts and rewrites the private bases to runner-local tunnel
 URLs for the readiness gate. This keeps `/internal/*` and `/metrics` off the
 public edge hostnames while still allowing GitHub-hosted runners to run the
-synthetic playback checks.
+synthetic playback checks. During edge deploy, the workflow also syncs each
+host's edge identity from these entries so registry checks and edge heartbeat
+IDs use the same source of truth. The workflow verifies registry rows over SSH
+from the control-plane host, using that host's `/etc/rend/rend-api.env`
+`DATABASE_URL`; GitHub does not need a separate production Postgres URL for this
+automated check.
+
+## Optional Secrets
+
+- `DATABASE_URL`: only needed when running
+  `scripts/verify-first-host-deploy.sh` registry checks directly from a laptop,
+  bastion, or custom workflow instead of the host-side SSH registry verifier.
+  Prefer a secret over a variable.
+
+## Optional Variables
+
+- `REND_VERIFY_ASSET_ID`: existing `hls_ready` synthetic/non-customer asset id
+  for the deeper warmed-playback path in `scripts/verify-first-host-deploy.sh`.
+  A manual workflow dispatch can override it with `verify_asset_id`. If omitted,
+  the workflow runs the verifier with `--skip-playback` and relies on the
+  synthetic playback readiness gate for upload/playback/telemetry proof.
 
 ## Rollback
 
 Use the previous `rend-release-manifest` artifact from a successful workflow
-run, then run the same host deploy helper manually in this order:
+run for edge rollback. For the control plane, switch back to the previous API
+slot without pulling/building:
 
 ```sh
 scripts/deploy-release-over-ssh.sh --role edge --host <edge-host> --user <user> --manifest <previous-manifest>
-scripts/deploy-release-over-ssh.sh --role control-plane --host <control-host> --user <user> --manifest <previous-manifest>
+scripts/deploy-release-over-ssh.sh --role control-plane --host <control-host> --user <user> --rollback
 ```
 
-Roll back edges first, then the media worker/API control plane. Treat database
-migrations as forward-only unless a tested rollback migration exists.
+Roll back edges first, then the media worker/API control plane. The
+control-plane rollback assumes the previous API slot is still running. Treat
+database migrations as forward-only unless a tested rollback migration exists.
+
+## Remaining Risk
+
+This automation makes the single control-plane host action-safe at the Docker
+service/Caddy-upstream layer, but it does not remove the single-host SPOF:
+kernel, VM, disk, network, Docker daemon, and Caddy process failures can still
+take the host down. Edge deploys are still in-place per edge host; deploy one
+edge at a time and keep the other edge serving, then verify with the playback
+readiness gate. A full edge blue/green transaction should add per-edge
+blue/green ports, a managed Caddy upstream snippet, candidate private
+`/readyz`/`/healthz`, and automatic snippet rollback matching the control-plane
+pattern.

docs/deployment-automation.md

@@ -321,10 +321,18 @@ liveness because the worker has no HTTP listener.
 
 ## Bootstrap And Migrations
 
-Postgres migrations are applied by `rend-api` through SQLx when
-`REND_API_AUTO_MIGRATE=true`. In local Compose, the worker waits for the API and
-sets `REND_API_AUTO_MIGRATE=false` to avoid duplicate startup migration work.
-For production, deploy or run the API migration step before starting workers.
+Postgres migrations are applied through the explicit one-shot
+`rend-api migrate` command. Local Compose still sets
+`REND_API_AUTO_MIGRATE=true` on the local API for developer convenience and sets
+the worker to `false` to avoid duplicate startup migration work. Production
+serving API and worker containers must use `REND_API_AUTO_MIGRATE=false`; the
+control-plane deploy helper runs the candidate image's `rend-api-migrate`
+service before any Caddy traffic promotion.
+
+Treat production Postgres migrations as expand/contract and rollback-hostile
+unless a tested rollback migration exists. A failed pre-promotion candidate
+keeps the old API slot serving, but an already-applied schema migration is not
+automatically reverted.
 
 ClickHouse schema is applied by the local `clickhouse-init` one-shot service on
 every Compose startup. The schema uses `CREATE DATABASE IF NOT EXISTS` and
@@ -370,18 +378,20 @@ scripts/preflight-edge-host.sh \
   --manifest .rend/releases/production-001.json
 ```
 
-The control-plane preflight checks Docker/Compose, compose/env files, manifest
-digest refs, manifest platform metadata, manifest image pull readiness, pulled
-image OS/architecture, managed dependency connectivity where local tools allow
-it, and host bind ports. The edge preflight checks Docker/Compose, edge env,
+The control-plane preflight checks Docker/Compose, compose/env files, Caddy
+upstream template wiring, manifest digest refs, manifest platform metadata,
+manifest image pull readiness, pulled image OS/architecture, and managed
+dependency connectivity where local tools allow it. The active blue/green API
+slot is expected to keep one private port bound, so control-plane preflight does
+not require API ports to be free. The edge preflight checks Docker/Compose, edge env,
 manifest digest ref, manifest platform metadata, manifest image pull readiness,
 pulled image OS/architecture,
 private-by-default direct port publishing, uid/gid `10001` cache and spool
 writeability, object-store health, control-plane register/heartbeat
 reachability, telemetry ingest reachability, and host bind ports.
 
-Use deploy helpers in dry-run mode first to print the exact Compose commands
-with manifest image refs:
+Use deploy helpers in dry-run mode first to print the exact Compose/Caddy
+transaction with manifest image refs:
 
 ```sh
 scripts/deploy-control-plane-host.sh \
@@ -393,6 +403,44 @@ scripts/deploy-edge-host.sh \
   --dry-run
 ```
 
+The control-plane helper is transactional on the host. It takes
+`/var/lock/rend-control-plane-deploy.lock`, records active/previous slot state
+under `/var/lib/rend/control-plane`, runs the one-shot migration, recreates only
+the inactive API slot (`rend-api-blue` or `rend-api-green`), probes candidate
+`/readyz` and `/healthz` directly, then atomically replaces
+`/etc/caddy/rend-control-plane-upstream.caddy` and reloads Caddy. If promotion
+or post-promotion checks fail, it restores the previous upstream. The previous
+API slot remains running after a successful promotion for immediate rollback.
+When invoked by `scripts/deploy-release-over-ssh.sh`, the control-plane
+transaction runs under `sudo systemd-run --wait --collect --pipe` so rollback can
+continue on the host if the GitHub runner or SSH session dies after the unit is
+started.
+
+The SSH wrapper also bootstraps production host files before preflight: it
+installs the current control-plane Compose template, patches an existing
+concrete Caddyfile to use the managed upstream snippet, creates
+`/etc/caddy/rend-control-plane-upstream.caddy` only when missing, removes
+legacy `admin off` Caddy settings, and preserves an existing upstream target.
+The bootstrap reloads Caddy while the upstream still points at the current slot;
+if an older running config cannot reload because admin was disabled, it performs
+one restart before the transaction starts so later blue/green promotions can use
+normal Caddy reloads. The managed upstream snippet must stay `0644`; preflight
+fails if the file is root-only because the `caddy` service user imports it on
+reload.
+
+The edge helper remains an in-place per-host deploy. For production, deploy
+edges serially and keep at least one edge serving while the other updates, then
+run the multi-edge verifier/readiness gate. A future edge hardening pass should
+mirror the control-plane slot model: `rend-edge-blue`/`rend-edge-green`, private
+candidate probes, a managed Caddy upstream snippet, and automatic rollback on
+post-promotion failures.
+
+The production workflow derives each edge host's `REND_EDGE_ID`,
+`REND_EDGE_REGION`, `REND_EDGE_BASE_URL`, and shared `REND_EXPECTED_EDGES` from
+`REND_READINESS_EDGES` before restarting that edge. Keep those entries aligned
+with the intended registry IDs; the verifier treats the registry as authoritative
+after deploy.
+
 After deploy, verify the first-host path:
 
 ```sh
@@ -528,24 +576,46 @@ usage or watch accounting.
 5. Run `scripts/validate-production-env.sh` and the relevant preflight script on
    each host.
 6. Run the deploy helper with `--dry-run`, then run it without `--dry-run`.
-7. Deploy `rend-api` with `REND_API_AUTO_MIGRATE=true` for the migration step.
-8. Start `rend-api` serving traffic after `/readyz` passes.
+7. On the control-plane host, let `scripts/deploy-control-plane-host.sh` run the
+   candidate image's one-shot `rend-api migrate` service.
+8. Let the control-plane helper start the inactive API slot, verify private
+   `/readyz` and `/healthz`, then promote Caddy to the candidate slot.
 9. Start `rend-edge` nodes with unique `REND_EDGE_ID`, `REND_EDGE_REGION`,
    API-reachable `REND_EDGE_BASE_URL`, cache volume, and telemetry spool volume.
-10. Start `rend-media-worker` with `REND_API_AUTO_MIGRATE=false`.
+10. Start or update `rend-media-worker` with `REND_API_AUTO_MIGRATE=false`.
 11. Run `scripts/verify-first-host-deploy.sh` with a provided `hls_ready` asset
     to confirm edge registration, signed playback, and telemetry analytics.
 12. Run `bun run playback:readiness -- --target configured --skip-local-stack`
     or pass `--run-readiness-gate` to the verifier before promoting traffic.
 
+The production GitHub workflow runs the first-host verifier when
+`run_first_host_verifier=true`. It first verifies edge registry rows from the
+control-plane host with `scripts/verify-edge-registry-over-ssh.sh`, using the
+host's deployed `/etc/rend/rend-api.env` instead of a separate GitHub
+`DATABASE_URL`. It then rewrites private edge targets through SSH tunnels before
+running API/edge health, ClickHouse, and public deny checks with
+`scripts/verify-first-host-deploy.sh --skip-registration`.
+When `REND_VERIFY_ASSET_ID` or `verify_asset_id` points at an existing
+synthetic/non-customer `hls_ready` asset, the verifier also runs warmed
+playback and analytics checks. If no asset id is configured, the workflow runs
+the verifier with `--skip-playback` and relies on the synthetic playback
+readiness gate for upload, playback, and telemetry proof.
+
 ## Rollback Basics
 
 Roll back services in dependency order from the edge inward:
 
 1. Roll back `rend-edge` first if playback cache behavior regresses.
 2. Roll back `rend-media-worker` if artifact generation or warming regresses.
-3. Roll back `rend-api` last. Treat Postgres migrations as forward-only unless a
-   tested rollback migration exists.
+3. Roll back `rend-api` last. For the control plane, prefer
+   `scripts/deploy-control-plane-host.sh --rollback` to switch Caddy back to the
+   previous slot without pulling or rebuilding. Treat Postgres migrations as
+   forward-only unless a tested rollback migration exists.
+
+For a production rollback drill in GitHub Actions, run the workflow manually
+with `verify_control_plane_rollback=true`. The workflow switches Caddy back to
+the previous control-plane slot, verifies public `/readyz`, then deploys the
+current digest manifest again to re-promote the candidate slot.
 
 Edge cache can be purged or discarded during rollback. Telemetry spool files can
 be retained for replay or deleted if the ingest contract changed incompatibly.
@@ -562,3 +632,14 @@ US East and London edge nodes differ only by environment and attached volumes:
 - local telemetry spool volume
 
 The same `rend-edge` image and command run in both regions.
+
+## Residual SPOFs
+
+The blue/green control-plane transaction prevents a failed deploy, failed
+candidate, failed Caddy reload, or failed post-promotion check from taking down
+the currently serving API process. It does not remove single-host or
+single-daemon failure modes. A kernel panic, VM outage, host network loss, disk
+failure, Docker daemon failure, or Caddy process failure on the control-plane
+host can still cause downtime. The current edge model is resilient only at the
+multi-edge operational level; each individual edge host still updates
+`rend-edge` in place.