feat: add ops schema reset functionality and update documentation

- Introduced a new script `run-ops-db-schema-reset-k8s.sh` to facilitate a full database schema reset in Kubernetes, addressing checksum mismatches.
- Added a new suspended CronJob `metaboost-db-drop-everything` to drop the public schema on both app and management databases.
- Updated documentation in `DB-MIGRATIONS.md` and `REMOTE-K8S-GITOPS.md` to include detailed instructions for the ops schema reset process.
- Enhanced the `rebootstrap-full-bootstrap.sh` script to create the `uuid-ossp` extension during rebootstrap.
- Updated `kustomization.yaml` to include the new CronJob resource.

Author: mitchdowney

.cursor/skills/linear-db-migrations/SKILL.md

@@ -32,6 +32,7 @@ Treat each chain as **ordered fresh applies**: migration `NNNN` may assume schem
 ## Runner and validation
 
 - Apply migrations: `bash scripts/database/run-linear-migrations.sh --database app|management` (always pass `--database`; there is no default).
+- **K8s schema reset (checksum mismatch):** `bash scripts/database/run-ops-db-schema-reset-k8s.sh` or `npm run db:ops:schema-reset:k8s` (requires `K8S_NAMESPACE`; runs drop → rebootstrap → migrate → verify → superuser-create jobs).
 - **Credentials:** **app** migrations use `DB_APP_MIGRATOR_USER`, `DB_APP_MIGRATOR_PASSWORD`, `DB_APP_NAME`, `DB_HOST`, and `DB_PORT`. **Management** migrations use `DB_MANAGEMENT_MIGRATOR_USER`, `DB_MANAGEMENT_MIGRATOR_PASSWORD`, `DB_MANAGEMENT_NAME`, `DB_HOST`, and `DB_PORT`. Optional: `infra/config/local/db.env` when keys are unset before sourcing.
 - K8s wrapper: `bash scripts/database/run-linear-migrations-k8s.sh` (`--database` required); validates the same keys from Secrets.
 - Validate: `bash scripts/database/validate-linear-migrations.sh` (and `--check-db` to compare on-disk checksums to `linear_migration_history` when a DB is available).
@@ -44,7 +45,7 @@ Treat each chain as **ordered fresh applies**: migration `NNNN` may assume schem
 ## Ops bundle (cache busting)
 
 - `infra/k8s/base/ops/kustomization.yaml` must list every `.sql` file under the app and management `source` directories so the ops jobs ConfigMaps stay in sync.
-- The ops migration-runtime ConfigMap also bundles `verify-bootstrap-contract.sh` and `rebootstrap-full-bootstrap.sh` for suspended CronJobs `metaboost-db-verify-bootstrap-contract` and `metaboost-db-rebootstrap-roles`.
+- The ops migration-runtime ConfigMap also bundles `verify-bootstrap-contract.sh` and `rebootstrap-full-bootstrap.sh` for suspended CronJobs `metaboost-db-verify-bootstrap-contract`, `metaboost-db-rebootstrap-roles`, and `metaboost-db-drop-everything`.
 - Kustomize may load paths outside the ops directory; when building, use e.g. `kubectl kustomize infra/k8s/base/ops --load-restrictor LoadRestrictionsNone`.
 - Local operator check: `make db_verify_bootstrap_contract` (wraps `scripts/database/verify-bootstrap-contract.sh`).
 

docs/development/DB-MIGRATIONS.md

@@ -70,6 +70,7 @@ npm run db:migrate:linear:dry-run:management
 
 Suspended CronJobs:
 
+- `metaboost-db-drop-everything` — drops `public` schema on app and management DBs (destructive)
 - `metaboost-db-migrate-app`
 - `metaboost-db-migrate-management`
 - `metaboost-db-verify-bootstrap-contract`
@@ -79,7 +80,44 @@ Suspended CronJobs:
 
 Trigger one-off jobs during first deploy and any deploy that introduces new migration files. After schema or role recovery, run verify-bootstrap-contract or rebootstrap-roles manually before app rollout.
 
-Example on-demand triggers:
+### Ops-only schema reset (checksum mismatch / edited historical SQL)
+
+When a database already applied older migration file contents and a new release changes those files
+(checksum mismatch in migrate job logs), **do not** edit `linear_migration_history` checksums by hand.
+For disposable environments (e.g. alpha), reset schema via ops jobs only:
+
+1. `metaboost-db-drop-everything`
+2. `metaboost-db-rebootstrap-roles`
+3. `metaboost-db-migrate-app`
+4. `metaboost-db-migrate-management`
+5. `metaboost-db-verify-bootstrap-contract`
+6. `metaboost-management-superuser-create`
+
+Then rollout-restart API workloads. Scale down app tiers first if you want a quiet cutover.
+
+From repo root (waits for each job):
+
+```bash
+export K8S_NAMESPACE=metaboost-alpha
+bash scripts/database/run-ops-db-schema-reset-k8s.sh
+```
+
+Or trigger each CronJob manually (Argo CD UI or kubectl):
+
+```bash
+kubectl -n <namespace> create job --from=cronjob/metaboost-db-drop-everything metaboost-db-drop-everything-manual-$(date +%s)
+kubectl -n <namespace> create job --from=cronjob/metaboost-db-rebootstrap-roles metaboost-db-rebootstrap-roles-manual-$(date +%s)
+kubectl -n <namespace> create job --from=cronjob/metaboost-db-migrate-app metaboost-db-migrate-app-manual-$(date +%s)
+kubectl -n <namespace> create job --from=cronjob/metaboost-db-migrate-management metaboost-db-migrate-management-manual-$(date +%s)
+kubectl -n <namespace> create job --from=cronjob/metaboost-db-verify-bootstrap-contract metaboost-db-verify-bootstrap-contract-manual-$(date +%s)
+K8S_NAMESPACE=<namespace> npm run management:superuser:create:k8s
+```
+
+**PVC wipe** (empty volume + docker-entrypoint init baselines) is an alternative; you do **not** need
+drop/rebootstrap when first-start init runs successfully. See
+[REMOTE-K8S-POSTGRES-REINIT.md](/docs/development/k8s/REMOTE-K8S-POSTGRES-REINIT.md).
+
+Example on-demand triggers (individual jobs):
 
 ```bash
 K8S_NAMESPACE=<namespace> npm run management:superuser:create:k8s

docs/development/k8s/REMOTE-K8S-GITOPS.md

@@ -218,6 +218,52 @@ kubectl -n "$NAMESPACE" get svc,ingress
 kubectl -n argocd get applications
 ```
 
+### 9. Post-sync DB bootstrap verification (required)
+
+After **db** sync on a **new empty volume**, or after the **ops schema-reset** job sequence (see §10),
+verify extension, grants, and `linear_migration_history` with the suspended ops CronJob:
+
+```fish
+kubectl -n $NAMESPACE create job --from=cronjob/metaboost-db-verify-bootstrap-contract metaboost-db-verify-bootstrap-contract-manual
+kubectl -n $NAMESPACE logs -f job/metaboost-db-verify-bootstrap-contract-manual
+```
+
+Expected result includes: `Bootstrap contract verification passed for app and management databases.`
+
+If it fails, do **not** proceed to app rollouts until the DB bootstrap contract is green.
+
+### 10. Ops-only schema reset (checksum mismatch / edited historical SQL)
+
+When migrate jobs fail with **checksum mismatch for already-applied migration** (common after a
+release edits `0001_*` or other files that an environment already applied), reset via **ops CronJobs**
+only — no PVC delete required. **Destructive:** drops `public` schema on app and management DBs.
+
+**Prerequisite:** `metaboost-<env>-ops` synced so `metaboost-db-drop-everything` exists (Metaboost
+`base/ops` at your pinned `?ref=`).
+
+Run jobs **in order** (Argo CD UI: create Job from each suspended CronJob, or from Metaboost repo root):
+
+1. `metaboost-db-drop-everything`
+2. `metaboost-db-rebootstrap-roles`
+3. `metaboost-db-migrate-app`
+4. `metaboost-db-migrate-management`
+5. `metaboost-db-verify-bootstrap-contract` (§9)
+6. `metaboost-management-superuser-create`
+
+One-shot from Metaboost monorepo (waits for each step):
+
+```bash
+export K8S_NAMESPACE=metaboost-alpha
+npm run db:ops:schema-reset:k8s
+```
+
+Then rollout-restart API deployments and re-sync web tiers. Full detail:
+[DB-MIGRATIONS.md](/docs/development/DB-MIGRATIONS.md).
+
+**PVC wipe alternative:** delete Postgres PVC and rely on first-start init baselines — see
+[REMOTE-K8S-POSTGRES-REINIT.md](/docs/development/k8s/REMOTE-K8S-POSTGRES-REINIT.md) §3. You do **not**
+need drop/rebootstrap when that init path completes successfully.
+
 ## GitOps overlay contract
 
 - Every deployed overlay uses immutable refs for remote Metaboost bases.

docs/development/k8s/REMOTE-K8S-POSTGRES-REINIT.md

@@ -12,7 +12,7 @@ Replace **`metaboost-alpha`** with your namespace if different.
 
 **Migrating from the old Deployment layout:** If the namespace still has PVC **`metaboost-postgres-data`** (Deployment-era), replacing it with this StatefulSet creates a **new** PVC **`db-data-metaboost-db-0`**. Back up data first, or delete the old Deployment/PVC when a wipe is acceptable so the new pod initializes cleanly.
 
-**Existing data / drift / password rotation without wipe:** Use **§4** (manual SQL from your machine) or delete the Postgres PVC and bring the pod back so **`PGDATA`** is empty and first-start init runs again (**§3**).
+**Existing data / drift / password rotation without wipe:** Use **§4** (manual SQL from your machine), the **ops schema-reset job sequence** in [DB-MIGRATIONS.md](/docs/development/DB-MIGRATIONS.md) (drop → rebootstrap → migrate → verify → superuser), or delete the Postgres PVC and bring the pod back so **`PGDATA`** is empty and first-start init runs again (**§3**).
 
 ---
 

infra/k8s/base/ops/db-drop-everything.cronjob.yaml

@@ -0,0 +1,50 @@
+# Pattern: Suspended CronJob (Manual Trigger Only)
+# DANGER: Drops public schema on both app and management logical databases. Only runs when explicitly triggered.
+# Recovery sequence: drop-everything → rebootstrap-roles → migrate-app → migrate-management →
+# verify-bootstrap-contract → management-superuser-create. See docs/development/DB-MIGRATIONS.md.
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: metaboost-db-drop-everything
+spec:
+  schedule: "0 0 1 1 *"
+  suspend: true
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 3
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      backoffLimit: 1
+      ttlSecondsAfterFinished: 300
+      template:
+        metadata:
+          annotations:
+            metaboost.example.org/ops-job: db-drop-everything
+        spec:
+          restartPolicy: Never
+          enableServiceLinks: false
+          containers:
+            - name: drop-everything
+              image: postgres:18.3
+              imagePullPolicy: Always
+              command:
+                - bash
+                - -c
+                - |
+                  set -euo pipefail
+                  export PGHOST="${DB_HOST:-metaboost-db}"
+                  export PGPORT="${DB_PORT:-5432}"
+                  export PGPASSWORD="${DB_APP_OWNER_PASSWORD}"
+                  psql -v ON_ERROR_STOP=1 -U "${DB_APP_OWNER_USER}" -d "${DB_APP_NAME}" \
+                    -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
+                  export PGPASSWORD="${DB_MANAGEMENT_OWNER_PASSWORD}"
+                  psql -v ON_ERROR_STOP=1 -U "${DB_MANAGEMENT_OWNER_USER}" -d "${DB_MANAGEMENT_NAME}" \
+                    -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'
+              envFrom:
+                - secretRef:
+                    name: metaboost-db-secrets
+              env:
+                - name: DB_HOST
+                  value: metaboost-db
+                - name: DB_PORT
+                  value: "5432"

infra/k8s/base/ops/db-rebootstrap-roles.cronjob.yaml

@@ -1,7 +1,7 @@
 # Pattern: Suspended CronJob (Manual Trigger Only)
 # Re-runs full StatefulSet bootstrap logic from `0001_create_app_db_users.sh` +
 # `0002_setup_management_database.sh` over TCP (script: rebootstrap-full-bootstrap.sh).
-# Run after DROP SCHEMA public CASCADE or role/password drift so migrator roles and grants match init.
+# Run after metaboost-db-drop-everything or DROP SCHEMA public CASCADE so migrator roles and grants match init.
 # Does not apply 0003 linear baselines — migrate jobs remain the forward path.
 apiVersion: batch/v1
 kind: CronJob

infra/k8s/base/ops/kustomization.yaml

@@ -2,6 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 
 resources:
+  - db-drop-everything.cronjob.yaml
   - db-migrate-app.cronjob.yaml
   - db-migrate-management.cronjob.yaml
   - db-verify-bootstrap-contract.cronjob.yaml

infra/k8s/base/ops/source/database/runner/rebootstrap-full-bootstrap.sh

@@ -171,4 +171,12 @@ ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO ${DB_MANAGEM
 ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON SEQUENCES TO ${DB_MANAGEMENT_READ_USER};
 SQL
 
+echo "rebootstrap-full-bootstrap: Phase D — uuid-ossp extension (same as 0003_apply_linear_baselines.sh)"
+
+export PGPASSWORD="${DB_APP_OWNER_PASSWORD}"
+psql -v ON_ERROR_STOP=1 -U "${DB_APP_OWNER_USER}" -d "${DB_APP_NAME}" -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";'
+
+export PGPASSWORD="${DB_MANAGEMENT_OWNER_PASSWORD}"
+psql -v ON_ERROR_STOP=1 -U "${DB_MANAGEMENT_OWNER_USER}" -d "${DB_MANAGEMENT_NAME}" -c 'CREATE EXTENSION IF NOT EXISTS "uuid-ossp";'
+
 echo "rebootstrap-full-bootstrap: completed successfully."

package.json

@@ -58,6 +58,7 @@
     "db:status:linear:k8s:management": "bash scripts/database/print-linear-migrations-status-k8s.sh --database management",
     "db:validate:linear": "bash scripts/database/validate-linear-migrations.sh",
     "db:validate:linear:db": "bash scripts/database/validate-linear-migrations.sh --check-db",
+    "db:ops:schema-reset:k8s": "bash scripts/database/run-ops-db-schema-reset-k8s.sh",
     "billing:process-due-renewals": "node scripts/api/process-due-renewals.mjs",
     "management:superuser:create": "node scripts/management-api/create-super-admin.mjs",
     "management:superuser:update": "node scripts/management-api/update-super-admin.mjs",

scripts/database/run-ops-db-schema-reset-k8s.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+# Trigger the full Metaboost ops DB schema reset sequence in Kubernetes (one CronJob per step).
+#
+# Use when linear_migration_history checksums disagree with current migration files (e.g. after
+# editing already-applied SQL on a non-greenfield database). Destructive: drops public schema on
+# app and management databases.
+#
+# Prerequisites:
+# - kubectl context targets the intended cluster/namespace
+# - ops overlay synced (suspended CronJobs present)
+# - metaboost-db Running
+#
+# Usage:
+#   export K8S_NAMESPACE=metaboost-alpha
+#   bash scripts/database/run-ops-db-schema-reset-k8s.sh
+#
+# Each step creates a Job from the matching CronJob and waits for completion (default 15m timeout).
+
+set -euo pipefail
+
+NAMESPACE="${K8S_NAMESPACE:-}"
+WAIT_TIMEOUT="${OPS_DB_RESET_WAIT_TIMEOUT:-900s}"
+
+if [[ -z "$NAMESPACE" ]]; then
+  echo "ERROR: K8S_NAMESPACE must be set (e.g. export K8S_NAMESPACE=metaboost-alpha)." >&2
+  exit 1
+fi
+
+CRONJOBS=(
+  metaboost-db-drop-everything
+  metaboost-db-rebootstrap-roles
+  metaboost-db-migrate-app
+  metaboost-db-migrate-management
+  metaboost-db-verify-bootstrap-contract
+  metaboost-management-superuser-create
+)
+
+run_cronjob_step() {
+  local cronjob="$1"
+  local job_name="${cronjob}-manual-$(date +%s)"
+
+  echo "==> Creating job ${job_name} from cronjob/${cronjob}"
+  kubectl -n "$NAMESPACE" create job --from="cronjob/${cronjob}" "$job_name"
+
+  echo "==> Waiting for job/${job_name} (timeout ${WAIT_TIMEOUT})"
+  if ! kubectl -n "$NAMESPACE" wait --for=condition=complete "job/${job_name}" --timeout="$WAIT_TIMEOUT"; then
+    echo "ERROR: job/${job_name} did not complete successfully." >&2
+    kubectl -n "$NAMESPACE" logs "job/${job_name}" --tail=80 || true
+    exit 1
+  fi
+
+  kubectl -n "$NAMESPACE" logs "job/${job_name}" --tail=40
+  echo ""
+}
+
+echo "Metaboost ops DB schema reset in namespace: ${NAMESPACE}"
+echo "Steps: ${CRONJOBS[*]}"
+echo ""
+
+for cronjob in "${CRONJOBS[@]}"; do
+  run_cronjob_step "$cronjob"
+done
+
+echo "Schema reset sequence completed."
+echo "Next: rollout restart deployment/api deployment/management-api (and web tiers if needed)."