feat(ci): nightly pg_dump backup workflow + restore runbook

[Otfrugger]

closes #165

Summary

• Adds .github/workflows/db-backup.yml: a scheduled (nightly, 03:00 UTC) + workflow_dispatch GitHub Actions workflow that runs pg_dump --format=custom --no-owner --no-privileges against each network's database independently, gzips the dump, and uploads it as a 14-day-retention artifact per network (wraith-db-backup-<network>-<run_id>).
• Adds ops/backup/dump.sh, the dump+gzip script the workflow calls, matching the style of the existing ops/canary/*.sh scripts.
• Adds docs/backup-restore.md: the restore runbook — required secrets, manual-dispatch instructions, and step-by-step pg_restore procedure into a fresh instance (e.g. Neon), including a warning against cross-restoring testnet/mainnet dumps.
• Updates docs/DUAL_NETWORK.md to mark Nightly pg_dump backup workflow + restore runbook #165 done and point at the new runbook + required secrets.

Design notes

• Backup secrets are DATABASE_URL_TESTNET / DATABASE_URL_MAINNET, separate from the runtime DATABASE_URL each deployment uses — mirroring the _TESTNET/_MAINNET suffix convention already used for SOROBAN_RPC_URL_MAINNET etc. in docs/DUAL_NETWORK.md. This lets one workflow back up both network databases regardless of which network any given deployment is running.
• If a network's secret isn't configured yet, that job logs a ::warning:: and skips cleanly rather than failing the whole workflow — so this can land before both databases exist.
• No app code changed; this is CI + docs only.

Verification

• pg_dump/docker aren't available in the environment I developed this in, so I could not run an actual end-to-end dump/restore — acceptance criterion "Restore runbook verified end-to-end once" still needs a maintainer to run it once against a real (or scratch) Postgres instance with real secrets, as noted at the bottom of docs/backup-restore.md.
• Validated: workflow YAML parses (yaml.safe_load), ops/backup/dump.sh passes bash -n syntax check.
• No TypeScript/app code touched, so tsc/build/tests are unaffected.

[drips-wave]

@Otfrugger Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Miracle656]

The workflow engineering is good — per-network jobs, a clean skip with a warning when the secret isn't configured rather than a red run, if-no-files-found: error so a silent empty backup can't masquerade as success, bounded timeout-minutes, and a runbook alongside it. That's the shape #165 asked for.

One blocker, and it's serious enough that I'd rather be explicit than tactful.

This publishes every webhook signing secret

Miracle656/wraith is a public repository. GitHub Actions artifacts on a public repo are downloadable by anyone — no write access needed, no org membership. And the dump is unfiltered and unencrypted:

- name: Upload backup artifact
  with:
    name: wraith-db-backup-testnet-${{ github.run_id }}
    path: backups/*.dump.gz
    retention-days: 14

prisma/schema.prisma:176:

model WebhookSubscription {
  url       String
  secret    String   // "The secret is used to HMAC-sign payloads"

Plaintext. So the artifact contains every subscriber's HMAC signing secret, and the concrete consequence is that anyone can forge webhook deliveries that pass signature verification — for every subscriber, for fourteen days, on a nightly schedule. The signature stops being evidence of anything.

It also ships every subscriber url, which is a customer list.

The mainnet job makes it worse, not better: that's where real subscribers will be.

This is the same issue I raised on lens#122, which is still open for the same reason. Worth coordinating so both land with the same approach.

Three ways to fix it, in the order I'd pick

1. Don't put the dump in an artifact at all. Push to object storage the repo doesn't expose — S3/R2/B2 with credentials in Actions secrets. This is what the job is really for; artifacts are a convenient default that happens to be public here.
2. Encrypt before upload. age or gpg --symmetric with the passphrase in a secret. Simple, keeps the artifact flow, and a leaked artifact is then inert. Make sure the restore runbook covers decryption, or you've built a backup nobody can use under pressure.
3. Exclude the sensitive tables — pg_dump --exclude-table=wraith.\"WebhookSubscription\" --exclude-table=wraith.\"WebhookDelivery\". Cheapest, but it makes the backup incomplete, and a restore that silently loses subscriptions is its own incident. Only reasonable combined with (1) or (2).

I'd take 1 or 2. Whichever you choose, please say in the runbook why — the next person will otherwise reintroduce a plain artifact upload as a simplification.

Smaller notes

• Verify the restore. The runbook describes restoring; nothing proves the dump is restorable. A step that restores into a scratch database and asserts a row count would turn this from a backup into a tested backup. That distinction usually gets discovered at the worst moment.
• Retention vs. schedule. 14 days of nightlies is 14 copies live at once. Fine once encrypted; worth stating deliberately in the runbook either way.
• docs/DUAL_NETWORK.md already warns that Render's free Postgres is deleted after 90 days — worth linking from the runbook, since that's the scenario this exists for.

Everything except the artifact exposure is ready. Fix that and I'll take it.

[Miracle656]

Approving and merging — I've pushed the encryption fix to your branch myself (93a4220), since the wave has closed and I'd rather this land with your points than sit open.

To be clear about what needed changing and what didn't: the workflow engineering here was right. Per-network jobs because each network has its own database; skipping a network with a warning instead of failing the run, so the workflow can exist before both databases are provisioned; --format=custom --no-owner --no-privileges, which is the correct trio for restoring into a differently-owned instance; if-no-files-found: error so a silent dump failure doesn't upload an empty artifact; and a runbook that says never cross-restore between networks — a real footgun given every row carries a network column but the instances are physically separate. The restore procedure was written by someone who imagined actually running it at 3am.

The one thing that had to change was where the artifact goes.

wraith is a public repository, and Actions artifacts on a public repo are downloadable by anyone with the run URL — no login, no permissions. And this dump isn't only indexed chain data. It contains WebhookSubscription, whose secret column (schema.prisma:176) is the plaintext HMAC signing key for each subscriber, sitting next to that subscriber's delivery URL. Published in the clear, that artifact lets a stranger forge webhook deliveries that pass signature verification against every subscriber, and hands them the subscriber list on the way past — 14 days of retention, refreshed nightly.

The fix is AES-256 symmetric encryption via a new BACKUP_PASSPHRASE secret, failing closed in three independent places rather than one:

1. dump.sh refuses to run at all without the passphrase — no silent fallback to a plaintext dump;
2. it deletes the cleartext .gz and then re-checks it is gone before exiting;
3. the upload glob is *.dump.gz.gpg, deliberately not *.dump.gz* — so if encryption ever fails silently, if-no-files-found: error fails the job instead of publishing the cleartext.

Any one of those alone would be a single point of failure on a mistake that cannot be walked back once the artifact exists.

I kept your skip-with-a-warning behaviour and extended it to the passphrase: a missing backup is recoverable here, because Wraith re-derives its data by re-indexing from chain. A published one is not. The runbook gains the decrypt step, a note that the decrypted dump holds live signing secrets, and a warning that losing the passphrase loses every retained backup.

Verified the gpg round trip locally: the right passphrase recovers the dump, a wrong one is refused, and only the .gpg survives. Workflow YAML parses.

Worth saying plainly — this is the same trap lens #122 hit independently, which makes it a gap in how I wrote the issue, not a lapse on your part. The issue said 'upload as an artifact' and never said 'this repo is public.' I'm fixing the issue text so nobody walks into it a third time.

Thanks for a genuinely well-built workflow.