feat(backup): scheduled database backup verification with restore testing

[aaronmanuel309-bot]

Motivation

Closes #101 — Scheduled Database Backup Verification with Restore Testing.

The backend depends on PostgreSQL for all persistent state, but a backup is
only useful if it can actually be restored. This change guarantees that by
running a three-stage pipeline on a schedule, so a "backup succeeded" claim is
backed by evidence that the data is restorable.

Description

Pipeline

1. Backup (takeBackup) — consistent pg_dump snapshot, gzip-compressed
   (zlib), encrypted at rest with AES-256 (OpenSSL PBKDF2), optionally copied
   off-site to S3, with a SHA-256 manifest written alongside the artifact.
2. Verification (verifyBackup) — decrypt/decompress, re-compute the
   checksum against the manifest, and validate the PostgreSQL dump header and
   CREATE TABLE statements.
3. Restore testing (restoreTestBackup) — restore the backup into an
   isolated scratch database (lumina_restore_test_*), compare the number of
   public tables and the row counts of key tables (BACKUP_VERIFY_TABLES)
   against the live database, then drop the scratch database (with lingering
   connections terminated first) in a finally block.

Scheduling

• BACKUP_CRON (default 0 2 * * *) — daily backup + integrity verification.
• BACKUP_RESTORE_TEST_CRON (default 0 4 * * 0) — weekly restore test of the
  latest verified artifact.
• Master switches: BACKUP_ENABLED, BACKUP_RESTORE_TEST_ENABLED.

Operations

• Admin API: GET /api/admin/backups/status, GET /api/admin/backups/history,
  POST /api/admin/backups/run, POST /api/admin/backups/restore-test.
• CLI: npm run backup:verify and npm run backup:restore-test
  (backend/scripts/run-backup-verification.js), exit code 0/1 for cron/CI.
• History: local JSONL audit trail (verification-history.jsonl) with
  pruning, plus in-report metrics.

Monitoring

• Prometheus metrics on the shared /metrics registry: attempts, failures,
  durations, artifact size, verification status, last-success timestamps, and
  per-table restore row deltas.
• Alerting rules (monitoring/prometheus/database-backup-verification-rules.yaml):
  DatabaseBackupFailed, DatabaseBackupStale, DatabaseBackupVerificationFailed,
  DatabaseBackupRestoreTestFailed, DatabaseBackupRestoreTestStale,
  DatabaseBackupRestoreRowDeltaMismatch.
• Grafana dashboard (monitoring/grafana/dashboards/database-backup-verification.json).

Documentation

• Architecture: backend/docs/architecture/database-backup-verification.md
• Runbook: backend/docs/runbooks/database-backup-verification.md
• Consolidated into BACKEND.md and mirrored in backend/.env.example.

Configuration

BACKUP_ENABLED=true
BACKUP_DIR=./backups
BACKUP_CRON=0 2 * * *
BACKUP_RESTORE_TEST_CRON=0 4 * * 0
BACKUP_RESTORE_TEST_ENABLED=true
BACKUP_RETENTION_DAYS=30
# BACKUP_ENCRYPTION_KEY=<aes-256 key, >= 16 chars, required in production>
# BACKUP_S3_BUCKET=s3://lumina-backups/prod
# BACKUP_S3_UPLOAD=false
BACKUP_VERIFY_TABLES=vaults,sub_schedules,beneficiaries
# PG_DB / PG_USER / PG_HOST / PG_PORT / PG_PASSWORD (DB_* fallbacks apply)

Testing

• 36 new unit tests across backend/test/backup/ covering configuration
  validation, the backup pipeline (with/without encryption, S3 upload, failure
  paths), integrity verification (pass, checksum tamper, missing manifest),
  restore testing (pass, row-count mismatch, table-count mismatch, scratch DB
  lifecycle), full-cycle reports, history, and retention cleanup.
• Full backend suite: 21 suites / 201 tests passing.

Deployment

Blue-green: deploy to green with backup jobs disabled, confirm metrics appear,
enable restore testing on green, then enable the full schedule and promote.
No schema migrations are involved, so rollback is a plain image revert.

Breaking Changes

None. All changes are additive and off the request critical path.