Skip to content

disaster-recovery.md

Latest commit

 

History

History
242 lines (157 loc) · 6.78 KB

disaster-recovery.md

File metadata and controls

242 lines (157 loc) · 6.78 KB
Raw

Disaster Recovery Plan — StellarTrustEscrow

Objectives

Metric Target
RTO (Recovery Time Objective) < 1 hour for SEV1, < 4 hours for SEV2
RPO (Recovery Point Objective) < 24 hours (daily backups), < 1 hour with WAL archiving

Covered Scenarios

  1. Database failure / data loss
  2. API server outage
  3. Smart contract exploit
  4. Stellar network disruption
  5. Secret/credential compromise

1. Database Failure

Detection: PagerDuty alert, GET /health returns DB error, Grafana DB connection metric drops.

Recovery steps:

# 1. Identify latest valid backup
ls -lt /var/backups/stellar-trust/backup_*.dump | head -5
# or from S3:
aws s3 ls $BACKUP_S3_BUCKET --recursive | sort | tail -5

1.1 WAL archiving and PITR

The system supports WAL archiving for point-in-time recovery (PITR). Enable in postgresql.conf:

wal_level = replica
archive_mode = on
archive_timeout = 60
archive_command = 'aws s3 cp "%p" "${WAL_ARCHIVE_S3_BUCKET}/wal/%f" --sse AES256'

On the backup host, configure:

  • WAL_ARCHIVE_S3_BUCKET (e.g. s3://my-bucket/stellar-trust/wal)
  • WAL_ARCHIVE_DIR (local staging path; typically /var/lib/postgresql/wal_archive)

Restore point-in-time

  1. Fetch a base backup (from backup.sh or pg_basebackup).
  2. Run:
bash scripts/restore_pitr.sh /var/backups/stellar-trust/basebackup_YYYYMMDDTHHMMSSZ.tar.gz "2026-03-28 13:08:00 UTC"
  1. Start Postgres with PGDATA=/var/lib/postgresql/pitr_data pg_ctl -D /var/lib/postgresql/pitr_data start.
  2. Confirm recovery with pg_wal/recovery.done (or pg_wal/recovery.signal).

1.2 Backup policy

  • Daily full backups (pg_dump) are generated by scripts/backup.sh.
  • Daily WAL segments are archived to S3.
  • 7-day retention is enforced locally & by S3 lifecycle policy.

Tip: scripts/backup.sh uploads using SSE AES256 and verifies in-place with pg_restore --list.

2. Restore to a fresh DB

createdb stellartrust_escrow_restored pg_restore
--host=$DB_HOST --port=$DB_PORT
--username=$DB_USER
--dbname=stellar_trust_escrow_restored
/var/backups/stellar-trust/backup.dump

3. Verify checksum before restoring

sha256sum -c /var/backups/stellar-trust/backup_.dump.sha256

4. Update DATABASE_URL in Vault / .env to point to restored DB

5. Restart backend: pm2 restart all OR docker compose restart api


**Runbook:** `docs/incidents/runbooks/database-outage.md`

---

## 2. API Server Outage

**Detection:** Health check fails, uptime monitor alerts, Sentry error spike.

**Recovery steps:**

```bash
# Docker deployment
docker compose down && docker compose up -d

# PM2 deployment
pm2 restart all

# Check logs
docker compose logs --tail=100 api
# or
pm2 logs --lines 100

If the server itself is gone, redeploy from the last known-good image:

# Re-run deploy script
bash scripts/deploy.sh

Runbook: docs/incidents/runbooks/sev1-critical-outage.md

3. Smart Contract Exploit

Detection: Anomalous on-chain transactions, Sentry alerts, user reports of unexpected fund movement.

Immediate actions (first 15 min):

  1. Pause any off-chain automation that submits transactions (stop escrowIndexer, eventIndexer services).
  2. Post SEV1 incident via API or Slack bot.
  3. Do NOT attempt on-chain fixes without contract owner key — coordinate with security lead.
  4. Preserve all logs and transaction hashes before any remediation.

Runbook: docs/incidents/runbooks/smart-contract-exploit.md

4. Stellar Network Disruption

Detection: Soroban RPC calls failing, Horizon returning errors, stellarService.js circuit breaker open.

Recovery steps:

  1. Switch SOROBAN_RPC_URL and STELLAR_HORIZON_URL to a backup provider or community RPC node.
  2. Restart backend to pick up new env values.
  3. Monitor GET /health until Stellar connectivity is confirmed.

Backup RPC endpoints (testnet):

  • https://soroban-testnet.stellar.org (official)
  • https://rpc-futurenet.stellar.org (fallback for testing)

For mainnet, maintain at least one alternative RPC URL in Vault under stellar-trust/app.

5. Secret / Credential Compromise

Detection: Unauthorized API calls, Vault audit log anomaly, breach notification.

Immediate actions:

# Rotate Vault AppRole secret
vault write -f auth/approle/role/stellar-trust/secret-id

# Revoke all active tokens for the role
vault token revoke -mode=path auth/approle/role/stellar-trust

# Rotate DB password
# Update DATABASE_URL in Vault and restart backend

Rotate all affected secrets before bringing services back online. See backend/config/vault-setup.sh for re-provisioning steps.

Backup Schedule

Backups are managed by scripts/backup.sh and scheduled via scripts/backup.cron.

Frequency Retention Storage
Daily (cron) 7 days local /var/backups/stellar-trust
Daily (cron) 30 days remote S3 (BACKUP_S3_BUCKET)

To run a manual backup with restore verification:

bash scripts/backup.sh --restore-test

Backup health is monitored by backend/services/backupMonitor.js — alerts fire to SLACK_BACKUP_WEBHOOK on failure.

DR Test Schedule

Test Frequency Owner
Backup restore drill Monthly On-call engineer
Full failover simulation Quarterly Engineering lead
Secret rotation drill Quarterly Security lead
Runbook walkthrough After each SEV1/SEV2 Incident commander

To run a restore drill:

bash scripts/backup.sh --restore-test

Document results in docs/incidents/ as a post-mortem entry.

Communication Plan

Audience Channel Who
Engineering team #incidents Slack On-call engineer
Stakeholders Email / status page Engineering lead
Users Status page update Engineering lead

Templates: docs/incidents/templates/

Escalation path: On-call → Secondary on-call → Engineering lead → CTO. See docs/incidents/on-call-guide.md for full escalation matrix.

Key Contacts & Access

Before an incident, confirm you have access to:

  • AWS console (S3 backups)
  • Vault (VAULT_ADDR, AppRole credentials)
  • Database host
  • Deployment platform (Docker host / Render / Heroku)
  • PagerDuty + Slack #incidents

Current on-call: GET /api/incidents/oncall