feat: full platform hardening pass — backup encryption, log rotation, Trivy alerts, platform credential rotation

- Backup encryption: optional AES-256-CBC via BACKUP_ENCRYPTION_KEY env var
- Restore/verify: auto-detect and decrypt .dump.enc files
- Trivy scan-images.sh: create GitHub Issues when vulnerabilities found
- rotate-credentials.sh: add --platform flag for master credential rotation
- bootstrap-server.sh: Loki retention 14d→90d, add logrotate config
- server-contract.md: document all new capabilities
- New runbook: rotate-ssh-keys.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: baijum

docs/runbooks/rotate-ssh-keys.md

@@ -0,0 +1,78 @@
+# Runbook: Rotate SSH Keys
+
+Rotate the SSH key pair used for deployments. Recommended schedule: every 90 days.
+
+## Prerequisites
+
+- Local machine with `ssh-keygen` and `ssh` installed
+- Current SSH access to the server as `deploy` user
+- `gh` CLI authenticated with access to all towlion app repos
+
+## Steps
+
+### 1. Generate a new key pair locally
+
+```bash
+ssh-keygen -t ed25519 -C "deploy@towlion-$(date +%Y%m%d)" -f ~/.ssh/towlion-deploy-new
+```
+
+### 2. Add the new public key to the server
+
+```bash
+ssh deploy@<SERVER_HOST> "cat >> ~/.ssh/authorized_keys" < ~/.ssh/towlion-deploy-new.pub
+```
+
+### 3. Test SSH access with the new key
+
+```bash
+ssh -i ~/.ssh/towlion-deploy-new deploy@<SERVER_HOST> "echo 'New key works'"
+```
+
+### 4. Update GitHub Actions secrets on all app repos
+
+```bash
+NEW_KEY=$(cat ~/.ssh/towlion-deploy-new)
+
+for repo in towlion/todo-app towlion/hello-world towlion/starter-app towlion/wit; do
+  gh secret set SERVER_SSH_KEY --repo "$repo" --body "$NEW_KEY"
+  echo "Updated $repo"
+done
+```
+
+### 5. Verify a deployment works
+
+Trigger a deploy on one app (e.g., push a no-op commit) and confirm it succeeds.
+
+### 6. Remove the old public key from the server
+
+```bash
+ssh -i ~/.ssh/towlion-deploy-new deploy@<SERVER_HOST>
+# On the server:
+# Edit ~/.ssh/authorized_keys and remove the old key line
+# The old key has a different comment/date than the new one
+```
+
+### 7. Replace the local key file
+
+```bash
+mv ~/.ssh/towlion-deploy-new ~/.ssh/towlion-deploy
+mv ~/.ssh/towlion-deploy-new.pub ~/.ssh/towlion-deploy.pub
+```
+
+## Rollback
+
+If the new key doesn't work:
+- The old key is still in `authorized_keys` until step 6
+- SSH in with the old key and remove the new public key
+- Re-set `SERVER_SSH_KEY` secrets to the old private key
+
+## Verification
+
+```bash
+# Confirm only one key in authorized_keys
+ssh deploy@<SERVER_HOST> "wc -l ~/.ssh/authorized_keys"
+# Should output: 1
+
+# Confirm deploy works
+gh workflow run deploy.yml --repo towlion/hello-world
+```

docs/server-contract.md

@@ -52,7 +52,7 @@ This document defines the contract between the platform infrastructure (bootstra
   loki/                                 # Loki log storage
   grafana/                              # Grafana state
   prometheus/                           # Prometheus data (created always, used when metrics enabled)
-  backups/postgres/                     # pg_dump backup files (7-day retention)
+  backups/postgres/                     # pg_dump backup files (7-day retention, optional .dump.enc encryption)
 ```
 
 ## Bootstrap to Deploy Lifecycle
@@ -104,15 +104,15 @@ All scripts live in the platform repo under `infrastructure/` and are copied to
 | `bootstrap-server.sh` | Transform fresh Debian into running platform | Manual (`sudo bash`) |
 | `verify-server.sh` | Read-only health check of server state | Manual (`bash`) |
 | `create-app-credentials.sh` | Provision per-app PostgreSQL user + MinIO bucket | Manual (`bash <script> <app-name>`) |
-| `backup-postgres.sh` | Per-database `pg_dump` with 7-day retention | Cron: daily at 02:00 |
+| `backup-postgres.sh` | Per-database `pg_dump` with 7-day retention, optional AES-256 encryption | Cron: daily at 02:00 |
 | `restore-postgres.sh` | Restore a database from backup | Manual (`bash <script>`) |
 | `check-alerts.sh` | Check container health, disk, memory; create GitHub Issues | Cron: every 5 minutes |
 | `update-images.sh` | Pull latest Docker images and recreate containers | Cron: weekly Sunday at 03:00 |
 | `usage-report.sh` | Generate 6-section resource usage report | Manual (`bash`) |
-| `scan-images.sh` | Scan running container images for vulnerabilities (Trivy) | Cron: weekly Sunday at 04:00 |
+| `scan-images.sh` | Scan running container images for vulnerabilities (Trivy), create GitHub Issues | Cron: weekly Sunday at 04:00 |
 | `deploy-blue-green.sh` | Zero-downtime blue-green deploy with automatic rollback | Called by `deploy.yml` workflow |
 | `verify-backup.sh` | Restore backups to temp DB and verify integrity | Cron: weekly Sunday at 05:00 |
-| `rotate-credentials.sh` | Rotate PostgreSQL/MinIO credentials without downtime | Manual (`bash <script> <app-name>`) |
+| `rotate-credentials.sh` | Rotate per-app or platform master credentials without downtime | Manual (`bash <script> <app-name>` or `--platform`) |
 
 ## Server Hardening
 
@@ -140,6 +140,14 @@ The bootstrap script applies several security measures automatically. Self-hoste
 
 **Docker event audit logging** — A systemd service (`docker-audit.service`) runs `docker events` with JSON output to `/var/log/docker-audit.log`. Promtail scrapes this file and forwards events to Loki (label: `job=docker-audit`). All container start, stop, die, and health_status events are captured.
 
+**Backup encryption** — Backups can be encrypted at rest using AES-256-CBC. Set the `BACKUP_ENCRYPTION_KEY` environment variable to the path of a key file. When set, `backup-postgres.sh` pipes `pg_dump` output through `openssl enc` and produces `.dump.enc` files. `restore-postgres.sh` and `verify-backup.sh` automatically detect encrypted backups and decrypt them before restoring. If the key file is not set, backups are stored unencrypted (with a warning).
+
+**Log rotation** — A logrotate config at `/etc/logrotate.d/towlion` rotates `/var/log/towlion-*.log` and `/var/log/docker-audit.log` daily, retaining 90 compressed copies. The `docker-audit.service` is restarted after rotation since it holds the log file open.
+
+**Log retention** — Loki retains logs for 90 days (`retention_period: 2160h`). The compactor runs retention enforcement with a 2-hour delete delay.
+
+**Platform credential rotation** — `rotate-credentials.sh --platform` rotates the PostgreSQL superuser password and/or MinIO root password. After rotation, all app health checks are verified. Use `--yes` to skip the confirmation prompt.
+
 **Image vulnerability scanning** — Trivy is installed via the Aqua Security apt repository. Every deploy runs a non-blocking `trivy image` scan of the newly built app image (HIGH/CRITICAL severity). A weekly cron job (`scan-images.sh`, Sunday 04:00) scans all running container images.
 
 **Mandatory Access Control (AppArmor)** — Debian 12 ships with AppArmor enabled by default. Docker automatically applies the `docker-default` AppArmor profile to all containers, which restricts capabilities like writing to `/proc` and `/sys`, mounting filesystems, and accessing raw sockets. No configuration is needed — this works out of the box.

infrastructure/backup-postgres.sh

@@ -23,13 +23,26 @@ error() {
 # Configuration
 BACKUP_DIR="/data/backups/postgres"
 COMPOSE_FILE="/opt/platform/docker-compose.yml"
+# BACKUP_ENCRYPTION_KEY: path to a key file for AES-256-CBC encryption (optional)
+ENCRYPTION_KEY="${BACKUP_ENCRYPTION_KEY:-}"
 
 # Create backup directory if it doesn't exist
 mkdir -p "$BACKUP_DIR"
 
 info "Starting PostgreSQL backup..."
 info "Backup directory: $BACKUP_DIR"
 
+if [ -n "$ENCRYPTION_KEY" ] && [ -f "$ENCRYPTION_KEY" ]; then
+    info "Encryption enabled (key file: $ENCRYPTION_KEY)"
+    ENCRYPT=true
+elif [ -n "$ENCRYPTION_KEY" ]; then
+    warn "BACKUP_ENCRYPTION_KEY is set but file not found: $ENCRYPTION_KEY"
+    warn "Backups will NOT be encrypted"
+    ENCRYPT=false
+else
+    ENCRYPT=false
+fi
+
 # List all databases except templates and postgres system database
 info "Fetching database list..."
 databases=$(docker compose -f "$COMPOSE_FILE" exec -T postgres psql -U postgres -tc \
@@ -48,12 +61,30 @@ backup_count=0
 
 # Backup each database
 for db in $databases; do
-    filename="${db}_$(date +%Y%m%d_%H%M%S).dump"
+    timestamp=$(date +%Y%m%d_%H%M%S)
+    if [ "$ENCRYPT" = true ]; then
+        filename="${db}_${timestamp}.dump.enc"
+    else
+        filename="${db}_${timestamp}.dump"
+    fi
     filepath="$BACKUP_DIR/$filename"
 
     info "Backing up database: $db"
 
-    if docker compose -f "$COMPOSE_FILE" exec -T postgres pg_dump -U postgres -Fc "$db" > "$filepath"; then
+    if [ "$ENCRYPT" = true ]; then
+        dump_ok=false
+        if docker compose -f "$COMPOSE_FILE" exec -T postgres pg_dump -U postgres -Fc "$db" \
+            | openssl enc -aes-256-cbc -pbkdf2 -pass "file:${ENCRYPTION_KEY}" -out "$filepath"; then
+            dump_ok=true
+        fi
+    else
+        dump_ok=false
+        if docker compose -f "$COMPOSE_FILE" exec -T postgres pg_dump -U postgres -Fc "$db" > "$filepath"; then
+            dump_ok=true
+        fi
+    fi
+
+    if [ "$dump_ok" = true ]; then
         file_size=$(stat -f%z "$filepath" 2>/dev/null || stat -c%s "$filepath" 2>/dev/null || echo "0")
         human_size=$(numfmt --to=iec-i --suffix=B "$file_size" 2>/dev/null || echo "${file_size}B")
         info "  ✓ Backed up $db to $filename ($human_size)"
@@ -68,7 +99,7 @@ done
 
 # Retention: delete backups older than 7 days
 info "Cleaning up old backups (older than 7 days)..."
-deleted_count=$(find "$BACKUP_DIR" -name "*.dump" -mtime +7 -delete -print | wc -l | tr -d ' ')
+deleted_count=$(find "$BACKUP_DIR" \( -name "*.dump" -o -name "*.dump.enc" \) -mtime +7 -delete -print | wc -l | tr -d ' ')
 if [ "$deleted_count" -gt 0 ]; then
     info "  Removed $deleted_count old backup(s)"
 else

infrastructure/bootstrap-server.sh

@@ -314,7 +314,7 @@ common:
       store: inmemory
 
 limits_config:
-  retention_period: 336h
+  retention_period: 2160h
 
 schema_config:
   configs:
@@ -329,6 +329,8 @@ schema_config:
 compactor:
   working_directory: /loki/compactor
   retention_enabled: true
+  retention_delete_delay: 2h
+  retention_delete_worker_count: 150
   delete_request_cancel_period: 10m
   delete_request_store: filesystem
 EOF
@@ -1329,6 +1331,27 @@ EOF
   info "Promtail config updated with docker-audit scrape target"
 fi
 
+# --- Log Rotation ---
+
+LOGROTATE_CONF="/etc/logrotate.d/towlion"
+LOGROTATE_CONTENT='/var/log/towlion-*.log /var/log/docker-audit.log {
+    daily
+    rotate 90
+    compress
+    missingok
+    notifempty
+    postrotate
+        systemctl restart docker-audit.service 2>/dev/null || true
+    endscript
+}'
+
+if [[ -f "$LOGROTATE_CONF" ]] && echo "$LOGROTATE_CONTENT" | diff -q - "$LOGROTATE_CONF" >/dev/null 2>&1; then
+  info "Logrotate config already up to date"
+else
+  echo "$LOGROTATE_CONTENT" > "$LOGROTATE_CONF"
+  info "Logrotate config created at $LOGROTATE_CONF"
+fi
+
 # --- Start Services ---
 
 echo

infrastructure/restore-postgres.sh

@@ -64,11 +64,34 @@ fi
 
 # Determine target database
 if [ -z "$TARGET_DB" ]; then
-    # Extract database name from filename pattern: <dbname>_YYYYMMDD_HHMMSS.dump
-    TARGET_DB=$(basename "$BACKUP_FILE" | sed 's/_[0-9]\{8\}_[0-9]\{6\}\.dump$//')
+    # Extract database name from filename pattern: <dbname>_YYYYMMDD_HHMMSS.dump[.enc]
+    TARGET_DB=$(basename "$BACKUP_FILE" | sed 's/_[0-9]\{8\}_[0-9]\{6\}\.dump\(\.enc\)\?$//')
     info "Extracted database name from filename: $TARGET_DB"
 fi
 
+# Handle encrypted backups
+RESTORE_FILE="$BACKUP_FILE"
+DECRYPTED_TEMP=""
+if [[ "$BACKUP_FILE" == *.dump.enc ]]; then
+    ENCRYPTION_KEY="${BACKUP_ENCRYPTION_KEY:-}"
+    if [ -z "$ENCRYPTION_KEY" ] || [ ! -f "$ENCRYPTION_KEY" ]; then
+        error "Encrypted backup detected but BACKUP_ENCRYPTION_KEY is not set or file not found"
+        exit 1
+    fi
+    DECRYPTED_TEMP=$(mktemp /tmp/restore_XXXXXXXXXX.dump)
+    info "Decrypting backup..."
+    if openssl enc -d -aes-256-cbc -pbkdf2 -pass "file:${ENCRYPTION_KEY}" -in "$BACKUP_FILE" -out "$DECRYPTED_TEMP"; then
+        info "Backup decrypted to temp file"
+        RESTORE_FILE="$DECRYPTED_TEMP"
+    else
+        rm -f "$DECRYPTED_TEMP"
+        error "Failed to decrypt backup"
+        exit 1
+    fi
+    # Clean up temp file on exit
+    trap 'rm -f "$DECRYPTED_TEMP"' EXIT
+fi
+
 # Confirmation
 if [ "$SKIP_CONFIRM" = false ]; then
     warn "This will DROP and recreate database '$TARGET_DB'"
@@ -102,7 +125,7 @@ fi
 
 # Restore backup
 info "Restoring backup to $TARGET_DB..."
-if cat "$BACKUP_FILE" | docker compose -f "$COMPOSE_FILE" exec -T postgres pg_restore -U postgres -d "$TARGET_DB" --no-owner --no-acl; then
+if cat "$RESTORE_FILE" | docker compose -f "$COMPOSE_FILE" exec -T postgres pg_restore -U postgres -d "$TARGET_DB" --no-owner --no-acl; then
     info "  ✓ Backup restored successfully"
 else
     error "Failed to restore backup"