fix: user photos survive redeploys — boot-time upload self-heal (#397)

## The bug
User avatars and cover images disappeared for everyone on **every
deploy**. This started after migrating to a replacement Railway service.

## Root cause
Avatars/covers are stored on the upload volume (not R2-primary); the DB
only holds the relative path (`/uploads/...`). On a service whose
uploads land on **ephemeral container disk** — no persistent volume
attached, or `ALLOW_EPHEMERAL_UPLOADS=true` — each redeploy starts with
an empty uploads dir while the rows still point at the files, so every
image 404s platform-wide. The volume→R2 mirror already existed, but
recovery was a **manual script** and the first backup pass only fired
24h after boot, so nothing self-healed.

## Fix (self-healing, defense in depth)
- **`uploadVolumeRestore.js`** (new): shared restore core — lists the R2
mirror and writes back any file missing locally (skip-if-exists,
traversal-guarded, **streams to a `.tmp` then atomic-renames** so an
OOM/eviction mid-write can't leave a truncated file). `restoreOnBoot()`
runs it non-blocking on startup (heartbeat-wrapped) so a fresh/wiped
container re-hydrates its photos with **no operator action**. A healthy
persistent volume pays only the R2 LIST cost.
- **`restoreVolumeFromR2.js`** (manual DR CLI) now delegates to that
core — the traversal guards live in one place, not two copies.
- **First backup pass ~2 min after boot**
(`UPLOAD_BACKUP_FIRST_PASS_DELAY_MS`) instead of a full interval later.
- **`storage.js`**: ephemeral-storage-in-production now logs an error +
Sentry alert (`storage.ephemeral_in_production`) instead of a silent
info line.
- `secretValidator` + `.env.example` document the two new env vars.

The durable config (persistent `/data` volume + all four R2 vars) is an
**operator action** documented in `RUNBOOK_DB_RESTORE.md` ("Photos
vanish on EVERY redeploy"). The code makes it self-healing once R2 is
configured; files that only ever lived on the wiped ephemeral disk can't
be recovered and need re-upload.

## Tests
- New `uploadVolumeRestore.test.js`: restore, skip-if-exists, force,
traversal refusal, pagination, dry-run, boot-enable policy.
- Full backend suite green (**3514 passed**); reviewed by code-reviewer
(2 findings — heap-OOM + non-atomic write — both fixed before commit).

## Summary by Sourcery

Add a shared upload-volume restore core and boot-time self-healing to
ensure user-uploaded files are automatically rehydrated from the R2
mirror after redeploys, and surface misconfigured ephemeral storage and
backup timing more loudly.

Bug Fixes:
- Ensure user avatars, covers, and other uploads are restored from the
R2 mirror on service startup so they no longer disappear after redeploys
when the uploads directory has been wiped or remounted.

Enhancements:
- Extract upload-volume restore logic into a reusable job module shared
by the disaster-recovery CLI and boot-time self-heal, including
traversal guards, streaming writes, and atomic file replacement.
- Run the first upload-volume backup pass shortly after boot instead of
waiting a full interval, reducing the window where new uploads exist
only on local disk.
- Raise a prominent error log and Sentry alert when running with
ephemeral upload storage in production instead of only logging at info
level.

Documentation:
- Document the new upload backup/restore configuration flags and the
photo-loss scenario in the release log and secret/env documentation.

Tests:
- Add unit tests for the upload-volume restore job covering restore
behavior, skip/force semantics, traversal protection, pagination,
dry-run mode, and boot-enable policy.
- Update storage validation tests to assert the new error-level logging
and Sentry alert when ephemeral storage is enabled in production.

Author: Apexone11

backend/.env.example

@@ -383,3 +383,13 @@ R2_BUCKET_UPLOAD_BACKUP=
 # Optional overrides — defaults to 24h cadence + 10 uploads/sec.
 UPLOAD_BACKUP_INTERVAL_MS=
 UPLOAD_BACKUP_RATE_LIMIT_PER_SEC=
+# Delay before the FIRST backup pass after boot (ms). Default 120000 (2 min).
+# Bounds how long a freshly-uploaded photo can sit only on local disk before
+# it reaches R2 — important on services that redeploy frequently.
+UPLOAD_BACKUP_FIRST_PASS_DELAY_MS=
+# Boot-time self-heal: on startup, pull any avatar/cover/attachment files
+# missing locally back from the R2 mirror (skip-if-exists, non-blocking).
+# Makes a wiped/ephemeral volume recover automatically instead of needing a
+# manual restoreVolumeFromR2.js run. Default ON in production (when R2 is
+# configured), OFF in dev. Set to "false" to disable, "true" to force on.
+UPLOAD_RESTORE_ON_BOOT=

backend/scripts/restoreVolumeFromR2.js

@@ -1,24 +1,17 @@
 /**
- * restoreVolumeFromR2.js — disaster-recovery for /data/uploads.
+ * restoreVolumeFromR2.js — disaster-recovery CLI for /data/uploads.
  *
- * Pairs with backend/src/lib/jobs/uploadVolumeBackup.js.
+ * Pairs with backend/src/lib/jobs/uploadVolumeBackup.js (the volume → R2
+ * mirror). The restore logic — listing, traversal guards, skip-if-exists —
+ * lives in src/lib/jobs/uploadVolumeRestore.js so the boot-time self-heal
+ * and this manual CLI share exactly one implementation. This file is the
+ * operator-facing wrapper: it parses flags, validates config, runs one pass,
+ * and reports.
  *
- * When to run: the Railway volume is gone, corrupted, or replaced. The
- * DB rows still point at /uploads/... paths; without restoring the
- * files every <img> 404s. Run this on the new instance BEFORE flipping
- * traffic back on.
- *
- * What it does:
- *   1. Lists every object under the `upload-volume-backup/` prefix in
- *      R2_BUCKET_UPLOAD_BACKUP.
- *   2. For each one, writes the file back into UPLOADS_DIR preserving
- *      the original relative path (`avatars/123.jpg` etc.).
- *   3. Refuses to overwrite existing files by default. Pass --force
- *      to overwrite (use ONLY when the volume is intentionally being
- *      wiped + repopulated; otherwise you risk replacing live newer
- *      content with stale backups).
- *   4. Skips R2 listing pagination boundaries cleanly — handles the
- *      ContinuationToken pattern S3 returns for buckets > 1000 keys.
+ * When to run: the Railway volume is gone, corrupted, or replaced and the
+ * boot-time restore didn't (or couldn't) cover it — e.g. you want a forced
+ * overwrite or a dry-run audit. The DB rows still point at /uploads/...
+ * paths; without the files every <img> 404s.
  *
  * Usage:
  *   # Inspect what would happen (no writes):
@@ -27,23 +20,21 @@
  *   # Actual restore (writes new files only, skip-if-exists):
  *   node backend/scripts/restoreVolumeFromR2.js
  *
- *   # Wipe + repopulate (use after volume reformat):
+ *   # Wipe + repopulate (use ONLY after a volume reformat — --force
+ *   # overwrites existing files and can replace live newer content with
+ *   # stale backups):
  *   node backend/scripts/restoreVolumeFromR2.js --force
  *
  * Required env: R2_BUCKET_UPLOAD_BACKUP + R2_ACCOUNT_ID +
  *               R2_ACCESS_KEY_ID + R2_SECRET_ACCESS_KEY.
  *
  * Exit codes: 0 success, 1 missing config / R2 unavailable, 2 partial
- * failure (some objects could not be restored — report shows which).
+ * failure (some objects could not be restored — log shows which).
  */
-const fs = require('node:fs')
 const path = require('node:path')
-const { ListObjectsV2Command, GetObjectCommand } = require('@aws-sdk/client-s3')
 
 require('dotenv').config({ path: path.resolve(__dirname, '../.env') })
 
-const { KEY_PREFIX } = require('../src/lib/jobs/uploadVolumeBackup')
-
 async function main() {
   const args = process.argv.slice(2)
   const dryRun = args.includes('--dry-run')
@@ -55,120 +46,30 @@ async function main() {
     process.exit(1)
   }
 
-  // Lazy-require the R2 client so we share the same singleton + config
-  // checks as the runtime app.
+  // Share the same credential check as the runtime app.
   const { isR2Configured } = require('../src/lib/r2Storage')
   if (!isR2Configured()) {
     console.error('FATAL: R2 credentials are not configured.')
     process.exit(1)
   }
 
-  // Build a dedicated S3 client for this script rather than reaching
-  // into r2Storage internals. Same credentials, same endpoint — keeps
-  // r2Storage's getClient() private and avoids tying script lifetimes
-  // to the runtime singleton's connection pool.
-  const { S3Client } = require('@aws-sdk/client-s3')
-  const client = new S3Client({
-    region: 'auto',
-    endpoint: `https://${process.env.R2_ACCOUNT_ID}.r2.cloudflarestorage.com`,
-    credentials: {
-      accessKeyId: process.env.R2_ACCESS_KEY_ID,
-      secretAccessKey: process.env.R2_SECRET_ACCESS_KEY,
-    },
-  })
-
   const { UPLOADS_DIR } = require('../src/lib/storage')
+  const { runRestorePass } = require('../src/lib/jobs/uploadVolumeRestore')
+
   console.log(`Restoring from R2 bucket "${bucket}" → ${UPLOADS_DIR}`)
   console.log(
     `Mode: ${dryRun ? 'DRY RUN' : force ? 'FORCE (overwrite existing)' : 'SAFE (skip-if-exists)'}`,
   )
 
-  let restored = 0
-  let skipped = 0
-  let failed = 0
-  let scanned = 0
-  let continuationToken
-
-  do {
-    const list = await client.send(
-      new ListObjectsV2Command({
-        Bucket: bucket,
-        Prefix: KEY_PREFIX,
-        ContinuationToken: continuationToken,
-      }),
-    )
-
-    for (const obj of list.Contents || []) {
-      scanned += 1
-      const relativePath = obj.Key.slice(KEY_PREFIX.length)
-
-      // Defense in depth: R2 is external storage. A compromised R2 credential
-      // or misconfigured bucket policy could inject keys like
-      // 'upload-volume-backup/../../etc/cron.d/payload', which path.join would
-      // happily resolve outside UPLOADS_DIR. Refuse any relative path that
-      // contains '..' segments or is absolute, and verify the resolved
-      // destination stays under UPLOADS_DIR.
-      if (
-        !relativePath ||
-        relativePath.startsWith('/') ||
-        relativePath.startsWith('\\') ||
-        path.isAbsolute(relativePath) ||
-        relativePath.split(/[\\/]/).includes('..')
-      ) {
-        console.warn(`REFUSED unsafe key (traversal attempt): ${obj.Key}`)
-        failed += 1
-        continue
-      }
-
-      const destination = path.join(UPLOADS_DIR, relativePath)
-
-      const resolvedDest = path.resolve(destination)
-      const resolvedRoot = path.resolve(UPLOADS_DIR)
-      if (resolvedDest !== resolvedRoot && !resolvedDest.startsWith(resolvedRoot + path.sep)) {
-        console.warn(`REFUSED escape from UPLOADS_DIR: ${obj.Key} → ${resolvedDest}`)
-        failed += 1
-        continue
-      }
-
-      if (fs.existsSync(destination) && !force) {
-        skipped += 1
-        continue
-      }
-      if (dryRun) {
-        console.log(`[dry-run] would restore ${obj.Key} → ${destination}`)
-        restored += 1
-        continue
-      }
-
-      try {
-        fs.mkdirSync(path.dirname(destination), { recursive: true })
-        const get = await client.send(new GetObjectCommand({ Bucket: bucket, Key: obj.Key }))
-        // The Body is a stream — collect to a buffer for the write.
-        // For multi-GB videos a streaming pipe would be better; we
-        // don't expect that volume here (avatars + small attachments).
-        const chunks = []
-        for await (const chunk of get.Body) chunks.push(chunk)
-        fs.writeFileSync(destination, Buffer.concat(chunks))
-        restored += 1
-        if (restored % 100 === 0) {
-          console.log(`Restored ${restored} files (skipped ${skipped}, failed ${failed})...`)
-        }
-      } catch (err) {
-        console.error(`FAILED ${obj.Key}: ${err?.message || err}`)
-        failed += 1
-      }
-    }
-
-    continuationToken = list.IsTruncated ? list.NextContinuationToken : undefined
-  } while (continuationToken)
+  const summary = await runRestorePass({ uploadsDir: UPLOADS_DIR, bucket, force, dryRun })
 
   console.log('')
   console.log('=== Restore summary ===')
-  console.log(`Scanned:  ${scanned}`)
-  console.log(`Restored: ${restored}`)
-  console.log(`Skipped:  ${skipped} (already on disk; use --force to overwrite)`)
-  console.log(`Failed:   ${failed}`)
-  process.exit(failed > 0 ? 2 : 0)
+  console.log(`Scanned:  ${summary.scanned}`)
+  console.log(`Restored: ${summary.restored}`)
+  console.log(`Skipped:  ${summary.skipped} (already on disk; use --force to overwrite)`)
+  console.log(`Failed:   ${summary.failed}`)
+  process.exit(summary.failed > 0 ? 2 : 0)
 }
 
 main().catch((err) => {

backend/src/index.js

@@ -905,6 +905,19 @@ async function startServer() {
     // RUNBOOK_DB_RESTORE.md "Upload Volume Recovery" section.
     const { startUploadVolumeBackup } = require('./lib/jobs/uploadVolumeBackup')
     const { UPLOADS_DIR } = require('./lib/storage')
+
+    // Self-heal: pull any missing avatar/cover/attachment files back from
+    // the R2 mirror on boot (skip-if-exists, non-blocking). When uploads
+    // sit on a non-persistent volume — a detached/remounted Railway volume
+    // or an ephemeral rebuild — a redeploy wipes the directory while the DB
+    // rows still point at /uploads/..., breaking every image platform-wide.
+    // Re-hydrating from R2 each boot makes that recoverable automatically
+    // instead of needing a manual restoreVolumeFromR2.js run. Runs BEFORE
+    // the backup is scheduled so a fresh container restores first, then
+    // mirrors. A healthy persistent volume pays only the R2 LIST cost.
+    const { restoreOnBoot } = require('./lib/jobs/uploadVolumeRestore')
+    restoreOnBoot({ uploadsDir: UPLOADS_DIR })
+
     startUploadVolumeBackup({ uploadsDir: UPLOADS_DIR })
 
     log.info({ port: PORT }, `Server running on http://localhost:${PORT}`)

backend/src/lib/jobs/uploadVolumeBackup.js

@@ -202,7 +202,7 @@ function startUploadVolumeBackup({ uploadsDir }) {
     'upload-volume backup scheduled',
   )
 
-  const handle = setInterval(() => {
+  const runPass = () => {
     // Lazy-require to avoid a boot-time require cycle through the
     // job runner.
     const { runWithHeartbeat } = require('./heartbeat')
@@ -211,7 +211,20 @@ function startUploadVolumeBackup({ uploadsDir }) {
       () => runBackupPass({ uploadsDir, bucket, rateLimitPerSec }),
       { slaMs: intervalMs / 2 },
     )
-  }, intervalMs)
+  }
+
+  // First pass shortly after boot rather than a full interval later. A
+  // short-lived container (frequent redeploys) could otherwise be torn
+  // down before its first nightly pass ever fired, so a freshly-uploaded
+  // photo would never reach R2 and would be lost on the next deploy.
+  // Default 2 min: long enough for boot to settle (and for the boot
+  // restore to finish first), short enough to bound the loss window.
+  const firstPassDelayMs =
+    Number.parseInt(process.env.UPLOAD_BACKUP_FIRST_PASS_DELAY_MS, 10) || 2 * 60 * 1000
+  const firstPass = setTimeout(runPass, firstPassDelayMs)
+  firstPass.unref()
+
+  const handle = setInterval(runPass, intervalMs)
   handle.unref()
   return handle
 }