Commit 14baa20
committed
Stream large backups to disk instead of buffering in memory (#5434)
Backups that include files failed and crashed the server once the uploads
directory got large (reporter hit it around 30GB). buildBackupBuffer read
every file into an adm-zip object and then called zip.toBuffer(), holding
the entire archive in RAM at once. That both exceeded Node's Buffer size
limit (RangeError) and exhausted the heap (OOM), so the process died.
Structure-only backups (~300KB) were unaffected, matching the report.
Replace the in-memory build with a streaming archiver(zip) pipeline that
writes each entry straight to a file on disk, keeping memory bounded no
matter how big the uploads are:
- buildBackupBuffer -> buildBackupFile(include, outPath): streams to disk,
returns a Promise. The temp VACUUM'd DB clone is unlinked only after the
archive stream closes.
- Download endpoint builds to a temp file then res.download()s it (no more
res.send of a multi-GB buffer), cleaning the temp file afterward.
- Auto-backup streams to a .partial file then renames it into place, so a
crash mid-write can't leave a truncated .zip. Prune/list now skip dotfiles.
Output is a standard zip; the existing restore path reads it back
byte-for-byte (verified via SHA-256 round-trip through the adm-zip reader).1 parent 5baa16e commit 14baa20
3 files changed
Lines changed: 1112 additions & 107 deletions
0 commit comments