Skip to content

Commit 14baa20

Browse files
committed
Stream large backups to disk instead of buffering in memory (#5434)
Backups that include files failed and crashed the server once the uploads directory got large (reporter hit it around 30GB). buildBackupBuffer read every file into an adm-zip object and then called zip.toBuffer(), holding the entire archive in RAM at once. That both exceeded Node's Buffer size limit (RangeError) and exhausted the heap (OOM), so the process died. Structure-only backups (~300KB) were unaffected, matching the report. Replace the in-memory build with a streaming archiver(zip) pipeline that writes each entry straight to a file on disk, keeping memory bounded no matter how big the uploads are: - buildBackupBuffer -> buildBackupFile(include, outPath): streams to disk, returns a Promise. The temp VACUUM'd DB clone is unlinked only after the archive stream closes. - Download endpoint builds to a temp file then res.download()s it (no more res.send of a multi-GB buffer), cleaning the temp file afterward. - Auto-backup streams to a .partial file then renames it into place, so a crash mid-write can't leave a truncated .zip. Prune/list now skip dotfiles. Output is a standard zip; the existing restore path reads it back byte-for-byte (verified via SHA-256 round-trip through the adm-zip reader).
1 parent 5baa16e commit 14baa20

3 files changed

Lines changed: 1112 additions & 107 deletions

File tree

0 commit comments

Comments
 (0)