Never clean up an install a live process is still executing from#226
Open
dndungu wants to merge 2 commits into
Open
Never clean up an install a live process is still executing from#226dndungu wants to merge 2 commits into
dndungu wants to merge 2 commits into
Conversation
do_clean_old_versions runs on every launch and deletes older-version install dirs unconditionally. If another process is still executing from one of those dirs, its code disappears from under it and it dies later with a nofile module-load kernel panic (io_lib_pretty nofile) the first time it lazily loads a module -- typically minutes to hours after the deletion, which makes the crash look spurious. Record a pidfile per launch under <install>/.burrito_live/ before the exec (which preserves the wrapper's PID, keeping the pidfile accurate for the app's whole lifetime), have do_clean_old_versions skip any install with a live pidfile, prune stale pidfiles opportunistically, and add a BURRITO_NO_CLEAN_OLD env escape hatch that skips cleanup entirely. Windows keeps the previous behavior (no cheap liveness probe; in-use executables are protected by mandatory file locking there).
Production callers pass an arena, but the zig tests use std.testing.allocator, whose leak detector fails the run otherwise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
do_clean_old_versionsruns on every launch and deletes older-version install dirs unconditionally. If another process is still executing from one of those dirs (a long-running instance of the app started before a new version shipped), its code is deleted out from under it. It doesn't die immediately — the BEAM lazily loads some ERTS modules (io_lib_prettyloads on the first pretty-print) — so it crashes minutes to hours later with:which is very hard to attribute back to the cleanup. We hit this repeatedly on a machine running several concurrent long-lived instances of a burrito-packaged CLI while shipping multiple releases per day: every release window killed the in-flight runs (kazi-org/kazi#1018).
Fix
<install_dir>/.burrito_live/<pid>right before the exec. Since the exec preserves the PID, the pidfile stays accurate for the app's whole lifetime; no app-side cooperation needed.do_clean_old_versionsskips any install whose.burrito_live/contains a live PID (kill(pid, 0);EPERMcounts as alive), and prunes pidfiles whose process is gone — so pidfiles never accumulate and dirs are reclaimed on a later launch once the process exits.BURRITO_NO_CLEAN_OLDenv var skips the cleanup pass entirely, as an escape hatch.Deleting too little, late, is the correct failure direction here — the previous behavior deleted too much, immediately. PID reuse can keep a dead install alive spuriously in rare cases; the next cleanup pass reclaims it.
Testing
zig testblocks added insrc/maintenance.zigcovering: own PID alive, PID 1 reports alive via EPERM, live pidfile blocks cleanup, stale pidfile is pruned, garbage pidfile names ignored. Green on aarch64-linux and compile-checked on aarch64-macos with Zig 0.15.2.