Skip to content

Never clean up an install a live process is still executing from#226

Open
dndungu wants to merge 2 commits into
burrito-elixir:mainfrom
kazi-org:payload-liveness-guard-upstream
Open

Never clean up an install a live process is still executing from#226
dndungu wants to merge 2 commits into
burrito-elixir:mainfrom
kazi-org:payload-liveness-guard-upstream

Conversation

@dndungu

@dndungu dndungu commented Jul 9, 2026

Copy link
Copy Markdown

Problem

do_clean_old_versions runs on every launch and deletes older-version install dirs unconditionally. If another process is still executing from one of those dirs (a long-running instance of the app started before a new version shipped), its code is deleted out from under it. It doesn't die immediately — the BEAM lazily loads some ERTS modules (io_lib_pretty loads on the first pretty-print) — so it crashes minutes to hours later with:

{error,[{io_lib_pretty,nofile}]} ... io_lib:test_modules_loaded/3
DEFAULT FORMATTER CRASHED
Kernel pid terminated (application_controller)

which is very hard to attribute back to the cleanup. We hit this repeatedly on a machine running several concurrent long-lived instances of a burrito-packaged CLI while shipping multiple releases per day: every release window killed the in-flight runs (kazi-org/kazi#1018).

Fix

  • The launcher records its PID at <install_dir>/.burrito_live/<pid> right before the exec. Since the exec preserves the PID, the pidfile stays accurate for the app's whole lifetime; no app-side cooperation needed.
  • do_clean_old_versions skips any install whose .burrito_live/ contains a live PID (kill(pid, 0); EPERM counts as alive), and prunes pidfiles whose process is gone — so pidfiles never accumulate and dirs are reclaimed on a later launch once the process exits.
  • BURRITO_NO_CLEAN_OLD env var skips the cleanup pass entirely, as an escape hatch.
  • Windows keeps the existing behavior (no cheap liveness probe there, and in-use executables are protected by mandatory file locking anyway).

Deleting too little, late, is the correct failure direction here — the previous behavior deleted too much, immediately. PID reuse can keep a dead install alive spuriously in rare cases; the next cleanup pass reclaims it.

Testing

  • zig test blocks added in src/maintenance.zig covering: own PID alive, PID 1 reports alive via EPERM, live pidfile blocks cleanup, stale pidfile is pruned, garbage pidfile names ignored. Green on aarch64-linux and compile-checked on aarch64-macos with Zig 0.15.2.
  • Best-effort semantics throughout: pidfile writing failures never block a launch.

dndungu added 2 commits July 9, 2026 12:49
do_clean_old_versions runs on every launch and deletes older-version
install dirs unconditionally. If another process is still executing from
one of those dirs, its code disappears from under it and it dies later
with a nofile module-load kernel panic (io_lib_pretty nofile) the first
time it lazily loads a module -- typically minutes to hours after the
deletion, which makes the crash look spurious.

Record a pidfile per launch under <install>/.burrito_live/ before the
exec (which preserves the wrapper's PID, keeping the pidfile accurate
for the app's whole lifetime), have do_clean_old_versions skip any
install with a live pidfile, prune stale pidfiles opportunistically,
and add a BURRITO_NO_CLEAN_OLD env escape hatch that skips cleanup
entirely. Windows keeps the previous behavior (no cheap liveness probe;
in-use executables are protected by mandatory file locking there).
Production callers pass an arena, but the zig tests use
std.testing.allocator, whose leak detector fails the run otherwise.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant