Skip to content

Graceful shutdown on SIGTERM + fix rootfs mount leak that wedges slots#16

Merged
guifranchi merged 1 commit into
mainfrom
fix/graceful-shutdown-and-mount-leak
Jul 16, 2026
Merged

Graceful shutdown on SIGTERM + fix rootfs mount leak that wedges slots#16
guifranchi merged 1 commit into
mainfrom
fix/graceful-shutdown-and-mount-leak

Conversation

@guifranchi

@guifranchi guifranchi commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #15. Draft for review discussion, not to be merged/deployed yet.

What

Two independent fixes, one per defect described in #15:

1. Graceful shutdown (new manager/src/shutdown.rs)

The v0.1.11 handler ignored SIGTERM/SIGINT (SigHandler::SigIgn), so the manager could never exit on a stop signal — every systemctl stop/restart hung for TimeoutStopSec and ended in systemd SIGKILLing the manager and all Firecracker VMs (how ci-hel3 ended up dead for 4 days).

Now: the signal sets an AtomicBool; a watcher thread SIGKILLs each registered Firecracker process group (they setsid() at spawn) so the slots' blocked child.wait() calls return; each slot loop observes the flag, runs teardown (stale-mount cleanup, PID file removal), and exits; the manager joins all slots and exits 0. The 20s boot-failure back-off is now interruptible, and reattached VMs (PID-file recovery path) are registered with the watcher too.

Design choice: running VMs are killed, not drained. Same job impact as today's SIGKILL, but with clean teardown and a fast, bounded stop. A drain-with-grace-period can be layered on later if wanted.

2. Rootfs mount leak

setup_run() fetched the GitHub registration token and injected config while the rootfs was mounted; any error in that window (a GitHub API hiccup is enough) returned with the mount still held. Mounts are kernel state. From then on every cycle fails lvcreate snapshot failed: ... contains a filesystem in use, retried every 20s forever. Three staging slots wedged this way in 5 days (appsignal/appsignal-ansible#806: ci-hel1 slot 0, ci-hel2 slot 7, ci-hel3 slot 6).

Fixes:

  • registration token fetched before mounting (no network call can fail while mounted);
  • unmount now also runs on the injection error path;
  • each cycle defensively unmounts a stale mnt left by a crash/SIGKILL before touching LVM, so a wedged slot self-heals on the next cycle instead of requiring manual fuser -km/umount/lvremove.

Testing

  • cargo check, cargo test --workspace (12 tests), cargo clippy, all clean.
  • Not yet run on a real host. Suggested verification on one staging host after build: systemctl restart actions-runner mid-job --> unit stops in seconds, exits 0 (no Failed with result 'timeout'), no stale /srv/runners/*/*/mnt mounts, slots come back cleanly.

Related: #14 (cache LV fixes), ideally both ship together in v0.1.12 so the fleet needs a single rollout. Ansible-side interim mitigation: appsignal/appsignal-ansible#811.

Two fixes for issue #15:

1. Replace the no-op SIGTERM/SIGINT handler with a real shutdown path:
   the signal sets a global flag, a watcher thread SIGKILLs registered
   Firecracker process groups so blocked child.wait() calls return, and
   each slot loop tears down (unmount, PID file) and exits, letting the
   manager exit 0 instead of hanging until systemd's stop timeout and
   getting SIGKILLed with everything it owns.

2. Fix the rootfs mount leak that wedges slots: setup_run() fetched the
   GitHub registration token and injected config while the rootfs was
   mounted, so any error in that window (a GitHub API hiccup suffices)
   returned with the mount held -- making every subsequent boot cycle
   fail 'lvcreate snapshot failed: contains a filesystem in use' forever.
   The token is now fetched before mounting, the unmount runs on the
   error path too, and each cycle defensively cleans up a stale mount
   left by a crash or SIGKILL before touching LVM.

Three staging slots wedged this way within 5 days
(appsignal/appsignal-ansible#806).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@backlog-helper

backlog-helper Bot commented Jul 14, 2026

Copy link
Copy Markdown

✔️ All good!

New issue guide | Backlog management | Rules | Feedback

@guifranchi
guifranchi marked this pull request as ready for review July 16, 2026 20:05
@guifranchi
guifranchi merged commit 5cacc0f into main Jul 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Manager ignores SIGTERM: every stop ends in SIGKILL after TimeoutStopSec; orphaned mounts wedge slots

2 participants