Force-kill hung ipdevpoll on stop/restart (fixes #4125)#4126
Open
adobloug wants to merge 2 commits into
Open
Conversation
DaemonService.stop() sent SIGTERM up to three times over a ~12 second window and then gave up, reporting failure. It never escalated to SIGKILL and only ever signalled the master pid. A daemon whose main thread is blocked or GIL-starved during shutdown can neither run its SIGTERM handler nor exit within that window, so "nav stop"/"nav restart" would time out and leave the process (and, for ipdevpoll -m, its worker children) behind. Send SIGTERM to the whole process group, poll for exit on a monotonic clock during a grace period, then escalate to SIGKILL -- which cannot be caught, blocked or deferred -- signalling the group again so orphaned or zombie child processes are cleaned up along with the master. Refs Uninett#4125 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ipdevpoll's SIGTERM handler is cooperative: it only schedules reactor.stop(), which relies on the main thread reaching the reactor loop to run the shutdown triggers. Under load -- especially in multiprocess mode -- the main thread can block in the thread pool join, or be starved of the GIL by a busy C-extension thread, so shutdown never completes and the process appears to ignore SIGTERM entirely, spinning at high CPU while its workers are left as <defunct> children. Arm a watchdog timer when a shutdown signal is received. If the reactor has not shut down cleanly within the configurable "shutdown_timeout" (default 30s), the watchdog calls os._exit(), bypassing the reactor and interpreter teardown to guarantee the process dies. The timer is cancelled on a clean shutdown. Fixes Uninett#4125 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
adobloug
force-pushed
the
bugfix/4125-ipdevpoll-stop-timeout
branch
from
July 9, 2026 08:37
bfd07f7 to
c2f7aa7
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4125
Problem
nav stop/nav restart ipdevpollfrequently time out and reportFailedwhen ipdevpoll runs in multiprocess mode (
ipdevpolld -m), especially on busyinstallations. The master process outlives the stop attempt and its workers are
left behind as
<defunct>(zombie) children.Root cause
Two independent defects combine:
nav stopnever force-kills.DaemonService.stop()sentSIGTERMup tothree times over ~12s and then gave up. It never escalated to
SIGKILL, andonly signalled the master pid — not the worker children.
only schedules
reactor.stop(), which needs the main thread to reach thereactor loop. During shutdown the main thread blocks in the thread pool join
(no timeout), or is starved of the GIL by a busy C-extension thread — so the
handler cannot run and
SIGTERMappears ignored (observed: master at 99% CPU,eight
<defunct>workers).Verified on NAV 5.19.1 — this is a shutdown/kill-handling bug, not a
DB-driver bug.
Fix
nav.startstop: sendSIGTERMto the whole process group, poll for exit ona monotonic clock during a grace period, then escalate to
SIGKILL(uncatchable)and signal the group again to sweep orphan/zombie children.
os._exit()if the reactorhas not shut down within a configurable
shutdown_timeout(default 30s),guaranteeing the process dies even when the reactor is wedged. Cancelled on a
clean shutdown.
Both fixes are workload- and driver-agnostic.
Tests
tests/unittests/startstop_test.py: SIGTERM→SIGKILL escalation ordering,process-group signalling, single-process fallback, already-gone handling.
tests/unittests/ipdevpoll/daemon_test.py: watchdog arm (once), cancel onclean shutdown, and forced
os._exitwhen it fires.🤖 Generated with Claude Code