test infra: abort watchdog so a deadlocked run cannot survive Jenkins abort and hold the device hub#15422
Open
Nir-Az wants to merge 3 commits into
Open
test infra: abort watchdog so a deadlocked run cannot survive Jenkins abort and hold the device hub#15422Nir-Az wants to merge 3 commits into
Nir-Az wants to merge 3 commits into
Conversation
…run cannot survive Jenkins abort and hold the device hub
sysrsbuild-gh-agentic
left a comment
sysrsbuild-gh-agentic
left a comment
Collaborator
There was a problem hiding this comment.
⚙️ Auto-generated review by rs-agentic bot
![aikido-pr-checks[bot]](https://avatars.githubusercontent.com/in/898896?s=60&v=4){kind=small}
AviaAv
reviewed
Jul 16, 2026
AviaAv
left a comment
AviaAv
left a comment
Contributor
There was a problem hiding this comment.
Looks good, some small comments
| ''' | ||
|
|
||
|
|
||
| def start_abort_watchdog(grace=None): |
Contributor
There was a problem hiding this comment.
unused parameter, the tests override WATCHDOG_GRACE_S directly
| def wait_for_ready(proc, timeout=20): | ||
| deadline = time.monotonic() + timeout | ||
| while time.monotonic() < deadline: | ||
| line = proc.stdout.readline() |
Contributor
There was a problem hiding this comment.
readline is blocking, so this wouldn't work here - let's use timeout(60) at the top of the file instead and simplify wait_for_ready
| already running). See _WATCHDOG_CODE above for why it must be a separate process. | ||
| """ | ||
| global _watchdog | ||
| if os.name != 'posix' or _watchdog is not None: |
Contributor
There was a problem hiding this comment.
_watchdog is never reset, let's use
if os.name != 'posix' or (_watchdog is not None and _watchdog.poll() is None):
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview: A Jenkins abort can no longer leave an immortal test process holding the Acroname hub: a small watchdog process guarantees the run dies even when it is deadlocked in native code.
Tracked on [RSDSO-21761]
Why
On 2026-07-14 every Linux CI agent started failing Acroname connect with result=25 (aErrConnection). Root cause: aborted builds left orphaned pytest processes deadlocked in native code (mutex/GIL AB-BA). The Python SIGTERM handler in
rspy.signalscan never run in that state — signal handlers only execute when the main thread returns to the interpreter loop — so the process survived the abort (Jenkins: "After 20s process did not stop") and kept the BrainStem USB link open, failing every subsequent build on the agent.Summary
rspy/signals.py:register_signal_handlers()now also starts a GIL-independent watchdog subprocess (POSIX only). It receives the same SIGTERM/SIGINT as the run (same process tree), waits a 60s grace period so normal cleanup can finish, then SIGKILLs the run if it is still alive. The kernel then closes all fds, releasing the hub.infra-tests/test_signals_watchdog.py): stuck-cleanup gets SIGKILLed, clean abort exits normally without watchdog interference, no watchdog process leaks after normal exit.Test plan
🤖 Generated by AI