test: tolerate non-zero Triton exit on Jetson graceful shutdown by Vinya567 · Pull Request #8881 · triton-inference-server/server

Vinya567 · 2026-07-13T20:47:15Z

What does the PR do?

Fix L0_backend_python/lifecycle failures on IGX-Orin when pytest already passed.

Root cause

Lifecycle pytest passed; the subtest failed during server shutdown.
On IGX-Orin, seven Python backend stubs can exceed Triton's default 30s exit timeout.
Server logged Exit timeout expired, exited non-zero, and wait $SERVER_PID failed the script under set -e.
Incomplete stub teardown could leave orphaned triton_python_backend_shm_region_* files in /dev/shm.

Fix

Jetson: set --exit-timeout-secs=${SERVER_TIMEOUT} and stub-timeout-seconds=${SERVER_TIMEOUT} in qa/L0_backend_python/lifecycle/test.sh.
Jetson: use wait $SERVER_PID || true in kill_server (qa/common/util.sh).
Jetson: snapshot shm before server start, check page count first, then cleanup only new regions after the assertion (qa/common/util.sh, lifecycle test).
Runtime follow-up: fix: ensure Python backend shm cleanup on forced shutdown python_backend#450

Checklist

PR title reflects the change and is of format <type>: <description>
Changes are described in the pull request.
Related issues are referenced.
Populated github labels field
Added test plan and verified test passes.
Verified that the PR passes existing CI.
Verified copyright is correct on all changed files.
Added succinct git squash message before merging
All template sections are filled out.

Commit Type:

Related PRs:

python_backend: fix: ensure Python backend shm cleanup on forced shutdown python_backend#450

Where should the reviewer start?

qa/L0_backend_python/lifecycle/test.sh
qa/common/util.sh

Test plan:

Local: qa/L0_backend_python/lifecycle/test.sh with TEST_JETSON=1.
GitLab Orin: job 366443271 passed at commit c1c44b50 (Lifecycle test PASSED, shm count unchanged).
Latest commit 3bbc4e05 (run shm check before cleanup) still needs Orin re-run.

Caveats:

Server QA changes are in this PR.
Python backend runtime changes are in PR Apply CustomInstance API wrapper #450.

Related Issues:

yinggeh

What is the root cause of "on Jetson, Triton may exit non-zero during graceful shutdown after Python backend stub teardown even when tests passed"?

Vinya567 · 2026-07-15T06:10:02Z

What is the root cause of "on Jetson, Triton may exit non-zero during graceful shutdown after Python backend stub teardown even when tests passed"?

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm. This PR handles both in the QA harness on Jetson. Fixed in e2c7891.

yinggeh · 2026-07-16T15:32:36Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Vinya567 · 2026-07-16T15:44:50Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

yinggeh · 2026-07-16T15:49:19Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Vinya567 · 2026-07-16T15:53:39Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Makes sense , I'll dig into the Python backend shutdown path on Orin to find the root cause and file an NVBUG if it's a backend issue. I'll update here once I have findings.

Vinya567 · 2026-07-17T23:15:15Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Root cause: lifecycle pytest passed, but Triton’s 30s shutdown timeout was too short for 7 Python stubs on Orin. Server exited with error, Python backend left an shm file behind, and the test failed on cleanup. We increased shutdown timeouts for Jetson, added harness cleanup as a safety net, and have a python_backend fix for proper stub teardown/shm cleanup. Verified green on Orin (job 366443271).

yinggeh · 2026-07-18T00:20:22Z

On Orin, pytest passes but wait $SERVER_PID returns non-zero after shutdown, and sometimes a Python backend shm file is left in /dev/shm.

I understand. But what's the root cause for this buggy behavior? Why it's hardware specific?

Good point , we only traced the CI symptom so far, not the full backend root cause. On IGX-Orin, pytest passes but shutdown sometimes exits non-zero before Python stub teardown finishes, which can leave shm files behind; we haven't reproduced that on x86. This PR is the QA harness fix for TRI-1330; I can dig into the Python backend shutdown path as well if you'd prefer that before merge.

Please do. The goal of fixing tests is not simply to work around the issue, but to identify the root cause and file an NVBUG with the appropriate team if necessary.

Root cause: lifecycle pytest passed, but Triton’s 30s shutdown timeout was too short for 7 Python stubs on Orin. Server exited with error, Python backend left an shm file behind, and the test failed on cleanup. We increased shutdown timeouts for Jetson, added harness cleanup as a safety net, and have a python_backend fix for proper stub teardown/shm cleanup. Verified green on Orin (job 366443271).

Great. Make sure you linked the python_backend PR in the description (following the template format).

yinggeh · 2026-07-18T00:25:39Z

          sed -i "s/TYPE_FP32/TYPE_UINT32/g" config.pbtxt)

 prev_num_pages=`get_shm_pages`
+if [ "${TEST_JETSON}" == "1" ]; then


With snapshot_triton_python_shm and cleanup_triton_python_shm_since_snapshot, the test won't detect any stale shm after server exits, which could be a problem.

you're right that cleanup in kill_server runs before the shm count check, so it can mask leaks. I'll move cleanup to after the assertion (or remove it from kill_server and rely on the exit-timeout fix + python_backend PR). Will also link the python_backend PR in the description.

Fixed in 3bbc4e0: the shm page-count assertion runs before cleanup_triton_python_shm_since_snapshot, and kill_server no longer does preemptive cleanup. Orin re-run for that commit is still pending.

Also if it helps the test we should make it universal to all platforms.

I looked at ShmLeakDetector vs the shell helpers.
They’re not quite the same: the detector is skipped on Jetson, and it checks free memory inside existing pools during pytest. The shell snapshot/cleanup handles leftover region files after server exit, which is what get_shm_pages() in test.sh was already watching.

Given that overlap, I’ll re-run Orin without snapshot/cleanup and drop them if it stays green. If we still need them, I’ll remove the TEST_JETSON guard and make them universal.

yinggeh · 2026-07-18T00:54:23Z


+TRITON_PYTHON_SHM_SNAPSHOT=""
+
+function snapshot_triton_python_shm () {


Check

server/qa/L0_backend_python/lifecycle/lifecycle_test.py

Lines 66 to 68 in c1b7cb7

class LifecycleTest(unittest.TestCase):

def setUp(self):

self._shm_leak_detector = shm_util.ShmLeakDetector()

and find out if the shm functions are redundant.

yinggeh · 2026-07-21T19:08:53Z

-        wait $SERVER_PID
+        # On Jetson, Triton may exit non-zero during graceful shutdown after
+        # Python backend stub teardown even when tests passed.
+        if [ "${TEST_JETSON}" == "1" ]; then


Do you still need special handling for Jetson?

Yeah, but just two things now , shm cleanup is already out.

Orin still needs the 120s timeout and wait || true here. Without them the lifecycle test fails even when pytest passes.

Orin is green with only those (2734eb5). I kept the Jetson guard so other platforms don't silently ignore a bad shutdown.

Nit: Have a generic wait $SERVER_PID || true for all platforms is fine. Just like your PR https://github.com/triton-inference-server/server/pull/

Done ,dropped the Jetson guard and made wait $SERVER_PID || true universal.

yinggeh

LGTM

pskiran1 · 2026-07-23T17:45:10Z

-# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# SPDX-FileCopyrightText: Copyright (c) 2018-2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.


I think you need to revert the copyright change. Please rebase with the main branch and run pre_commit again.

Similar happened to me in the logging PR.
#8858 (comment)

Done ,reverted the SPDX header, rebased on main, pre-commit passes. PR is now a clean 2-file diff.

Extend the exit-timeout window and Python stub-teardown window on Jetson (${SERVER_TIMEOUT}, currently 120s) for L0_backend_python/lifecycle, and make kill_server in qa/common/util.sh tolerate a non-zero wait exit on all platforms so "set -e" does not abort a suite when the server exits non-zero after shutdown. Root cause on IGX-Orin: pytest passes but the default 30s shutdown window is too short to tear down all Python stubs, so the server exits non-zero and can leave a shm file behind, causing "wait $SERVER_PID" under "set -e" to abort the rest of the suite.

greptile-apps · 2026-07-23T18:13:55Z

Greptile Summary

This PR tolerates non-zero Triton exit codes during graceful shutdown on Jetson/IGX-Orin, where seven Python backend stubs can exceed the default 30 s exit timeout, causing wait $SERVER_PID under set -e to fail the test script even after pytest passed.

qa/L0_backend_python/lifecycle/test.sh: Under TEST_JETSON=1, appends --exit-timeout-secs=${SERVER_TIMEOUT} and --backend-config=python,stub-timeout-seconds=${SERVER_TIMEOUT} to SERVER_ARGS so Triton and its stubs get up to 120 s to shut down cleanly.
qa/common/util.sh: Changes wait $SERVER_PID to wait $SERVER_PID || true in kill_server to prevent a non-zero server exit from failing the surrounding script; however, this change is unconditional and applies to all non-Windows platforms, not just Jetson.

Confidence Score: 4/5

Safe to merge for its intended Jetson fix; the global || true is a mild regression for non-Jetson CI visibility.

The Jetson-conditional SERVER_ARGS change is well-scoped and correct. The || true in kill_server is intentional for Jetson but applied unconditionally to every non-Windows test run. On non-Jetson hosts a server crash during shutdown is now silently accepted; tests without a follow-on shm-page check lose their last crash signal. This is unlikely to hide real bugs in practice — pytest results are the primary gate — but it does reduce CI coverage depth outside Jetson.

qa/common/util.sh — the unconditional wait $SERVER_PID || true affects every non-Windows test in the repo and should be reviewed if any test relies on kill_server propagating the server's exit code.

Important Files Changed

Filename	Overview
qa/common/util.sh	Added `
qa/L0_backend_python/lifecycle/test.sh	Adds Jetson-guarded `--exit-timeout-secs` and `stub-timeout-seconds` SERVER_ARGS using `SERVER_TIMEOUT` (default 120 s), which may be insufficient if stub teardown is sequential and slow under load.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[kill_server called] --> B{Windows?}
    B -- Yes --> C[tasklist / taskkill]
    B -- No --> D{MSYSTEM set?}
    D -- Yes --> E[taskkill //F //IM tritonserver.exe]
    D -- No --> F[kill SERVER_PID SIGTERM]
    F --> G["wait SERVER_PID || true"]
    G --> H{Exit code?}
    H -- 0 clean shutdown --> I[kill_server returns 0]
    H -- non-zero timeout or crash --> J["|| true swallows exit"]
    J --> I
    I --> K[Caller continues under set -e]
    K --> L{Has shm-page check?}
    L -- Yes --> M[Orphaned pages detected as RET=1]
    L -- No --> N[Server crash silently hidden]

_{Reviews (1): Last reviewed commit: "test: fix L0_backend_python/lifecycle sh..." | Re-trigger Greptile}

greptile-apps · 2026-07-23T18:13:59Z

        # Non-windows...
        kill $SERVER_PID
-        wait $SERVER_PID
+        wait $SERVER_PID || true


Unconditional || true masks server exit codes on all platforms

The || true is applied to every non-Windows host, not just Jetson. When a server exits non-zero during kill_server due to a crash or timeout on a standard x86 CI runner, the error is silently swallowed and the calling script under set -e sees success. Only tests that also perform a shm-page count check have a secondary safety net; tests without that check now lose the last signal that the server process ended badly.

Suggested change

wait $SERVER_PID || true

if [ "${TEST_JETSON}" == "1" ]; then

wait $SERVER_PID || true

else

wait $SERVER_PID

fi

greptile-apps · 2026-07-23T18:14:00Z

+if [ "${TEST_JETSON}" == "1" ]; then
+    SERVER_ARGS="${SERVER_ARGS} --exit-timeout-secs=${SERVER_TIMEOUT} --backend-config=python,stub-timeout-seconds=${SERVER_TIMEOUT}"
+fi


stub-timeout-seconds is unbounded by SERVER_TIMEOUT semantics

SERVER_TIMEOUT (default 120 s) is designed as a readiness-wait budget, not a stub-shutdown budget. On Jetson it now doubles as both. If a Jetson CI runner is heavily loaded and stub teardown takes longer than 120 s in total (the root cause was seven stubs exceeding Triton's total 30 s budget), the same condition could recur with a different threshold. Consider defining a separate STUB_TIMEOUT with a value chosen specifically for Jetson stub teardown (e.g. 300 s), or at minimum adding a comment documenting why SERVER_TIMEOUT is adequate here.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Vinya567 self-assigned this Jul 15, 2026

Vinya567 requested review from mc-nv, whoisj and yinggeh July 15, 2026 01:38

yinggeh reviewed Jul 15, 2026

View reviewed changes

Comment thread qa/L0_backend_python/common.sh Outdated

Comment thread qa/common/util.sh Outdated

Vinya567 requested a review from yinggeh July 15, 2026 06:12

mc-nv reviewed Jul 16, 2026

View reviewed changes

Comment thread qa/common/util.sh Outdated

yinggeh reviewed Jul 18, 2026

View reviewed changes

Vinya567 mentioned this pull request Jul 18, 2026

fix: ensure Python backend shm cleanup on forced shutdown triton-inference-server/python_backend#450

Merged

5 tasks

yinggeh reviewed Jul 18, 2026

View reviewed changes

Vinya567 requested a review from yinggeh July 21, 2026 18:55

yinggeh reviewed Jul 21, 2026

View reviewed changes

Vinya567 requested a review from yinggeh July 21, 2026 19:44

yinggeh previously approved these changes Jul 21, 2026

View reviewed changes

Vinya567 dismissed yinggeh’s stale review via a6048a0 July 21, 2026 20:25

Vinya567 requested a review from yinggeh July 21, 2026 20:26

yinggeh previously approved these changes Jul 21, 2026

View reviewed changes

Vinya567 dismissed yinggeh’s stale review via f90c582 July 21, 2026 20:29

Vinya567 requested review from pskiran1 and yinggeh July 21, 2026 20:37

pskiran1 reviewed Jul 23, 2026

View reviewed changes

Vinya567 force-pushed the vinyak/tri-1330-fix-ci-test-l0_backend_python-igx-orin branch from cdc4d7e to 6b19f0f Compare July 23, 2026 18:10

greptile-apps Bot reviewed Jul 23, 2026

View reviewed changes

Vinya567 requested a review from pskiran1 July 23, 2026 18:14

yinggeh approved these changes Jul 23, 2026

View reviewed changes

Vinya567 merged commit 3515afd into main Jul 23, 2026
4 checks passed

Vinya567 deleted the vinyak/tri-1330-fix-ci-test-l0_backend_python-igx-orin branch July 23, 2026 18:20


		TRITON_PYTHON_SHM_SNAPSHOT=""

		function snapshot_triton_python_shm () {

	class LifecycleTest(unittest.TestCase):
	def setUp(self):
	self._shm_leak_detector = shm_util.ShmLeakDetector()

Uh oh!

Conversation

Vinya567 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does the PR do?

Root cause

Fix

Checklist

Commit Type:

Related PRs:

Where should the reviewer start?

Test plan:

Caveats:

Related Issues:

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Vinya567 commented Jul 15, 2026

Uh oh!

yinggeh commented Jul 16, 2026

Uh oh!

Uh oh!

Vinya567 commented Jul 16, 2026

Uh oh!

yinggeh commented Jul 16, 2026

Uh oh!

Vinya567 commented Jul 16, 2026

Uh oh!

Vinya567 commented Jul 17, 2026

Uh oh!

yinggeh commented Jul 18, 2026

Uh oh!

yinggeh Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinggeh Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vinya567 Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

pskiran1 Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Jul 23, 2026

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Uh oh!

greptile-apps Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Jul 23, 2026

Vinya567 commented Jul 13, 2026 •

edited

Loading

yinggeh Jul 18, 2026 •

edited

Loading

yinggeh Jul 21, 2026 •

edited

Loading

Vinya567 Jul 21, 2026 •

edited

Loading

pskiran1 Jul 23, 2026 •

edited

Loading