[TRTLLM-13409][fix] hard-kill all ranks when one rank's executor loop crashes by JunyiXu-nv · Pull Request #16592 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-07-20T05:22:30Z

Summary by CodeRabbit

Bug Fixes
- Improved distributed executor crash handling to prevent peer processes from stalling.
- Added a configurable grace period before terminating the full distributed job after a crash.
- Preserved local cleanup and original error reporting before shutdown.
- Safely handles disabled or invalid grace-period settings without introducing additional failures.
Tests
- Added coverage for single- and multi-process crash handling, grace periods, configuration validation, cleanup ordering, and clean exits.

Description

When a rank's executor loop dies on an exception, the rank stops participating in collectives but nothing tells its peers: every peer blocks in its next collective until its own HangDetector fires 300s later, and the whole multi-GPU test session burns that long for an error that was already known (the A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop).

Single-rank worlds are exempt (no peers to unblock), and the kill helper never raises (it runs in a finally where an exception would mask the loop's original error).

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

… crashes When a rank's executor loop dies on an exception, the rank stops participating in collectives but nothing tells its peers: every peer blocks in its next collective until its own HangDetector fires 300s later, and the whole multi-GPU test session burns that long for an error that was already known (the A4 AutoDeploy catches are this signature: peers crash, the survivor wedges in ADP until the 300s backstop). Escalate at error time instead: after the loop's local cleanup has woken rank-local waiters, hard-kill the world via the ST-1 propagation path. A grace period (TLLM_RANK_CRASH_HARD_KILL_GRACE, default 10s, negative disables) lets the cleaner error paths win the race first -- the stashed error reaches rank-local response waiters, the init-phase ready handshake returns the real exception to the proxy, and the worker future completes with the original error -- so the client reports the actual failure rather than a bare worker death. Single-rank worlds are exempt (no peers to unblock), and the kill helper never raises (it runs in a finally where an exception would mask the loop's original error). Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-07-20T05:23:46Z

/bot run --disable-fail-fast

coderabbitai · 2026-07-20T05:25:01Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1691863c-d53a-4259-a487-e5d7cc028201

📥 Commits

Reviewing files that changed from the base of the PR and between 56c3734 and cf7fdc1.

📒 Files selected for processing (3)

tensorrt_llm/_torch/pyexecutor/hang_detector.py
tensorrt_llm/_torch/pyexecutor/py_executor.py
tests/unittest/_torch/executor/test_hang_detector_kill.py

📝 Walkthrough

Walkthrough

Adds configurable rank-crash hard-kill behavior with a grace period, integrates it into PyExecutor event-loop crash handling, and tests propagation, disabling, fallback parsing, cleanup ordering, exception preservation, and clean exits.

Changes

Rank crash handling

Layer / File(s)	Summary
Configurable rank-crash kill policy `tensorrt_llm/_torch/pyexecutor/hang_detector.py`	Adds the grace-period environment setting, parsing with fallback and disable behavior, delayed hard-kill propagation, and defensive exception handling.
Executor crash integration `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Tracks event-loop crashes and invokes `hard_kill_on_rank_crash` after local cleanup while preserving the original exception behavior.
Kill behavior validation `tests/unittest/_torch/executor/test_hang_detector_kill.py`	Tests rank-size handling, grace timing, disabled and invalid configuration, propagation failures, cleanup ordering, exception recording, and clean exits.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PyExecutor._event_loop_wrapper
  participant hard_kill_on_rank_crash
  participant propagate_hard_kill
  PyExecutor._event_loop_wrapper->>PyExecutor._event_loop_wrapper: Store exception and run cleanup
  PyExecutor._event_loop_wrapper->>hard_kill_on_rank_crash: Pass world size
  hard_kill_on_rank_crash->>propagate_hard_kill: Propagate hard kill after grace period

Suggested reviewers: nv-xtf

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title matches the PR’s main change and follows the required ticket-and-type format.
Description check	✅ Passed	The description explains the issue and fix, but the Test Coverage section is left empty.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

tensorrt-cicd · 2026-07-20T05:29:40Z

PR_Github #60270 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

tensorrt-cicd · 2026-07-20T08:49:03Z

PR_Github #60270 [ run ] completed with state SUCCESS. Commit: cf7fdc1
/LLM/main/L0_MergeRequest_PR pipeline #48629 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-21T02:48:21Z

/bot run

tensorrt-cicd · 2026-07-21T02:55:20Z

PR_Github #60547 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

tensorrt-cicd · 2026-07-21T03:08:02Z

PR_Github #60547 [ run ] completed with state FAILURE. Commit: cf7fdc1
/LLM/main/L0_MergeRequest_PR pipeline #48859 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-07-21T05:04:02Z

/bot run

tensorrt-cicd · 2026-07-21T05:09:51Z

PR_Github #60580 [ run ] triggered by Bot. Commit: cf7fdc1 Link to invocation

JunyiXu-nv requested review from a team as code owners July 20, 2026 05:22

JunyiXu-nv requested review from YihuiLu512, nv-xtf, reasonsolo and zhaoyangwang-nvidia July 20, 2026 05:22

github-actions Bot assigned JunyiXu-nv Jul 20, 2026

JunyiXu-nv force-pushed the dev-junyix-feat-rank-crash-hard-kill branch from 03e5e9e to cf7fdc1 Compare July 20, 2026 05:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[TRTLLM-13409][fix] hard-kill all ranks when one rank's executor loop crashes#16592

[TRTLLM-13409][fix] hard-kill all ranks when one rank's executor loop crashes#16592
JunyiXu-nv wants to merge 1 commit into
NVIDIA:mainfrom
JunyiXu-nv:dev-junyix-feat-rank-crash-hard-kill

JunyiXu-nv commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

JunyiXu-nv commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JunyiXu-nv commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

JunyiXu-nv commented Jul 20, 2026

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

JunyiXu-nv commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JunyiXu-nv commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading