[None][fix] mpi_session: guard MpiPoolSession.shutdown against partial init#16613
[None][fix] mpi_session: guard MpiPoolSession.shutdown against partial init#16613lowsfer wants to merge 1 commit into
Conversation
…l init MpiPoolSession.shutdown() is reachable from __del__ (via shutdown_abort) and may run on a session that never completed __init__ -- e.g. after release_exit_joins() marks the pool dead, or in the released-session path exercised by test_proxy_fast_death.py:: test_pool_session_shutdown_never_blocks_after_release, which builds the object with __new__. shutdown() read self.n_workers (log line) and self._wait_shutdown unguarded, raising AttributeError, while the _pool_dead check right above already used getattr defensively. Guard both accesses with getattr, matching the existing _pool_dead style, so a released/partially-constructed session shuts down without crashing. This unbreaks a cross-PR merge-skew regression on main: the unguarded self.n_workers log came from NVIDIA#16456 while the __new__-based test came from NVIDIA#16312; each was green alone but the combination fails deterministically. Also add the missing D205 docstring blank lines the ruff-legacy gate flagged on the touched file. Verified: the whole test_proxy_fast_death.py file now passes (23 passed, was 1 failed). Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughChangesMPI session updates
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Comment |
|
/bot run --disable-fail-fast |
|
PR_Github #60349 [ run ] triggered by Bot. Commit: |
|
PR_Github #60349 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60380 [ run ] triggered by Bot. Commit: |
|
PR_Github #60380 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60420 [ run ] triggered by Bot. Commit: |
|
PR_Github #60420 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #60600 [ run ] triggered by Bot. Commit: |
Summary
MpiPoolSession.shutdown()readsself.n_workers(in a log line) andself._wait_shutdownunguarded, but it is reachable on a session that never completed__init__:__del__→shutdown_abort()→shutdown()on a partially-constructed object, andtests/unittest/executor/test_proxy_fast_death.py::test_pool_session_shutdown_never_blocks_after_release, which builds the object withMpiPoolSession.__new__(MpiPoolSession).In those states the attributes are absent and
shutdown()raisesAttributeError: 'MpiPoolSession' object has no attribute 'n_workers'(then... '_wait_shutdown'). The_pool_deadcheck immediately above already usesgetattr(self, '_pool_dead', False)defensively for exactly this reason.Fix
Guard both accesses with
getattr(...), matching the existing_pool_deadstyle, so a released or partially-constructed session shuts down cleanly instead of raising.Why main is currently red
This is a cross-PR merge skew, each half green on its own:
self.n_workerslog line was added by [None][test] disagg startup: detect and fail fast on worker OOM/crash #16456 (2026-07-16), and__new__-based test that trips it was added by [TRTLLM-13409][fix] make proxy shutdown non-blocking when the engine is dead #16312 (2026-07-20).Neither failed in isolation; the combination on
mainfails deterministically.mainhas been red ontest_pool_session_shutdown_never_blocks_after_releasesince #16312 landed, and the fix is a small robustness guard.Test
test_proxy_fast_death.pynow passes in full (23 passed, was 1 failed):(No behavior change for fully-initialized sessions:
n_workers/_wait_shutdownare always set by__init__; the guards only affect objects that bypass it.)🤖 Generated with Claude Code
Summary by CodeRabbit
Documentation
Bug Fixes