[Bugfix] Keep the multi-port DP supervisor serving through fault-tolerant recovery

[Etelis]

With --data-parallel-multi-port-external-lb --enable-fault-tolerance, a failed DP rank halts the group, the engines recover in place within a few seconds after scale_down, and the removed rank's API server exits when its own sentinel gives up (engine_recovery_timeout_sec, 120s). DPSupervisor treats both as a group failure: _monitor_children tears everything down as soon as any child pid exits, and _probe_all_children does the same as soon as any child /health is not 200. So the supervisor kills a pool that already recovered, ~120s after the fault:

(EngineCore_DP0 pid=2490) INFO [engine_core_sentinel.py:231] [FT] Engine 0 scale_down complete: removed [1], cumulative dead_dp_ranks=[1]
...
(DPSupervisor pid=1) INFO [dp_supervisor.py:497] DPSupervisor found 1 exited DP Servers.
(DPSupervisor pid=1) INFO [dp_supervisor.py:534] DPSupervisor forwarding SIGTERM to DP Servers.

The same probe keeps /ready at 200 while the whole group is halted, because a rank's /health only checks that its engine process is alive.

Fix: once fault tolerance is on and the group has been ready, the supervisor polls each remaining child's /fault_tolerance/status instead of /health. A rank reporting unhealthy is recovering; a child that exited while peers are alive was removed by the engine and its port is dropped from the probe set. The supervisor tears down only after dp_supervisor_probe_failure_threshold consecutive sweeps in which no rank is serving or recovering, the same tolerance the /health probe had. /ready reflects "at least one rank healthy"; /health becomes a liveness signal that stays 200 until the supervisor itself decides to shut down, so a liveness probe on it does not race the recovery. Nothing changes without --enable-fault-tolerance.

Depends on #46370 (fault tolerance in external-LB mode). Related: #41438 (in-engine supervisor for internal-LB), ai-dynamo/grove#793.

Test

.venv/bin/python -m pytest tests/entrypoints/launchers/test_dp_supervisor.py -v   # 26 passed

E2E on H200: two pools of DeepSeek-V2-Lite, each one pod with --data-parallel-size-local 4 --data-parallel-multi-port-external-lb --enable-expert-parallel --all2all-backend nixl_ep --enable-eplb --eplb-config '{"num_redundant_experts": 24}' --enable-fault-tolerance, #46370 applied, behind an istio Gateway + llm-d endpoint picker v0.10.0 (InferencePool targetPorts 8000-8003). Pod readiness on :9256/ready, liveness on :9256/health. Load: 16 clients, ~16 rps, unique prefixes, 300s; at t=60s SIGKILL of Worker_DP1_EP1 in pool A. Same image and gateway for both columns.

	stock supervisor	this PR
scale_down sent at +32s: pool A at +125s	killed (found 1 exited DP Servers), 2/2 runs	serving on 3 ranks, 3/3 runs
scale_down sent at +32s: failed requests after +120s	16, 9	0, 0, 0
no scale_down: pod NotReady at	+132s	+34s, +38s, +35s
no scale_down: failed requests	51 / 1704	19 / 2006, 15 / 2054, 14 / 2066

Supervisor log with the fix:

(EngineCore_DP0 pid=2490) INFO 09-02 12:10:22 [engine_core_sentinel.py:231] [FT] Engine 0 scale_down complete: removed [1], cumulative dead_dp_ranks=[1]
(DPSupervisor pid=1) INFO 09-02 12:11:54 [dp_supervisor.py:549] DPSupervisor keeps serving; APIServer_DPRank_1 exited.

Failures inside the halt window (requests already parked on the pool, or routed to the dead port until it exits) are the same with and without this PR; readiness is per pod while the picker's endpoints are per port.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Etelis]

@njhill @tylertitsworth
Wdyt?