Implement multi-node vLLM health check#7
Implement multi-node vLLM health check#7TomerBN-Nvidia wants to merge 4 commits intoultra-rl-v0.17from
Conversation
- Add configurable RPC timeouts (VLLM_RPC_TIMEOUT=300s default) - Add load progress monitor: detects stalled workers by tracking shard-by-shard weight loading progress (VLLM_LOAD_STALL_TIMEOUT=120s) - Implement real check_health() in Worker (GPU probe + NCCL allreduce) - Add __ray_check_health__ to RayWorkerWrapper for Ray-native detection - Restore NCCL_ASYNC_ERROR_HANDLING (was silently removed, broke error detection) - Add orphaned process cleanup on startup (nvidia-smi based) - Add SIGTERM/atexit shutdown handlers for clean teardown - Add get_load_progress() API for health monitor polling Part of Nemotron Ultra 3 RL resiliency effort.
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run You ask your reviewers to trigger select CI tests on top of Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. 🚀 |
- Use TORCH_NCCL_ASYNC_ERROR_HANDLING (old name deprecated) - get_load_progress() now reads per-shard progress from default_loader during active loading, enabling the stall monitor to track real progress
- Remove LoadProgressMonitor (RPC polling doesn't work — Ray actors are single-threaded, can't poll while load_model is running) - Use simple collective_rpc timeout for load_model instead (VLLM_MODEL_LOAD_TIMEOUT, default 600s) - Remove per-shard progress tracking from worker and loader - Keep: RPC timeouts, check_health(), NCCL fix, orphan cleanup
The compiled DAG bypasses dead actors, so collective_rpc('check_health')
can succeed even when a worker is dead. Fix: ping each actor with a
lightweight RPC (get_node_ip) first. RayActorError means the worker
was killed; GetTimeoutError means it's hung.
1 participant
Part of Nemotron Ultra 3 RL resiliency effort.
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.