Describe the bug
AppLauncher currently registers _abort_signal_handle_callback as the
SIGTERM handler. The callback calls SimulationApp.close() and then returns.
Installing a Python signal handler replaces the operating system's default
SIGTERM termination behavior. Returning from the callback therefore allows
Python execution to continue instead of terminating the process.
During distributed training shutdown, this can leave a worker alive after the
torchrun parent and TCPStore have exited. The surviving worker's
ProcessGroupNCCL heartbeat thread continues accessing the closed TCPStore and
repeatedly prints:
[rank1] TCPStore.cpp:106 sendBytes failed ... Broken pipe
[rank1] ProcessGroupNCCL.cpp:1802 Failed to check the "should dump" flag
on TCPStore ... Broken pipe
Steps to reproduce
From the Isaac Lab repository root, start a two-rank RSL-RL training job:
CUDA_VISIBLE_DEVICES=1,2 ./isaaclab.sh -p \
-m torch.distributed.run \
--standalone \
--nnodes=1 \
--nproc_per_node=2 \
scripts/reinforcement_learning/rsl_rl/train.py \
--task Isaac-Cartpole \
--num_envs 2 \
--max_iterations 5000 \
--distributed \
--viz none
After both ranks enter training, send SIGTERM to the
torch.distributed.run parent process:
kill -TERM <torchrun-pid>
Expected behavior:
- Both workers terminate.
- Normal Python cleanup paths run.
- No training processes remain.
- TCPStore
Broken pipe messages are not repeatedly emitted.
Actual behavior:
- The SIGTERM callback returns after closing
SimulationApp.
- A worker may continue executing.
- Its ProcessGroupNCCL heartbeat thread continues accessing the closed
TCPStore.
- The terminal repeatedly prints
Broken pipe errors.
System Info
- Commit:
6f991d4becf764b151e0a1c775561ddd9c406f72
- Isaac Sim Version:
6.0.1.0
- OS: Ubuntu 24.04.4 LTS
- GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
- CUDA: 12.8
- GPU Driver: 610.43.02
Additional context
PR #5886 identified the missing SIGTERM termination behavior as an independent
latent bug. That PR was closed because it also bundled SIGHUP handling and a
force-exit mechanism. Its closing comment explicitly welcomed a smaller,
standalone fix.
PR #5933 addresses renderer/Fabric multi-GPU contention and does not change the
SIGTERM handler.
I searched the repository's open and closed issues and pull requests. There is
no equivalent issue and no matching fix that is currently open or already
merged.
Checklist
Acceptance Criteria
Describe the bug
AppLaunchercurrently registers_abort_signal_handle_callbackas theSIGTERM handler. The callback calls
SimulationApp.close()and then returns.Installing a Python signal handler replaces the operating system's default
SIGTERM termination behavior. Returning from the callback therefore allows
Python execution to continue instead of terminating the process.
During distributed training shutdown, this can leave a worker alive after the
torchrunparent and TCPStore have exited. The surviving worker'sProcessGroupNCCL heartbeat thread continues accessing the closed TCPStore and
repeatedly prints:
Steps to reproduce
From the Isaac Lab repository root, start a two-rank RSL-RL training job:
After both ranks enter training, send SIGTERM to the
torch.distributed.runparent process:Expected behavior:
Broken pipemessages are not repeatedly emitted.Actual behavior:
SimulationApp.TCPStore.
Broken pipeerrors.System Info
6f991d4becf764b151e0a1c775561ddd9c406f726.0.1.0Additional context
PR #5886 identified the missing SIGTERM termination behavior as an independent
latent bug. That PR was closed because it also bundled SIGHUP handling and a
force-exit mechanism. Its closing comment explicitly welcomed a smaller,
standalone fix.
PR #5933 addresses renderer/Fabric multi-GPU contention and does not change the
SIGTERM handler.
I searched the repository's open and closed issues and pull requests. There is
no equivalent issue and no matching fix that is currently open or already
merged.
Checklist
Acceptance Criteria
Broken pipeerrors.