Skip to content

[Bug Report] AppLauncher SIGTERM handler does not terminate Python workers #6530

Description

@RX-02333

Describe the bug

AppLauncher currently registers _abort_signal_handle_callback as the
SIGTERM handler. The callback calls SimulationApp.close() and then returns.

Installing a Python signal handler replaces the operating system's default
SIGTERM termination behavior. Returning from the callback therefore allows
Python execution to continue instead of terminating the process.

During distributed training shutdown, this can leave a worker alive after the
torchrun parent and TCPStore have exited. The surviving worker's
ProcessGroupNCCL heartbeat thread continues accessing the closed TCPStore and
repeatedly prints:

[rank1] TCPStore.cpp:106 sendBytes failed ... Broken pipe
[rank1] ProcessGroupNCCL.cpp:1802 Failed to check the "should dump" flag
on TCPStore ... Broken pipe

Steps to reproduce

From the Isaac Lab repository root, start a two-rank RSL-RL training job:

CUDA_VISIBLE_DEVICES=1,2 ./isaaclab.sh -p \
  -m torch.distributed.run \
  --standalone \
  --nnodes=1 \
  --nproc_per_node=2 \
  scripts/reinforcement_learning/rsl_rl/train.py \
  --task Isaac-Cartpole \
  --num_envs 2 \
  --max_iterations 5000 \
  --distributed \
  --viz none

After both ranks enter training, send SIGTERM to the
torch.distributed.run parent process:

kill -TERM <torchrun-pid>

Expected behavior:

  • Both workers terminate.
  • Normal Python cleanup paths run.
  • No training processes remain.
  • TCPStore Broken pipe messages are not repeatedly emitted.

Actual behavior:

  • The SIGTERM callback returns after closing SimulationApp.
  • A worker may continue executing.
  • Its ProcessGroupNCCL heartbeat thread continues accessing the closed
    TCPStore.
  • The terminal repeatedly prints Broken pipe errors.

System Info

  • Commit: 6f991d4becf764b151e0a1c775561ddd9c406f72
  • Isaac Sim Version: 6.0.1.0
  • OS: Ubuntu 24.04.4 LTS
  • GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition
  • CUDA: 12.8
  • GPU Driver: 610.43.02

Additional context

PR #5886 identified the missing SIGTERM termination behavior as an independent
latent bug. That PR was closed because it also bundled SIGHUP handling and a
force-exit mechanism. Its closing comment explicitly welcomed a smaller,
standalone fix.

PR #5933 addresses renderer/Fabric multi-GPU contention and does not change the
SIGTERM handler.

I searched the repository's open and closed issues and pull requests. There is
no equivalent issue and no matching fix that is currently open or already
merged.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

  • SIGTERM terminates the Python worker instead of returning to the interrupted execution path.
  • Normal Python cleanup paths remain available.
  • No worker processes remain after distributed shutdown.
  • Distributed shutdown does not repeatedly emit TCPStore Broken pipe errors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions