How is this issue impacting you?
Application hang
Description
ncclCheckpointRestore() re-establishes transports, and the paths it uses to do so have no deadline anywhere. When one of them stalls, the call never returns and never logs. From the application's point of view the restore entry point simply does not come back.
Observation (With 8 × L4, TP=8, vLLM)
23:19:45 sandbox restored
23:19:45 Rank 0-7: ncclCheckpointRestore() <- all 8 enter
23:26:45 our own RPC timeout fires <- 7 minutes, nothing in between
In those seven minutes NCCL emitted nothing at NCCL_DEBUG=WARN. Notably NCCL_CHECKPOINT_KVS_TIMEOUT (300 s) also did not fire, so the ranks had already completed rendezvous — the stall is downstream of it, in transport re-establishment, which no timeout covers. Only our own RPC deadline eventually broke the process out, into a cold start.
Where
Three loops on this path retry forever and exit only on abortFlag (and no one is setting it):
Asks
We have not identified which step stalls. We narrowed our own instance to a specific allocation path and reported that separately (see #2350) and it may turn out to be a limitation of the sandboxed runtime rather than anything in NCCL. We are raising this one on its own because the operability problem seemed worth separating from the cause: whatever is stalling, we had no way to distinguish "deadlocked" from "slow", and that is what made it expensive to chase.
NCCL Version
NCCL v2.30.7-1
How is this issue impacting you?
Application hang
Description
ncclCheckpointRestore() re-establishes transports, and the paths it uses to do so have no deadline anywhere. When one of them stalls, the call never returns and never logs. From the application's point of view the restore entry point simply does not come back.
Observation (With 8 × L4, TP=8, vLLM)
In those seven minutes NCCL emitted nothing at NCCL_DEBUG=WARN. Notably NCCL_CHECKPOINT_KVS_TIMEOUT (300 s) also did not fire, so the ranks had already completed rendezvous — the stall is downstream of it, in transport re-establishment, which no timeout covers. Only our own RPC deadline eventually broke the process out, into a cold start.
Where
Three loops on this path retry forever and exit only on abortFlag (and no one is setting it):
Asks
We have not identified which step stalls. We narrowed our own instance to a specific allocation path and reported that separately (see #2350) and it may turn out to be a limitation of the sandboxed runtime rather than anything in NCCL. We are raising this one on its own because the operability problem seemed worth separating from the cause: whatever is stalling, we had no way to distinguish "deadlocked" from "slow", and that is what made it expensive to chase.
NCCL Version
NCCL v2.30.7-1