[Issue]: NCCL Checkpoint - Restore deadlocks forever in SHM's cuMem host-NUMA path; NCCL_CUMEM_HOST_ENABLE=0 fixes it

[nicolexin]

How is this issue impacting you?

Application hang

Share Your Debug Logs

Environment

NCCL 2.30.7-1, CUDA driver 580.126.20, 8× L4 (PCIe, no NVLink), TP=8 as 8 separate worker processes in one pod, vLLM 0.19.1, GKE 1.36.2 with GKE Pod Snapshots (cuda-checkpoint inside a gVisor sandbox checkpoint).

Symptom

After the sandbox checkpoint completes, all 8 ranks enter ncclCheckpointRestore() and none return. No error, no warning at NCCL_DEBUG=WARN, and NCCL_CHECKPOINT_KVS_TIMEOUT (300 s) never fires — so the ranks did rendezvous.

Possible Root Cause (bisected to one env var)

• First time run into this issue without NCCL_SHM_DISABLE (which causes the hang). After setting NCCL_SHM_DISABLE=1 the hang is gone and checkpoint / restore was successful.
• Second run I remove NCCL_SHM_DISABLE=1 and added NCCL_CUMEM_HOST_ENABLE=0 the hang is also gone and checkpoint / restore was successful.

What we have not determined: We know which path fails, not which step in it. The stall could be in NCCL checkpoint shim and it could be a gVisor limitation or a driver limitation around cuda-checkpoint-restored contexts rather than anything in NCCL.

Questions

1. Is checkpoint/restore expected to work when SHM selects the cuMem host-NUMA allocator for multi-GPU checkpoint/restore? If it is not expected to, that is a completely reasonable answer and mostly a documentation question for us.
2. If it is expected to work, we are happy to help localize. Just tell us what would be most useful and we will run it.
3. Separately, whatever is stalling, we could not tell that anything was stalling — no output at all for seven minutes. That part we have filed on its own as [Issue]: NCCL Checkpoint - ncclCheckpointRestore() can block indefinitely with no timeout and no diagnostic #2349.

Steps to Reproduce the Issue

No response

NCCL Version

NCCL 2.30.7-1

Your platform details

No response

Error Message & Behavior

No response