Better Error messages/handling when shuffle corrupts data

[ayushdg]

Context:
I was running RMPF shuffle on a machine which has some configuration issue. 2x A100 GPUs with no nvlink (PCIE.

 

  ┌─────┬──────────┬─────────────┬──────────────┐

  │ GPU │ PCI addr │ IOMMU group │ Root complex │

  ├─────┼──────────┼─────────────┼──────────────┤

  │ 0   │ 21:00.0  │ 46          │ pci0000:20   │

  ├─────┼──────────┼─────────────┼──────────────┤

  │ 1   │ 41:00.0  │ 33          │ pci0000:40   │

  └─────┴──────────┴─────────────┴──────────────┘

The two GPUs are in different IOMMU groups under different PCIe root complexes, with IOMMU enabled (70 groups, no iommu=pt in /proc/cmdline). GPU↔GPU P2P DMA across separate root complexes, routed through IOMMU translation, is a well-known silently-broken configuration.

This was corrupting data silently during the shuffle leading to weird error messages:

=== EXCEPTION ===
RayTaskError(MemoryError): std::bad_alloc: out_of_memory: RMM failure ...
  Maximum pool size exceeded (failed to allocate 16.000000 EiB)
  failed to allocate 18446744072992228352 bytes (~2^64 -> integer overflow)

=== TRACEBACK (trimmed) ===
  ShuffleStageAdapter.extract_and_write()
  .../stages/deduplication/fuzzy/lsh/stage.py:160  extract_and_write()
  .../stages/deduplication/fuzzy/lsh/lsh.py:308     extract_and_write()
  .../stages/deduplication/fuzzy/lsh/lsh.py:273     extract_and_group()
  .../stages/deduplication/shuffle_utils/rapidsmpf_shuffler.py:277  unpack_and_concat(unspill_partitions(...))
  rapidsmpf/integrations/cudf/partition.pyx:225  unpack_and_concat
  MemoryError: std::bad_alloc: out_of_memory (CUDA cudaErrorMemoryAllocation)

Disabling P2P by moving CUDA_IPC from UCX_TLS helps resolve the issue.
Side note: Even NCCL workflows required NCCL_P2P_DISABLE=1 to work as expected.

The open question here is if there can be better error messages/handling for these cases and if it's worth pursuing.
For context NCCL doesn't silently pass corrupt data but ends up hanging somehow.

[wence-]

We kind of rely on the underlying transport to be able to send messages. If we can't do that how can we trust that a checksum will make it correctly?

Is there something we can do instead to diagnose this misconfiguration?

[ayushdg]

Do any of the tests/validation benchmarks a user can run on a given machine cover this case? If so we can use that as a proxy to validate things like the transport being broken in the first place.

possibly related: #235. It doesn't have to be ray specific

[pentschev]

This is something we have observed previously, and I remember debugging this on a different system (L4 I think) with @quasiben about 2 months ago. The problem is this is a niche hardware case, and I've seen this happen in at least 3 different systems (including yours now), but because of this being niche its priority is likely to continue being low unless there's a surge in such systems.

For short-term the best option is simply to have UCX_TLS=^cuda_ipc in those systems.

[ayushdg]

Yeah, feel free to close this issue or leave it open if it helps tracking occurrences if this comes up in the future or others run into it. I agree that it's hard to check at runtime if the transport is corrupted or not. If more cases do come in, some recommendation to run test/validation scripts either from nccl/ucx or something that rapidsmpf bundles that users can use to troubleshoot will be helpful.