Summary
On NVIDIA GB10 systems (Dell Pro Max FCM1253, DGX OS 7.5.0), ib_write_bw RC RDMA WRITE fails deterministically for any message size above 65,536 bytes, with a responder-side local protection error (0x3b 0x0 0x9d) on the first WR. A minimal libibverbs program performing the same operation — same nodes, same RDMA devices, same MR/WR geometry — passes at every size tested up to 8 MiB, with responder-side content verification. NCCL subsequently sustained 24.0 GB/s busbw (all_reduce, 3 nodes) across the same fabric with zero validation errors. The failure appears to be a property of the instrument's interaction with this platform, not of the transport.
We believe this is the same defect class as two unresolved prior reports:
Three environments (x86 bare-metal, x86 KVM, ARM GB10), two link layers (IB, RoCE), three firmware generations, one tool, one boundary, one syndrome.
Environment
| Item |
Value |
| Nodes |
3× NVIDIA GB10 (Dell Pro Max FCM1253), byte-uniform software/firmware |
| OS / kernel |
DGX OS 7.5.0 / Ubuntu 24.04.4 / 6.17.0-1026-nvidia (aarch64) |
| NIC |
ConnectX-7, FW 28.45.4028, PSID NVD0000000087 (latest existing for this PSID) |
| perftest |
distro 24.01 / ib_write_bw v6.20 |
| Fabric |
RoCEv2, dual-rail, MikroTik CRS804 switches, MTU 9000, GID index 3 (IPv4) |
| IOMMU |
fault invariant to domain type (reproduced under both DMA-FQ and passthrough/identity) |
Reproduction
ib_write_bw (RC, defaults per v6.20), any message size > 65,536 bytes, any node pair, any of the four RDMA devices per node:
- ≤ 65,536 B: passes, sustains line rate indefinitely
- > 65,536 B (65,537 upward): initiator reports
Failed status 10: wr_id 0 syndrom 0x88 on the first WR; responder dmesg logs local protection error (0x3b 0x0 0x9d); boundary is exactly 16 × 4 KiB pages; device-, node-, and direction-independent (72/72 failing cells in our matrix)
Eliminated by test (each with a dedicated experiment)
- SMMU/IOMMU domain type (passthrough discriminator with untouched control node)
- Kernel/driver era (6.17.0-1014 vs -1026, mixed and homogeneous node pairings)
- Accumulated device state (fresh-image first-touch failures)
- Firmware currency (28.45.4028 is the newest firmware that exists for this PSID)
- MR extent, WR extent, WR offset, sequential coverage, repeat count, responder MR access flags, responder page touch-state, ODP, PCIe relaxed ordering (all via the independent probe — every geometry passes)
The discriminating instrument
Attached: fabric-probe.c — minimal libibverbs RC WRITE with responder-side content verification (position-derived pattern). It moves 4 MiB in a single WR flawlessly, byte-exact, on the identical stack where ib_write_bw fails at 68 KiB. It also reproduces perftest's exact buffer geometry (128K WR / 256K MR): passes.
Instrument-rigor note: our probe's v1.0 had its own connection-setup race (client posting before responder RTR; presents as REM_INV_REQ NAK). We characterized and fixed it (RTS-ready handshake) before drawing conclusions — the previously failing cells then passed 12/12 stress repeats. We are keenly aware of the irony and have applied the same skepticism to both instruments.
What we did not determine
The exact perftest-side mechanism (its QP attribute set or post pattern on this platform). RoCE data is NIC-offloaded, so an on-wire diff needs hardware-level capture we didn't pursue. Version-specificity beyond distro 24.01 / v6.20 is untested. Happy to run candidate patches or bisections on this hardware.
Why it matters
This tool is the community-standard fabric verifier. On this platform it reports a hard transport ceiling that does not exist — we (and, we suspect, the two prior reporters) lost significant time to a false hardware-defect conclusion. NCCL acceptance on the same fabric: 24.0 GB/s busbw, 3 nodes, dual-rail, per-MAC byte counters balanced to 0.02%, zero errors.
Summary
On NVIDIA GB10 systems (Dell Pro Max FCM1253, DGX OS 7.5.0),
ib_write_bwRC RDMA WRITE fails deterministically for any message size above 65,536 bytes, with a responder-side local protection error (0x3b 0x0 0x9d) on the first WR. A minimal libibverbs program performing the same operation — same nodes, same RDMA devices, same MR/WR geometry — passes at every size tested up to 8 MiB, with responder-side content verification. NCCL subsequently sustained 24.0 GB/s busbw (all_reduce, 3 nodes) across the same fabric with zero validation errors. The failure appears to be a property of the instrument's interaction with this platform, not of the transport.We believe this is the same defect class as two unresolved prior reports:
0x3b 0x0 0x9d— https://forums.developer.nvidia.com/t/ib-write-bw-local-protection-error-when-size-65536-in-kvm/282142Three environments (x86 bare-metal, x86 KVM, ARM GB10), two link layers (IB, RoCE), three firmware generations, one tool, one boundary, one syndrome.
Environment
Reproduction
ib_write_bw(RC, defaults per v6.20), any message size > 65,536 bytes, any node pair, any of the four RDMA devices per node:Failed status 10: wr_id 0 syndrom 0x88on the first WR; responder dmesg logslocal protection error (0x3b 0x0 0x9d); boundary is exactly 16 × 4 KiB pages; device-, node-, and direction-independent (72/72 failing cells in our matrix)Eliminated by test (each with a dedicated experiment)
The discriminating instrument
Attached:
fabric-probe.c— minimal libibverbs RC WRITE with responder-side content verification (position-derived pattern). It moves 4 MiB in a single WR flawlessly, byte-exact, on the identical stack where ib_write_bw fails at 68 KiB. It also reproduces perftest's exact buffer geometry (128K WR / 256K MR): passes.Instrument-rigor note: our probe's v1.0 had its own connection-setup race (client posting before responder RTR; presents as REM_INV_REQ NAK). We characterized and fixed it (RTS-ready handshake) before drawing conclusions — the previously failing cells then passed 12/12 stress repeats. We are keenly aware of the irony and have applied the same skepticism to both instruments.
What we did not determine
The exact perftest-side mechanism (its QP attribute set or post pattern on this platform). RoCE data is NIC-offloaded, so an on-wire diff needs hardware-level capture we didn't pursue. Version-specificity beyond distro 24.01 / v6.20 is untested. Happy to run candidate patches or bisections on this hardware.
Why it matters
This tool is the community-standard fabric verifier. On this platform it reports a hard transport ceiling that does not exist — we (and, we suspect, the two prior reporters) lost significant time to a false hardware-defect conclusion. NCCL acceptance on the same fabric: 24.0 GB/s busbw, 3 nodes, dual-rail, per-MAC byte counters balanced to 0.02%, zero errors.