Cross-node NVMe → DRAM bandwidth aggregation benchmark

Aggregates the read bandwidth of every NVMe drive in a 2-node cluster into the DRAM of a single node ("the sink"):

        SINK NODE                                        SOURCE NODE
  ┌──────────────────────────────────┐     ┌──────────────────────────────────┐
  │ nvme0n1 ┐                        │     │                        ┌ nvme0n1 │
  │ nvme1n1 ┼─ O_DIRECT pread ──┐    │     │    ┌── O_DIRECT pread ─┼ nvme1n1 │
  │ nvme2n1 ┘                   ▼    │     │    ▼                   └ nvme2n1 │
  │                    ╔═════════════╗│     │╔═══════════╗                     │
  │                    ║  sink DRAM  ║◄─────╢│  staging  ║  registered MRs     │
  │                    ╚═════════════╝│ RDMA│╚═══════════╝                     │
  └──────────────────────────────────┘WRITE└──────────────────────────────────┘
                                    400G InfiniBand

• Same-node drives land in sink DRAM directly via O_DIRECT pread.
• Remote drives are read into registered staging DRAM on the source node and pushed into sink DRAM with one-sided RDMA WRITE. The sink's CPU is not involved in receiving — no recv, no CQE, no interrupt.
• Drives are opened O_RDONLY|O_DIRECT. The benchmark never writes to a disk.

Results

Measured on 2 × Xeon Platinum 8480C nodes (2 TB DRAM each, 3 NVMe each, 8 × 400G IB NDR + 1 × 100G RoCE), 1 MiB reads, 12 threads/drive.

drives	local	RDMA	total	ideal	% of ideal
1	3.57	—	3.57	3.57	100.0%
2	7.13	—	7.13	7.14	99.9%
3	10.71	—	10.71	10.71	100.0%
4	10.66	3.57	14.23	14.28	99.6%
5	10.71	7.14	17.85	17.85	100.0%
6	10.71	10.70	21.41	21.42	100.0%

Scaling is linear across the local→remote boundary — the RDMA hop costs nothing. A remote drive delivers the same 3.57 GB/s into sink DRAM as a local one. Symmetric in both directions (either node as sink). Every aggregate run passed the end-to-end data verification.

Only half the total crosses the network: 10.7 GB/s (85.6 Gb/s), unidirectional, which is ~21% of a single 400G port. The drives are the bottleneck, not the fabric.

Why 3.57 GB/s per drive

All six drives advertise LnkCap2: Supported Link Speeds: 2.5-8GT/s — PCIe Gen3 x4:

8 GT/s × 4 lanes = 32 GT/s raw ÷ 128b/130b ≈ 3.94 GB/s − TLP overhead ≈ 3.6 GB/s

Measured 3.57 GB/s, flat at 8/12/24/48 threads — a link ceiling, not a concurrency or software limit. This is a drive limit, not a platform one: two of the three upstream ports advertise 2.5-32GT/s with Target Link Speed: 32GT/s and report the link (downgraded), i.e. the host is requesting Gen5 and the drive answers Gen3. Nothing in BIOS, setpci, or the driver can raise it.

Note the drives are SAMSUNG MZ1L21T9HCLS-00AMV / MZ1L2960HCJR-00AMV (fw GDC79M4Q) — PM9A3 part numbers, a Gen4 family rated ~6.5 GB/s. Whether the Gen3 cap is silicon or a firmware/strap limit is unresolved.

Faster-drive projections

Per drive: Gen4 x4 ≈ 7.2 GB/s usable link (drives deliver ~6.8); Gen5 x4 ≈ 14.5 (drives deliver ~13.5). At Gen3 the link is the limit; at Gen4/Gen5 the drive is.

Caveat: on this platform nvme0's slot on each node is Gen3-only at the upstream port (LnkCap2: 2.5-8GT/s), so a faster drive there gains nothing until that port is fixed. The other four slots are Gen5-ready.

scenario	6-drive aggregate
Gen3 (today, measured)	21.4 GB/s
Gen4, 2 slots still Gen3	~34 GB/s
Gen4, all slots fixed	~41 GB/s
Gen5, 2 slots still Gen3	~61 GB/s
Gen5, all slots fixed	~81 GB/s

At Gen5 the remote half (~40 GB/s = 324 Gb/s) reaches ~88% of one 400G port — still fine across multiple NICs (QPs already spread over 8 ports), but "RDMA is free" stops being automatic. Also watch nvme2's switch chain, which contains two Gen2 x16 bridges (~8 GB/s) that would cap a Gen5 drive.

Files

File	Purpose
nvmebw.c	the benchmark — single binary, three roles
run_bench.sh	two-node orchestration (run from a host shell)
scale_sweep.sh	sweeps 1→6 drives, produces scale_results.txt
plot_scaling.py	renders scaling.png
start_nvmebw.sh	starts the container on a node

Running

Start the container on both nodes (privileged, host network, RDMA + NVMe access):

./start_nvmebw.sh          # on each node

Build (inside the container):

gcc -O2 -o nvmebw nvmebw.c -libverbs -lpthread

Then from either node's host shell:

./run_bench.sh aggregate    # all 6 drives -> sink DRAM
./run_bench.sh local        # per-node local read only
./run_bench.sh baseline     # one drive at a time

Choose the destination node and tuning via env vars:

SINK_NODE=nodeA SRC_NODE=nodeB ./run_bench.sh aggregate
DURATION=60 THREADS=16 CHUNK=2M ./run_bench.sh aggregate

Manual invocation (start the sink first):

# sink node
./nvmebw --role sink --port 18515 --dram 16G --duration 20
# source node
./nvmebw --role src --peer <sink-ip> --port 18515 --duration 20

Options

--role local|sink|src   local = single node, no RDMA
--peer HOST             sink address (source only)
--disks a,b,c           default: all /dev/nvmeXnY
--devs d1,d2            RDMA devices (default: all ACTIVE InfiniBand ports)
--threads N             reader threads per disk (default 12)
--depth N               outstanding RDMA writes per worker (default 4)
--chunk BYTES           IO / RDMA size (default 1M)
--span BYTES            bytes of each disk to sweep (default: whole disk)
--dram BYTES            sink DRAM buffer (default 16G)
--duration / --warmup   measurement window (default 20s / 3s)
--gid N                 GID index (default 0 for IB, 3 for RoCE)
--no-verify             skip the end-to-end data check

RDMA device names differ per node (mlx5_0…8 vs ibp7s0, ibp29s0, …). Auto-detection handles this; only pass --devs with names valid on that node.

How it works

• Disk side — threads threads per drive, each owning a disjoint region and sweeping it sequentially with O_DIRECT (page cache bypassed, so numbers are real device reads). Threads are pinned to the NUMA node of their drive.
• Network side — one RC QP per worker (36 QPs for 3 drives × 12 threads), spread across IB ports on the same NUMA node as the drive. Each worker keeps depth RDMA WRITEs in flight over depth staging slots, so disk reads and NIC transmission overlap. A 1 MiB message is segmented by the HCA into 256 × 4 KB wire packets and surfaces as one WQE/CQE.
• Accounting — bytes count only on the CQE, i.e. once the NIC confirms the write landed remotely. Because one-sided RDMA gives the sink no visibility into inbound writes, the remote total is measured at the source and summed with the sink's local total. Valid here because both streams are steady (<0.5% variance, measurement windows overlap 99.8%); it would not be valid for bursty traffic.
• Destination layout — the sink buffer splits: lower half receives RDMA writes (one disjoint slice per remote worker), upper half takes local reads. Every stream owns a private slice, so no cross-stream contention.
• Verification — the source reads a real chunk off its disk, hashes it (FNV-1a), RDMA-writes it to a reserved canary slot, and the sink re-hashes what landed, proving the bytes in sink DRAM are the bytes that came off the remote NVMe. Reported as RDMA data verification: PASS.

Limitations

• Roles are mutually exclusive — there is no simultaneous bidirectional mode. Adding --role both would be a small change (each node holding both a destination MR and staging buffers), useful for fabric stress but it would not raise the aggregate, which is fixed by the drives.
• Data is staged through the source node's DRAM (NVMe → DRAM → NIC), costing a memory round-trip. Free at these rates; PCIe peer-to-peer (p2pdma) would remove it and starts to matter at Gen5 speeds.
• Chunk size was not swept — 1 MiB was chosen as a known-good default. Thread count was swept (8/12/24/48, flat).
• Requires libibverbs and an RC-capable fabric. liburing is not used.