Many-Field CPU Copies

[lightsighter]

Summary

Commit 442cab5 (#284) added a fast path for copies over hundreds or thousands of fields per instance, but only wired it into the CUDA channels. This change generalizes the same optimization to the CPU-side DMA channels — MemcpyChannel (same-node host memory) and RemoteWriteChannel (cross-node via active messages / RDMA) — so that Legion workloads with many-field copies benefit on CPU targets too.

The generic infrastructure (InstanceLayoutGeneric::idindexed_fields, IDIndexedFieldsIterator, FieldBlock, field-aware AddressListCursor, MIN_IDINDEXED_FIELDS) already shipped with #284 — this change implements the CPU-side progress routines and removes the runtime flag that gated the feature, letting Realm pick the fast-vs-slow path automatically from the copy parameters.

Cost model

The decision is made from knowns at graph-construction time (no user-facing knob).

Given N fields over domain volume V with per-field size S, bandwidth B, and total bytes W = N · V · S:

T_slow = N · C_iter + ⌈W/P⌉ · C_pkt + W/B // per-field iterator
T_fast = C_iter + C_fb + ⌈W/P⌉ · C_pkt + W/B // IDIndexedFieldsIterator + FieldBlock
↓
T_slow − T_fast = (N−1) · C_iter − C_fb

Bandwidth and packetization terms cancel; only iterator/XferDes setup overhead differs. C_fb is a single alloc_obj + field-ID memcpy, comparable in magnitude to C_iter, so the break-even is N ≈ 2. The existing MIN_IDINDEXED_FIELDS = 2 constant already encodes this and is now documented inline. At N = 1 only the slow path is valid; the fast path is gated off via this threshold.

Three gates decide fast-path eligibility at graph-construction time:

1. Layout supports it (idindexed_fields && is.dense()) — existing.
2. Channel knows how to batch for this memory pair (Channel::support_idindexed_fields(src, dst), virtual, default false) — existing, now overridden on CPU channels.
3. Field count crosses the break-even (fields.size() >= MIN_IDINDEXED_FIELDS) — existing.

Changes

MemcpyChannel — local host-memory fast path

src/realm/transfer/memcpy_channel.{h,cc}

• Override support_idindexed_fields: returns true for any memory pair the channel already accepts via add_path (local system memory + remote-shared-memory segments reachable through get_direct_ptr). The multi-field branch reuses the same memcpy_1d/2d/3d machinery as the legacy path, so no additional capability check is needed.
• Restructure MemcpyXferDes::progress_xd inner loop to:
  • Read FieldBlock state from each cursor (may be asymmetric at IB-boundary XDs: one port instance-backed with FieldBlock, the other IB-backed without).
  • On a full-rect consume on both cursors, loop over every field in the block and issue N memcpy_Xd calls at per-field offsets src_fields[f] * src_fstride / dst_fields[f] * dst_fstride, then advance each cursor with f = fields_left in one call.
  • On partial-rect consumes or asymmetric cases, copy one field and advance with f = 1. AddressListCursor::advance only promotes partial_fields on a full-rect consume, so the cursor state naturally resumes at the same field on the next iteration. DEBUG_REALM assertion enforces that f > 1 is only passed on full-rect consumes (the one invariant of the cursor API that the multi-field code can violate).

RemoteWriteChannel — cross-node fast path

src/realm/transfer/channel.{h,cc}

• Override support_idindexed_fields: returns true for any memory pair the channel already serves.
• Multi-field branch in RemoteWriteXferDes::progress_xd's 1D-dst / 1D-src path (the only path reachable under idindexed_fields; 2D dst, scatter dst, 2D src, and gather src are assert(0) or #ifdef-gated today). One Write1DMessage per field per rect slab — the AM framing cost is unchanged, but XferDes/iterator setup is amortized across the whole block instead of paid N times.

Remove -ll:dma_multi_field flag

src/realm/runtime_impl.{h,cc}, src/realm/transfer/transfer.cc

The flag is replaced by the cost-model gates described above. All three conditions are known locally at graph-construction time, so no runtime config is needed. MIN_IDINDEXED_FIELDS = 2 is annotated with the derivation so the threshold isn't mystery-magic.

Tests

tests/unit_tests/memcpy_multi_field_test.cc (new, registered in tests/CMakeLists.txt):

9 end-to-end cases driving a real MemcpyXferDes with IDIndexedFieldsIterator on both ports against malloc'd CPU buffers:

• 1D/2D/3D rects with 2, 4, and many fields.
• Field-ID reordering (src and dst field lists in different orders — e.g., src=[3,1,0,2], dst=[0,2,3,1]) to verify per-field pairing tracks src_fields[k] ↔ dst_fields[k].
• 1024-field scale to exercise the realistic Legion workload case and confirm the 256-KiB per-field budget keeps each progress_xd iteration bounded.
• Single-field (fields.size() == 1) sanity case — still attaches a FieldBlock but n collapses to 1 throughout, confirming the fast-path code does the right thing when there's nothing to batch.

tests/multifield_transfer.cc — generalized to CPU or GPU:

• New required -memkind cpu|sysmem|gpu|fbmem argument; intent is stated explicitly rather than inferred from what's available.
• Dropped implicit GPU_FB_MEM assumption (bench_timing_task) and CUDA::cudart link dependency (the test has no CUDA code).
• CMake registers two CTest entries from the same binary: multifield_transfer (CPU, always built) and multifield_transfer_gpu (opt-in when CUDA is enabled).
• CTest args default to -max_ops 1 -verify 1 for deterministic verification; raise max_ops with -verify 0 for throughput stress testing (concurrent copies on the same dst instance race by design).

Test plan

• ctest -L "unit|integration" → 723/723 pass (was 673 before; +9 multi-field unit cases, +1 CPU-mode multifield_transfer).
• New MemcpyMultiField.* unit cases pass (1D/2D/3D, reorder, permutation, 1024 fields, single-field).
• multifield_transfer passes in CPU mode with field counts from 1 to 512 at -max_ops 1 -verify 1.
• Toggling -ll:dma_multi_field on the old main tree vs. removing it on this branch: same 722/722 pass rate; no regressions at -fields 1 or -aos 0 (legacy paths).
• Multi-node correctness under MPI/UCX/GASNet-EX, 2/4/8 ranks
• Baseline-vs-new throughput comparison on target cluster at N ∈ {1, 4, 16, 64, 256, 1024, 4096} and per-field sizes sweeping the bandwidth-bound crossover.

How to run many-field performance tests

Single-node CPU-to-CPU, field sweep

for F in 1 4 16 64 256 1024 4096; do
./tests/memspeed -copies 1 -tasks 0 -fields $F -aos 1 -b 4 -reps 20
-ll:cpu 4 -ll:csize 16384
done

Multi-node cross-node pairs (RemoteWriteChannel exercised automatically)

mpirun -np 2 ./tests/memspeed -copies 1 -tasks 0 -fields 4096 -aos 1 -b 4
-reps 20 -ll:cpu 4 -ll:csize 16384

Integration test at scale, CPU mode

./tests/multifield_transfer -memkind cpu -num_fields 1024 -size 256
-max_ops 1 -ll:cpu 4 -ll:csize 8192 -verify 1

Out of scope

• GASNetChannel — follow-up; would need the same support_idindexed_fields override and per-field-offset branch in GASNetXferDes::progress_xd.
• AddressSplitChannel, file/disk channels, MemfillChannel, MemreduceChannel — not in the many-field hot path.
• Serdez fields — layout detection already rejects them (requires uniform field size), so the fast path is never selected there.
• Packing multiple fields into a single Write1DMessage AM on RemoteWriteChannel — would require a new AM schema (or memory-contiguity detection when fields form a monotone run). Current change amortizes XferDes/iterator setup but still sends one AM per field per rect slab.

[codecov]

Codecov Report

❌ Patch coverage is 79.42387% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.50%. Comparing base (d393ba4) to head (427a57c).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/realm/transfer/channel.cc	0.00%	42 Missing ⚠️
src/realm/transfer/memcpy_channel.cc	92.85%	2 Missing and 3 partials ⚠️
tests/unit_tests/memcpy_multi_field_test.cc	97.70%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #421      +/-   ##
==========================================
+ Coverage   29.20%   29.50%   +0.29%     
==========================================
  Files         195      196       +1     
  Lines       40493    40705     +212     
  Branches    14614    14738     +124     
==========================================
+ Hits        11825    12008     +183     
- Misses      27715    28270     +555     
+ Partials      953      427     -526     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.