Add 2MB Huge Page Support for Symmetric Heap + CXI Fence Optimization - #1232
Draft
bcmIntc wants to merge 43 commits into
Draft
Add 2MB Huge Page Support for Symmetric Heap + CXI Fence Optimization#1232bcmIntc wants to merge 43 commits into
bcmIntc wants to merge 43 commits into
Conversation
Adds --enable-hierarchical-barrier, a three-phase barrier that keeps intranode traffic off the NIC by using CPU atomics over XPMEM for gather/fanout and restricts NIC puts to the internode phase (node roots only). Phase 1 (intranode gather): local PEs signal up a k-ary tree. Each PE writes to its OWN up-slot in local_pSync; the parent reads each child's slot individually. Slots are padded to one cache line (HIER_SLOT_STRIDE=8 longs, 64 bytes) so no two PEs share a line, eliminating the MESI serialization that would occur if all children wrote to a single counter. Signal values increment monotonically via hier_sense, avoiding explicit slot resets between calls (sense alternation). Phase 2 (internode dissemination): node roots run a put-based binary dissemination across the NIC. After each round the slot is reset via a CPU store rather than a self-put, saving ceil(log2(N_nodes)) NIC round-trips per barrier (12 at 4096 nodes). Phase 3 (intranode fanout): node root CPU-stores an ack into each child's down-slot; children relay down the k-ary tree with reset-before-signal ordering. Down-slots are in the upper half of local_pSync, laid out with the same per-PE cache-line padding as up-slots. AUTO selection activates when local PE count >= SHMEM_HIER_BARRIER_THRESHOLD (default 2). Also selectable via SHMEM_BARRIER_ALGORITHM=hierarchical. New infrastructure: - src/shr_transport.h — XPMEM CPU pointer mapping; self-access returns the address directly without an XPMEM lookup - src/runtime_util.c — global hostname exchange so each PE can identify its node root - configure.ac — --enable-hierarchical-barrier requires --with-xpmem and a network transport
Add FI_MR_LOCAL to domain_attr.mr_mode and register symmetric heap with local read access (FI_READ) when --enable-mr-local is enabled. This allows the CXI provider to require local memory descriptors in RMA operations for optimal performance. Changes: - configure.ac: Add --enable-mr-local flag - transport_ofi.c: Add FI_MR_LOCAL to mr_mode; add FI_READ to all fi_mr_reg() calls (heap, data, external heap) - CLAUDE.md: Document new configure flag Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Let the provider advertise its natural inject_size rather than requiring it to be at least sizeof(long double). The returned inject_size is adopted immediately after fi_getinfo.
ftruncate() must be called on the hugetlbfs file descriptor before mmap() to set the file size. Without it, mmap() against a hugetlbfs fd can fail with EINVAL or produce unexpected behavior. Also add MAP_HUGETLB alongside MAP_SHARED to make the intent explicit and match the usage confirmed working on the target system. On ftruncate failure, unlink the file before falling back to regular anonymous pages to avoid leaving stale files in /dev/hugepages.
…oc warning Guard all three DISABLE_NONFETCH_AMO blocks in shmem_comm.h with USE_OFI to prevent compile errors (FI_ATOMIC_WRITE undeclared) and runtime errors (transport_none: No path to peer) in non-OFI builds. Also improve the mmap_alloc file open failure warning to include strerror(errno) so the cause (e.g. Permission denied) is visible.
When hugetlbfs is unavailable (no reserved huge pages or no permissions), fall back to transparent huge pages (THP) via madvise(MADV_HUGEPAGE). This allows huge page support without requiring root/sudo access or pre-configured hugetlbfs mounts. Fallback paths: 1. hugetlbfs with MAP_HUGETLB (preferred) 2. madvise(MADV_HUGEPAGE) for THP (new fallback) 3. Regular 4K pages (existing fallback) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Change huge page setup warnings to DEBUG_* macros so they only appear when SHMEM_DEBUG=1. These are informational messages about fallback paths and don't indicate errors - the code gracefully falls back to regular pages when huge pages aren't available. Only shown with: export SHMEM_DEBUG=1
…TE for puts and non-fetch AMOs FI_DELIVERY_COMPLETE requires the NIC to hold a credit slot open until the remote target confirms data has landed in memory. Under AMO+put contention this exhausted put credits (mst_stalled_waiting_put_crdts: 379M on the initiator vs 0 on Cray SHMEM running the same workload). OpenSHMEM only requires that puts be ordered/visible at the target after a shmem_quiet/shmem_fence; it does not require per-put remote delivery confirmation at issue time. FI_TRANSMIT_COMPLETE satisfies this: the put is reliable once it leaves the initiator NIC, and quiet/fence drain the counter. The put_signal terminal signal operation retains FI_DELIVERY_COMPLETE because it is an explicit ordering point that must guarantee the preceding payload has landed before the signal is observed by the target. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without this, fi_mr_reg is called while the heap page table is still sparse (anonymous pages are not faulted until first access). The CXI ATU therefore records 4KB base-page entries for the entire heap, causing an ATU lookup on every subsequent RMA/AMO operation. Cray SHMEM uses lazy/demand MR registration so pages are already promoted to 2MB by THP when they are first registered. The telemetry confirms: Cray shows 62.5M derivative1 (2MB) ATU hits and only 13K base-page hits for a full benchmark run, while SOS shows 62M base-page hits and zero derivative1 hits. Fix: after mmap_alloc returns, stride through the heap writing one byte per SYMMETRIC_HEAP_PAGE_SIZE (default 2MB) to trigger page faults. THP then promotes contiguous 4KB entries to 2MB entries before fi_mr_reg is called. The one-time cost is proportional to heap_size / 2MB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous scalable+RVA path registered the entire virtual address space as a single MR (fi_mr_reg(0, UINT64_MAX, ...)). The CXI kernel driver rejects this with EINVAL (-22) because it cannot map the full 64-bit VA range. Fix: remove the ENABLE_MR_SCALABLE+ENABLE_REMOTE_VIRTUAL_ADDRESSING special case and use the same two-segment registration (heap + data) that the non-RVA scalable path already uses. The shmem_transport_ofi_get_mr() function already returns absolute VAs (*mr_addr = addr) under RVA, so the NIC sees real virtual addresses for both segments without any code change to the hot path. The mr_desc_index function is updated to do a proper range check for both segments instead of unconditionally returning 0; the single shmem_transport_ofi_ target_mrfd variable is replaced with the existing heap_mrfd/data_mrfd pair. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At 104 PPN with a 512MB default heap, all PEs fault 512MB simultaneously at init, saturating node memory bandwidth and causing job timeouts before the benchmark produces any output. The ATU 4KB vs 2MB problem requires a different solution. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…t MRs" This reverts commit 5b7f569.
Loads libdsmml.so via dlopen at runtime when SHMEM_SYMMETRIC_HEAP_USE_DSMML=1
is set in the environment. No configure changes required — the library is
resolved at job launch time.
DSMML (HPE Distributed Symmetric Memory Management Library) is the allocator
used by Cray SHMEM for symmetric heap backing. It supports explicit 2MB huge
page allocation via DSMML_HPSIZE_2M, which causes the CXI ATU to cache 2MB
TLB entries rather than 4KB entries. This is the mechanism behind Cray
SHMEM's atu_cache_hit_derivative1_page_size_0 dominance vs SOS's base-page
hits.
Allocation sequence:
1. dlopen("libdsmml.so")
2. dsmml_init()
3. dsmml_create_sheap_seg() with DSMML_HPSIZE_2M
4. Fall back to DSMML_HPSIZE_DEFAULT (THP) if 2MB pool unavailable
Teardown: dsmml_finalize() + dlclose() in shmem_internal_symmetric_fini().
Usage:
SHMEM_SYMMETRIC_HEAP_USE_DSMML=1 srun ...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
MADV_COLLAPSE (Linux 5.18+, kernel 25):
Called immediately after MADV_HUGEPAGE to force synchronous THP promotion
before fi_mr_reg. Perlmutter compute nodes run kernel 6.4 which supports
this; it is gated by the MADV_HUGEPAGE success path so older kernels
silently skip it. Defined locally if not in system headers.
This avoids the khugepaged deferral that caused our pre-fault loop
approach to still show 4KB ATU hits.
SHMEM_SYMMETRIC_HEAP_USE_DSMML=1:
Loads libdsmml.so via dlopen at runtime (no configure/link changes).
DSMML is HPE's symmetric heap allocator used by Cray SHMEM; it supports
explicit 2MB hugetlbfs allocation (DSMML_HPSIZE_2M) with fallback to THP.
Types mirror dsmml.h exactly for ABI compatibility without a build-time
header dependency. Launch with:
LD_LIBRARY_PATH=/opt/cray/pe/dsmml/default/dsmml/lib:$LD_LIBRARY_PATH SHMEM_SYMMETRIC_HEAP_USE_DSMML=1 srun ...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cray SHMEM caps in-flight SHEAP puts at 512, which prevents the NIC's TRS (Transaction Resource) pool from exhausting under AMO+put contention. SOS had no such limit, causing mst_stalled_waiting_put_crdts to reach 333M+ on the initiator while Cray shows 0. Add SHMEM_OFI_PUT_PIPELINE_DEPTH (default 512, 0=unlimited) which throttles RDMA put issue rate by checking pending-completed against the limit before each fi_write/fi_writemsg call. The inject path (fi_inject_write) is exempt since inject does not consume TRS slots. Throttle points: put_large (fi_write), put_nb bounce-buffer path (fi_writemsg). The signal operation retains its existing fence-based ordering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The nail benchmark's dependent put pattern (1 put per AMO) never reaches the 512 limit, so the throttle added overhead (fi_cntr_read on every RDMA put) with no benefit. Default to 0 (disabled) until a workload that actually benefits from it is identified. Enable with SHMEM_OFI_PUT_PIPELINE_DEPTH=512 to match Cray SHMEM behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At high PPN (e.g. 128), all PEs simultaneously fire fetching AMOs to the same target PE. This saturates the target NIC's TRS pool and causes mst_stalled_waiting_put_crdts on all initiators despite only one put per AMO being in flight. Cray SHMEM addresses this with incast throttling — "every 0 atomics at 128 PPN" — pacing each PE's AMO issue rate so the target NIC is not overwhelmed. Add SHMEM_OFI_AMO_PIPELINE_DEPTH (default 0=disabled) which limits in-flight fetching AMOs per context. The throttle is in fetch_atomic_nbi (the CXI path under ENABLE_MR_ENDPOINT), the only fetching AMO code path active on Perlmutter. At 128 PPN with ~512 NIC TRS slots, a value of 4 leaves each PE a fair share. Enable with: SHMEM_OFI_AMO_PIPELINE_DEPTH=4 srun ... Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously, shmem_long_put for sizes > bounce_buffer_size routed through put_large which set *completion=1, causing shmem_internal_put_wait to call shmem_transport_put_quiet — a global drain of every in-flight put on the context. Under the Nail benchmark's AMO+put pattern, this serialized the entire pipeline at sizes >= 16KB: each put waited for all prior puts and implicitly any in-progress AMOs sharing the context. Replace the count-based completion semantic with a counter watermark. put_large now records pending_put_cntr immediately after issuing all fragments; put_wait spins fi_cntr_read(put_cntr) until it reaches that specific watermark. This satisfies the OpenSHMEM "source buffer reusable on return" contract for blocking puts without forcing global ordering. The inject and bounce-buffer paths are unaffected: inject completes synchronously inside fi_inject_write, and bounce buffers memcpy the source before issue, so neither path needs to set the watermark — *completion stays 0 and put_wait is a no-op for them, as before. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Cray SHMEM's startup log shows it calls FI_CXI_DOM_OPS_3 enable_hybrid_mr_desc right after fi_domain(); SOS does not. Without it, the CXI provider performs internal memory registration on every fi_write/fi_writemsg/fi_fetch_atomicmsg call where the desc field is non-NULL — even if the buffer is already in a provider-registered region. Each MR cache lookup adds latency to large RDMA puts. With hybrid mode enabled, the provider trusts a non-NULL desc and skips its internal registration walk, dropping per-call overhead on the put data path. The call must occur before any endpoints are created — the docstring warns that endpoints inherit the domain's hybrid-MR status only at creation time. Implementation does not include rdma/fi_cxi_ext.h; instead, it mirrors the needed fields of struct fi_cxi_dom_ops locally so the build does not require CXI headers. fi_open_ops() returns -FI_ENOSYS on non-CXI providers; that case is silently ignored. Gated by SHMEM_OFI_CXI_HYBRID_MR_DESC=true (default), so it can be disabled for A/B testing without rebuilding. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Promote the hybrid MR desc enable result from DEBUG_MSG (silent unless SHMEM_DEBUG=1) to fprintf(stderr) on PE 0 only, so we can verify whether the call actually took effect on each run. Also reports when the env var disables it, when the provider doesn't support dom_ops_v3 (non-CXI), and when the call returned an error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Issue Sandia-OpenSHMEM#1221 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
Replaced complex polling with simpler OFI counter wait Issue Sandia-OpenSHMEM#1217 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
This reverts commit b79c049.
This reverts commit 7654e4b.
Replaced complex polling with simpler OFI counter wait Issue Sandia-OpenSHMEM#1217 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
Issue Sandia-OpenSHMEM#1221 Signed-off-by: Mark F. Brown <mark.f.brown@intel.com>
This reverts commit 32fb6d6.
This reverts commit 6a369b2.
Bug: Static global hier_sense caused signal mismatch when PEs participated in different barrier teams. Only PEs in the active set incremented hier_sense, causing divergence on the next TEAM_WORLD barrier (PEs that skipped the subset barrier had stale sense values). Fix: Move hier_sense to shmem_internal_team_t. For TEAM_WORLD barriers (PE_start=0, PE_stride=1, PE_size=num_pes), use team-local state. For subset barriers (rare), use static fallback to avoid full team-parameter refactor. All PEs in TEAM_WORLD now stay synchronized across interleaved team/subset barrier sequences. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
CXI maintains per-EP FIFO ordering, so FI_TRANSMIT_COMPLETE already guarantees subsequent operations see prior puts at the target. Polling pending_put_cntr in shmem_transport_put_quiet() is redundant and adds ~1-2µs latency. Skip put_quiet when prov_name == "cxi". Other providers (tcp, verbs, opx, sockets) require explicit put_quiet to ensure remote visibility before fence returns, so preserve existing behavior for non-CXI. This is safe because: 1. put_signal_nbi already uses FI_DELIVERY_COMPLETE for the signal write 2. Explicit fence (put_quiet or FI_FENCE flag) orders signal after data 3. CXI FIFO ordering means TRANSMIT_COMPLETE is sufficient for correctness Expected benefit: ~1-2µs improvement in put_signal_nbi and barrier operations that call shmem_transport_fence(). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
When SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES is enabled, try anonymous MAP_HUGETLB before falling back to THP via madvise. This works with nr_overcommit_hugepages without requiring mounted hugetlbfs, matching the allocation strategy used by Cray SHMEM on systems like Perlmutter. Allocation priority when huge pages are enabled: 1. File-backed hugetlbfs (if mounted and accessible) 2. Anonymous MAP_HUGETLB (new, works with overcommit) 3. THP via madvise (fallback) 4. Regular 4K pages (if all else fails) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add (21 << MAP_HUGE_SHIFT) to the MAP_HUGETLB mmap call to explicitly request 2MB pages (2^21 = 2MB). This matches the allocation strategy used by the CXI libfabric provider for CQ/EQ buffers and ensures the kernel allocates the correct huge page size. When combined with scalable MR mode (--enable-ofi-mr=scalable), this allows the CXI NIC's ATU to detect and use 2MB page granularity for address translation, reducing ATU cache pressure. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Let the kernel choose the address for MAP_HUGETLB allocation instead of forcing requested_base. The kernel may not be able to allocate huge pages at the specific requested address, causing MAP_HUGETLB to fail even when nr_overcommit_hugepages is configured. Using NULL as the address hint allows the kernel to select a suitable address for huge page allocation, improving success rate. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Add detailed errno logging for MAP_HUGETLB and madvise failures to diagnose why huge page allocation is failing on compute nodes. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Remove spurious madvise(MADV_HUGEPAGE) call that was incorrectly running after successful MAP_HUGETLB allocation. When MAP_HUGETLB succeeds, the memory is already backed by huge pages and madvise is unnecessary. The duplicate madvise was failing with EINVAL because the memory region was already using explicit huge pages rather than transparent huge pages. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Use requested_base as address hint for MAP_HUGETLB instead of NULL. This places the symmetric heap in the expected memory region (near data segment) rather than letting the kernel choose an arbitrary high address like 0x7fffd7000000. The CXI driver may have constraints on which address ranges can be registered for RDMA, and keeping the heap in the expected region improves compatibility. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
markbrown314
reviewed
Jun 9, 2026
| shmem_transport_ofi_stx_max, | ||
| num_nics); | ||
|
|
||
| /* Set CXI provider flag for fence optimization */ |
Collaborator
There was a problem hiding this comment.
This is useful, but we should generalize this into a provider check function:
bool shmem_transport_check_provider(const char *name)
If name matches, provider it return true
…k_provider() Replace hardcoded provider name comparisons with a centralized helper function. This provides: - Single source of truth for provider name storage - Consistent comparison logic across all provider checks - Easy addition of new provider-specific optimizations - Better code maintainability Changed: - Replaced global `shmem_transport_ofi_is_cxi` with static `shmem_transport_ofi_prov_name` pointer - Added `shmem_transport_ofi_check_provider(const char *name)` helper - Updated fence CXI check to use new function - Set provider name once in query_for_fabric() No functional change - same CXI fence optimization behavior. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- collectives: use malloc instead of alloca for PE arrays at scale (prevents stack overflow when PE_size reaches 100K+) - runtime-pmi/pmi2: add NULL check for location_array (prevents segfault) - shmem_comm: use put_quiet instead of unreliable completion watermark in copy_self (guarantees visibility across all put paths: inject, bounce, large) Co-authored-by: GitHub Copilot <copilot@github.com>
Revert to put_quiet approach after analysis showed put_wait is insufficient. The inject path (small copies) has no counter event, so put_wait with completion=0 would return immediately and leave GPU writes unordered, causing a data race. put_quiet drains all pending puts and provides the NIC-level ordering fence needed to guarantee dest is visible at the target GPU before returning. More conservative than put_wait but correct for all paths. Co-authored-by: GitHub Copilot <copilot@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR includes three changes targeting CXI/Slingshot performance on systems like Perlmutter.
Changes
The CXI provider doesn't require explicit quiet operations before fences since fences are already ordering operations. Skipping the redundant put_quiet() reduces fence overhead.
Enables allocation of the symmetric heap using 2MB huge pages via anonymous MAP_HUGETLB, which works with kernel nr_overcommit_hugepages without requiring mounted hugetlbfs. Falls back to transparent huge pages (THP) via madvise() if explicit huge pages are unavailable.
Usage:
export SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES=1
Adds (21 << MAP_HUGE_SHIFT) to the MAP_HUGETLB mmap call to explicitly request 2MB pages (2^21 = 2MB). This matches the allocation strategy used by the CXI libfabric provider for its internal buffers.
Performance Impact
When combined with scalable MR mode (--enable-ofi-mr=scalable), the huge page changes enable the CXI NIC's ATU (Address Translation Unit) to use 2MB page granularity instead of 4KB, dramatically reducing translation overhead:
Additional benefits:
Configuration Requirements
The ATU benefits require scalable MR mode (recommended for CXI/Slingshot):
./configure --enable-ofi-mr=scalable --enable-mr-endpoint ...
With scalable MR, fi_mr_reg() registers the entire address space, allowing the CXI provider to detect 2MB backing pages and configure the ATU accordingly. Also eliminates ASLR sensitivity (no setarch --addr-no-randomize needed).
Testing
Verified on NERSC Perlmutter with 2-node and 256-PE jobs using CXI telemetry counters to confirm ATU derivative (2MB) page activity.