Add huge page support for symmetric heap to improve CXI ATU performance - #1234
Draft
bcmIntc wants to merge 7 commits into
Draft
Add huge page support for symmetric heap to improve CXI ATU performance#1234bcmIntc wants to merge 7 commits into
bcmIntc wants to merge 7 commits into
Conversation
When SYMMETRIC_HEAP_USE_HUGE_PAGES is enabled but no hugetlbfs mount is configured, use anonymous MAP_HUGETLB with explicit 2MB page size instead of falling back to transparent huge pages (THP). This allows the kernel's nr_overcommit_hugepages mechanism to dynamically allocate surplus huge pages on demand without requiring pre-reserved HugePages_Total. Anonymous MAP_HUGETLB with (21 << MAP_HUGE_SHIFT) explicitly requests 2MB pages, matching the CXI provider's behavior. When used with scalable MR mode (--enable-ofi-mr=scalable), this enables the CXI NIC's ATU to use 2MB page translations (derivative1) instead of 4KB base pages, significantly improving ATU cache hit rates. Fallback to THP via madvise(MADV_HUGEPAGE) still occurs if MAP_HUGETLB fails, ensuring compatibility across different kernel configurations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Change the hugetlbfs file open failure from RAISE_WARN_STR to DEBUG_MSG since the fallback to anonymous MAP_HUGETLB works correctly. The warning was misleading because huge pages were still being allocated successfully via the anonymous MAP_HUGETLB path. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
bcmIntc
changed the title
bcmIntc
changed the title
markbrown314
reviewed
Jun 9, 2026
| /* Try anonymous MAP_HUGETLB first (works with nr_overcommit_hugepages). | ||
| * Explicitly request 2MB pages via MAP_HUGE_SHIFT (21 << MAP_HUGE_SHIFT = 2^21 = 2MB). */ | ||
| ret = mmap(requested_base, bytes, PROT_READ | PROT_WRITE, | ||
| MAP_ANON | MAP_PRIVATE | MAP_HUGETLB | (21 << MAP_HUGE_SHIFT), -1, 0); |
Collaborator
There was a problem hiding this comment.
You need to check the environmental variable SHMEM_SYMMETRIC_HEAP_PAGE_SIZE to see if it is set to 2MB before setting the 2MB page flag.
The default for SHMEM_SYMMETRIC_HEAP_PAGE_SIZE is set to 2MB so the additional check will not change the behavior.
…ndling - Fix double free: set directory/file_name to NULL after freeing - Fix size bug: preserve original bytes, only use hugetlbfs_bytes for file path - Fix fallback: use NULL address hint after MAP_HUGETLB failure - Add debug visibility: log which allocation path succeeded - Change hugetlbfs warnings to debug messages (fallback works correctly) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…in fallbacks - Fix Issue 1 (munmap size mismatch): add size_t *mapped_bytes out-parameter to mmap_alloc(). On the hugetlbfs success path the mapping is rounded up to a huge-page boundary (hugetlbfs_bytes > bytes); the previous code passed the original unrounded size to munmap and transport registration (OFI, Portals4, UCX, XPMEM), leaking the tail pages from the huge-page pool. mmap_alloc now reports the actual mapped size, and shmem_internal_symmetric_init updates shmem_internal_heap_length accordingly so munmap, registration, and bounds checks all use the correct extent. - Fix Issue 2 (requested_base dropped on fallback): both THP fallback paths (ftruncate failure and MAP_HUGETLB failure) previously used mmap(NULL, ...) unconditionally. This discards the requested_base hint (data segment + 2 GB, 1 GB-aligned) that is required for --enable-remote-virtual-addressing to maintain symmetric virtual addresses across PEs. The fallbacks now first attempt mmap(requested_base, ...) and only resort to mmap(NULL, ...) if that also fails. Remove the incorrect comment claiming requested_base will not work after MAP_HUGETLB failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use put_quiet instead of unreliable completion watermark. The watermark approach fails because put_nb uses different code paths (inject, bounce buffer, put_large) that don't all update the completion counter reliably. put_quiet guarantees all in-flight puts complete regardless of path taken. Co-authored-by: GitHub Copilot <copilot@github.com>
Update comment to better explain why put_quiet is necessary. The inject path has no counter event, so put_wait with completion=0 would return immediately and leave GPU writes unordered, causing a data race. put_quiet provides the NIC-level ordering fence needed for correctness. Co-authored-by: GitHub Copilot <copilot@github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for 2MB huge pages in the symmetric heap allocation, enabling the CXI NIC's Address Translation Unit (ATU) to use more efficient 2MB page translations instead of 4KB base pages. This significantly improves ATU cache hit rates on systems with CXI interconnects.
Changes
Symmetric heap: Anonymous MAP_HUGETLB allocation
When
SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES=1is set, the symmetric heap now uses anonymousMAP_HUGETLBwith explicit 2MB page size:MAP_HUGETLB | (21 << MAP_HUGE_SHIFT)to request 2MB pages explicitlynr_overcommit_hugepagesmechanism for on-demand surplus huge page allocationHugePages_Total) requiredmadvise(MADV_HUGEPAGE)Implementation details
mmap_alloc()insrc/symmetric_heap_c.crequested_baseas address hint to keep heap in expected memory regionPerformance Impact
Testing on Perlmutter (NERSC) with
--enable-ofi-mr=scalableconfiguration:Without huge pages, the ATU would predominantly use 4KB translations, resulting in higher cache miss rates and increased address translation overhead during RDMA operations.
Configuration
Build-time
Run-time
export SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGES=1System requirements
MAP_HUGE_SHIFTsupport)vm.nr_overcommit_hugepages> 0 (check withcat /proc/sys/vm/nr_overcommit_hugepages)sudo sysctl -w vm.nr_overcommit_hugepages=<N>Compatibility
SHMEM_SYMMETRIC_HEAP_USE_HUGE_PAGESis not set or 0#ifdef __linux__guards)Testing
Validated on:
Commits
b08a5e3f- symmetric heap: use anonymous MAP_HUGETLB for huge page allocationc3770a50- Formatting8f0d93d4- symmetric heap: change hugetlbfs file warning to debug message