UCP/RNDV: Skip inter-node rkey_ptr without exportable memory - #11754
Open
tvegas1 wants to merge 2 commits into
Open
UCP/RNDV: Skip inter-node rkey_ptr without exportable memory#11754tvegas1 wants to merge 2 commits into
tvegas1 wants to merge 2 commits into
Conversation
Detect UCS_MEM_FLAG_RKEY_PTR_INTER_NODE for fabric-capable CUDA buffers and exclude rkey_ptr MDs from rendezvous key packing on inter-node endpoints when the flag is absent, so legacy CUDA over MNNVL does not advertise unopenable cuda_ipc keys.
tvegas1
commented
Aug 11, 2026
tvegas1
commented
Aug 11, 2026
Contributor
Author
|
relates to #11712 |
The test skipped because both peers share a process, so cuda_ipc is unreachable and no lane is created. Move it to the mock suite, which fakes the peer PID, and inspect the rndv/rtr md_map to avoid depending on put/rndv being probed. Restore the CUDA registrable assertions in the memory query tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Introduce
UCS_MEM_FLAG_RKEY_PTR_INTER_NODEfor fabric-capable CUDA memory, and skip inter-noderkey_ptrMDs (e.g.cuda_ipc) inucp_proto_rndv_ctrl_get_md_map()when that flag is absent. Legacy CUDA over NVLink on the same node remains supported.Why?
On MNNVL, when receiving an RTS, proto selection can assume the local target is
cuda_ipc-accessible and pickrndv/rtr. The sender then has to use that path and often ends up on an emulated/AM fallback after the IPC open fails.If we detect that the CUDA buffer is not fabric-exportable, we skip
rndv/rtrand fall back torndv/rtr/mtype: allocate registrable frags and let the sender do HCA zcopy into them.How?
cuda_copymem query (same idea asuct_cuda_ipc_mem_add_reg) and setUCS_MEM_FLAG_RKEY_PTR_INTER_NODE.UCT_MD_FLAG_RKEY_PTR/UCT_COMPONENT_FLAG_RKEY_PTRunless the flag is set (cuda_ipcreports only the MD flag today).required_mem_flags, and do not infer it inucp_proto_rndv_rkey_mem_flags_estimate().Repro: