Skip to content

UCP/RNDV: Skip inter-node rkey_ptr without exportable memory - #11754

Open
tvegas1 wants to merge 2 commits into
openucx:masterfrom
tvegas1:rma_rndv_mnnvl
Open

UCP/RNDV: Skip inter-node rkey_ptr without exportable memory#11754
tvegas1 wants to merge 2 commits into
openucx:masterfrom
tvegas1:rma_rndv_mnnvl

Conversation

@tvegas1

@tvegas1 tvegas1 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What?

Introduce UCS_MEM_FLAG_RKEY_PTR_INTER_NODE for fabric-capable CUDA memory, and skip inter-node rkey_ptr MDs (e.g. cuda_ipc) in ucp_proto_rndv_ctrl_get_md_map() when that flag is absent. Legacy CUDA over NVLink on the same node remains supported.

Why?

On MNNVL, when receiving an RTS, proto selection can assume the local target is cuda_ipc-accessible and pick rndv/rtr. The sender then has to use that path and often ends up on an emulated/AM fallback after the IPC open fails.
If we detect that the CUDA buffer is not fabric-exportable, we skip rndv/rtr and fall back to rndv/rtr/mtype: allocate registrable frags and let the sender do HCA zcopy into them.

How?

  1. Detect fabric-exportable CUDA in cuda_copy mem query (same idea as uct_cuda_ipc_mem_add_reg) and set UCS_MEM_FLAG_RKEY_PTR_INTER_NODE.
  2. On inter-node endpoints, skip MDs with UCT_MD_FLAG_RKEY_PTR / UCT_COMPONENT_FLAG_RKEY_PTR unless the flag is set (cuda_ipc reports only the MD flag today).
  3. Do not use this as MD required_mem_flags, and do not infer it in ucp_proto_rndv_rkey_mem_flags_estimate().

Repro:

UCX_RNDV_THRESH=auto UCX_RNDV_FRAG_SIZE=host:4M,cuda:4M \
	UCX_PROTO_INFO=used \
	UCX_RNDV_PIPELINE_ERROR_HANDLING=y UCX_MAX_RMA_RAILS=4 \
	UCX_MAX_RNDV_RAILS=4 UCX_IB_NUM_PATHS=2 \
	UCX_NET_DEVICES=mlx5_0,mlx5_1 UCX_RNDV_FRAG_MEM_TYPES=cuda \
nixlbench \
      --enable_vmm=$1 \
      --etcd-endpoints http://$HOST:2379 \
      --pipeline_depth 1 \
      --backend UCX \
      --initiator_seg_type VRAM \
      --target_seg_type VRAM \
      --op_type WRITE \
      --check-consistency=1 \
      --start_block_size=$SIZE \
      --num_iter=1001 \
      --warmup_iter=100 \
      --max_block_size=$((128*1024*1024))

Detect UCS_MEM_FLAG_RKEY_PTR_INTER_NODE for fabric-capable CUDA buffers
and exclude rkey_ptr MDs from rendezvous key packing on inter-node
endpoints when the flag is absent, so legacy CUDA over MNNVL does not
advertise unopenable cuda_ipc keys.
Comment thread test/gtest/ucp/test_ucp_proto.cc Outdated
Comment thread test/gtest/uct/cuda/test_switch_cuda_device.cc
@tvegas1

tvegas1 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

relates to #11712

The test skipped because both peers share a process, so cuda_ipc is
unreachable and no lane is created. Move it to the mock suite, which
fakes the peer PID, and inspect the rndv/rtr md_map to avoid depending
on put/rndv being probed. Restore the CUDA registrable assertions in the
memory query tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant