Summary
UCP currently treats UCT atomic offload support as an all-or-nothing capability for each enabled atomic width. As a result, a transport that supports a useful subset of atomic operations cannot offload even the operations it does support.
For an enabled width, UCP requires:
op_flags: ADD, AND, OR, XORfop_flags: ADD, AND, OR, XOR, SWAP, CSWAP
If any one of these bits is missing, the interface is excluded from device atomic selection and is not advertised as atomic-capable in the worker address.
Current behavior
The relevant checks are in:
ucp_context_uct_atomic_iface_flags()ucp_worker_init_device_atomics()ucp_address_pack_iface_attr()ucp_wireup_add_amo_lanes()
ucp_worker_init_device_atomics() requires the full masks before adding a resource to worker->atomic_tls. In non-unified address exchange, ucp_address_pack_iface_attr() sets UCP_ADDR_IFACE_FLAG_ATOMIC32/64 only when the complete corresponding masks are present.
Consequently, if a UCT supports, for example, only 64-bit fetch-add, UCP does not select it for hardware AMO offload even for a ucp_atomic_fetch_add operation. In a normal host-memory configuration with a usable AM bcopy lane, UCP instead falls back to the software AMO protocol.
Why this is limiting
Several transports or devices expose only a subset of AMO operations efficiently or in hardware. The current full-mask requirement prevents UCP from using those hardware operations at all, even when the requested operation is supported.
Requested behavior
Please consider representing and selecting AMO capabilities per operation and per width, so UCP can offload a supported operation while using an alternative protocol only for unsupported operations.
I understand that mixing device and software atomics may violate ordering or atomicity guarantees unless both paths share a consistency domain. If selective offload is not semantically safe, please document that the full-mask requirement is intentional and explain the required consistency constraint.
A possible implementation direction would be to exchange the actual op_flags and fop_flags masks in the worker address and make protocol selection operation-aware, subject to a safe ordering rule.
Summary
UCP currently treats UCT atomic offload support as an all-or-nothing capability for each enabled atomic width. As a result, a transport that supports a useful subset of atomic operations cannot offload even the operations it does support.
For an enabled width, UCP requires:
op_flags: ADD, AND, OR, XORfop_flags: ADD, AND, OR, XOR, SWAP, CSWAPIf any one of these bits is missing, the interface is excluded from device atomic selection and is not advertised as atomic-capable in the worker address.
Current behavior
The relevant checks are in:
ucp_context_uct_atomic_iface_flags()ucp_worker_init_device_atomics()ucp_address_pack_iface_attr()ucp_wireup_add_amo_lanes()ucp_worker_init_device_atomics()requires the full masks before adding a resource toworker->atomic_tls. In non-unified address exchange,ucp_address_pack_iface_attr()setsUCP_ADDR_IFACE_FLAG_ATOMIC32/64only when the complete corresponding masks are present.Consequently, if a UCT supports, for example, only 64-bit fetch-add, UCP does not select it for hardware AMO offload even for a
ucp_atomic_fetch_addoperation. In a normal host-memory configuration with a usable AM bcopy lane, UCP instead falls back to the software AMO protocol.Why this is limiting
Several transports or devices expose only a subset of AMO operations efficiently or in hardware. The current full-mask requirement prevents UCP from using those hardware operations at all, even when the requested operation is supported.
Requested behavior
Please consider representing and selecting AMO capabilities per operation and per width, so UCP can offload a supported operation while using an alternative protocol only for unsupported operations.
I understand that mixing device and software atomics may violate ordering or atomicity guarantees unless both paths share a consistency domain. If selective offload is not semantically safe, please document that the full-mask requirement is intentional and explain the required consistency constraint.
A possible implementation direction would be to exchange the actual
op_flagsandfop_flagsmasks in the worker address and make protocol selection operation-aware, subject to a safe ordering rule.