Description
shmem_transport_ofi_max_buffered_send was initialized to sizeof(long double)
(16 bytes) before being passed as tx_attr.inject_size in the fi_getinfo
hints. The CXI provider treats the hint value as a ceiling rather than a floor,
returning exactly 16 — even though CXI's natural inject_size is 192 bytes.
As a result, any put in the 17-192 byte range was sent via RDMA (with MR
registration lookup) rather than the faster inline inject path.
Root cause
src/transport_ofi.c, function query_for_fabric:
shmem_transport_ofi_max_buffered_send = sizeof(long double); // hint = 16
...
tx_attr.inject_size = shmem_transport_ofi_max_buffered_send;
...
ret = fi_getinfo(..., &hints, &info);
...
shmem_transport_ofi_max_buffered_send = info->tx_attr->inject_size; // adopts 16, not 192
Fix
Initialize shmem_transport_ofi_max_buffered_send = 0 (no minimum requirement)
so the provider returns its natural inject_size, which is then adopted. This works for all providers except VERBS,
which will provide a conservative injection size.
See: #1231
Impact
Puts in the 17-192 byte range now use the inline inject path on CXI. Operations
outside that range (AMOs ≤ 16 bytes, large puts > 192 bytes) are unaffected.
Description
shmem_transport_ofi_max_buffered_sendwas initialized tosizeof(long double)(16 bytes) before being passed as
tx_attr.inject_sizein thefi_getinfohints. The CXI provider treats the hint value as a ceiling rather than a floor,
returning exactly 16 — even though CXI's natural inject_size is 192 bytes.
As a result, any put in the 17-192 byte range was sent via RDMA (with MR
registration lookup) rather than the faster inline inject path.
Root cause
src/transport_ofi.c, functionquery_for_fabric:Fix
Initialize shmem_transport_ofi_max_buffered_send = 0 (no minimum requirement)
so the provider returns its natural inject_size, which is then adopted. This works for all providers except VERBS,
which will provide a conservative injection size.
See: #1231
Impact
Puts in the 17-192 byte range now use the inline inject path on CXI. Operations
outside that range (AMOs ≤ 16 bytes, large puts > 192 bytes) are unaffected.