Skip to content

OFI transport: inject_size capped at 16 bytes due to fi_getinfo hint #1230

Description

@bcmIntc

Description

shmem_transport_ofi_max_buffered_send was initialized to sizeof(long double)
(16 bytes) before being passed as tx_attr.inject_size in the fi_getinfo
hints. The CXI provider treats the hint value as a ceiling rather than a floor,
returning exactly 16 — even though CXI's natural inject_size is 192 bytes.

As a result, any put in the 17-192 byte range was sent via RDMA (with MR
registration lookup) rather than the faster inline inject path.

Root cause

src/transport_ofi.c, function query_for_fabric:

shmem_transport_ofi_max_buffered_send = sizeof(long double);  // hint = 16
...
tx_attr.inject_size = shmem_transport_ofi_max_buffered_send;
...
ret = fi_getinfo(..., &hints, &info);
...
shmem_transport_ofi_max_buffered_send = info->tx_attr->inject_size;  // adopts 16, not 192

Fix

Initialize shmem_transport_ofi_max_buffered_send = 0 (no minimum requirement)
so the provider returns its natural inject_size, which is then adopted. This works for all providers except VERBS,
which will provide a conservative injection size.

See: #1231

Impact

Puts in the 17-192 byte range now use the inline inject path on CXI. Operations
outside that range (AMOs ≤ 16 bytes, large puts > 192 bytes) are unaffected.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions