Describe the bug
On machines with LACP-bonded ConnectX-7 NICs (mlx5_bond_0), registering memory through the GGA transport crashes with a segfault.
The root cause is in uct_ib_mlx5_devx_umr_modify_qp() (src/uct/ib/mlx5/dv/ib_mlx5dv_md.c): the INIT→RTR transition uses ibv_modify_qp(), which cannot set the lag_tx_port_affinity field required by firmware on LAG devices. The transition fails with ENODEV.
The segfault is a secondary consequence: after the failure, md->umr.qp is destroyed but not NULLed, and md->umr.mkey_pool is never initialized (list init only runs on the success path). On the next mkey_pack call, the code skips re-initialization (dangling pointer passes the NULL check), then ucs_list_extract_head dereferences mkey_pool.next which is zero → NULL pointer dereference.
The same LAG port affinity handling already exists for RC/DC QPs via uct_ib_mlx5_devx_set_qpc_port_affinity(), but the UMR QP path was missed.
Steps to Reproduce
Hardware: ConnectX-7 with 2+ ports in LACP bond (802.3ad mode 4), resulting in mlx5_bond_0.
import torch
from nixl._api import nixl_agent, nixl_agent_config
config = nixl_agent_config(True, True, 0, backends=['UCX'])
agent = nixl_agent('test', config)
torch.cuda.init()
gpu_tensor = torch.zeros(1024 * 1024, dtype=torch.uint8, device='cuda:0')
desc = agent.register_memory(gpu_tensor) # segfault hereOutput:
Segmentation fault (core dumped)
dmesg:
python[3560119]: segfault at 0 ip 00007f0a6e4f9297 sp 00007ffc3710ac20 error 4 in libuct_ib_mlx5.so.0.0.0[7f0a6e4ef000+49000]
- UCX version: master (HEAD), configured with
--with-mlx5dv --with-devx
- NIXL version: v0.2.1
- No special UCX environment variables (default
UCX_TLS)
- Workaround:
UCX_TLS=^gga_mlx5
Setup and versions
- OS:Linux , x86_64
- Kernel:
5.10.134-16.3.al8.x86_64
- RDMA:
- HCA: ConnectX-7 (
MT41692), firmware 32.39.8002
- 4x CX-7 per node, each with 2 ports in LACP bond
- Link layer: Ethernet, 200 Gb/sec (2X NDR), MTU 9000
$ rdma dev show
8: mlx5_bond_0: node_type ca fw 32.39.8002 node_guid c470:bd03:0057:5226 sys_image_guid c470:bd03:0057:5226
$ cat /proc/net/bonding/bond0
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
Slave Interface: reth0 (200000 Mbps)
Slave Interface: reth1 (200000 Mbps)
Additional information
Proposed fix: replace ibv_modify_qp() with DevX commands for the INIT→RTR and RTR→RTS transitions of the UMR QP, allowing lag_tx_port_affinity to be set via the existing uct_ib_mlx5_devx_set_qpc_port_affinity() helper — the same pattern used for RC/DC QP connections. RST→INIT can remain via ibv_modify_qp() since LAG affinity is not required at that stage.
Describe the bug
On machines with LACP-bonded ConnectX-7 NICs (
mlx5_bond_0), registering memory through the GGA transport crashes with a segfault.The root cause is in
uct_ib_mlx5_devx_umr_modify_qp()(src/uct/ib/mlx5/dv/ib_mlx5dv_md.c): the INIT→RTR transition usesibv_modify_qp(), which cannot set thelag_tx_port_affinityfield required by firmware on LAG devices. The transition fails withENODEV.The segfault is a secondary consequence: after the failure,
md->umr.qpis destroyed but not NULLed, andmd->umr.mkey_poolis never initialized (list init only runs on the success path). On the nextmkey_packcall, the code skips re-initialization (dangling pointer passes theNULLcheck), thenucs_list_extract_headdereferencesmkey_pool.nextwhich is zero → NULL pointer dereference.The same LAG port affinity handling already exists for RC/DC QPs via
uct_ib_mlx5_devx_set_qpc_port_affinity(), but the UMR QP path was missed.Steps to Reproduce
Hardware: ConnectX-7 with 2+ ports in LACP bond (802.3ad mode 4), resulting in
mlx5_bond_0.Output:
dmesg:
--with-mlx5dv --with-devxUCX_TLS)UCX_TLS=^gga_mlx5Setup and versions
5.10.134-16.3.al8.x86_64MT41692), firmware32.39.8002Additional information
Proposed fix: replace
ibv_modify_qp()with DevX commands for the INIT→RTR and RTR→RTS transitions of the UMR QP, allowinglag_tx_port_affinityto be set via the existinguct_ib_mlx5_devx_set_qpc_port_affinity()helper — the same pattern used for RC/DC QP connections. RST→INIT can remain viaibv_modify_qp()since LAG affinity is not required at that stage.