UCT/IB/MLX5: Non-DevX rc_mlx5 CQ notification misses after Direct Verbs polling

[Zacky-ran]

UCT/IB/MLX5: Non-DevX rc_mlx5 CQ notification misses after Direct Verbs polling

Describe the bug

In the non-DevX rc_mlx5 transport, the CQ is polled through the mlx5 Direct Verbs path but armed through the generic verbs notification path.

The Direct Verbs polling path advances the consumer index maintained by UCX (uct_ib_mlx5_cq_t::cq_ci). However, the non-DevX arm path calls ibv_req_notify_cq(), which uses the consumer index maintained internally by the provider/libibverbs path.

These two consumer-index states are not synchronized. As a result, a CQ event notification can be missed even though a CQE is already available.

This report is specific to the non-DevX rc_mlx5 path. The DevX CQ event path is not affected.

Steps to reproduce

Environment:

• UCX: upstream master, commit e5fbdbb97
• OS: Rocky Linux 9
• Architecture: x86_64
• HCA: Mellanox ConnectX-4 Lx VF (mlx5_0)
• Port: mlx5_0:1
• Link layer: Ethernet/RoCE
• Firmware: 14.27.4062
• UCX_IB_MLX5_DV=y
• UCX_IB_MLX5_DEVX=n
• UCX_TLS=rc_mlx5,ud_mlx5
• UCX_NET_DEVICES=mlx5_0:1

Build:

./autogen.sh
CPPFLAGS=-D_GNU_SOURCE ./contrib/configure-devel --enable-debug --with-verbs --with-mlx5 --with-rdmacm --prefix=$PWD/install
make -j8
make -C test/gtest -j8
make -C src/tools/perf -j8

Run:

export UCX_TLS=rc_mlx5,ud_mlx5
export UCX_NET_DEVICES=mlx5_0:1
export UCX_IB_MLX5_DV=y
export UCX_IB_MLX5_DEVX=n
export UCX_RNDV_THRESH=0
export UCX_RNDV_SCHEME=get_zcopy
export UCX_WARN_UNUSED_ENV_VARS=n
export UCX_WAKEUP_PROBE_TIMEOUT_MS=10

./src/tools/perf/ucx_perftest -l -t ucp_get -s 1048576 -n 70000 -w 1000 -O 1 -T 1 -M single -c 20 -I -E sleep -f

The wakeup probe performs ucp_worker_progress(), ucp_worker_arm(),
poll(completion_fd, 10 ms), and then calls ucp_worker_progress() again
if poll times out. A timeout is productive when the final progress call
returns a non-zero value.

Observed behavior

Unmodified upstream result:

wakeup waits:          261419
completion fd ready:   259373
fd timeouts:             2046
productive timeouts:     2046
max UCX CQ CI:        9025270

On the tested device, the first stable miss was observed after the UCX CQ
consumer index crossed approximately 2^23 (8388608). The exact boundary
may depend on CQ size, hardware and provider details.

Root cause

The non-DevX path combines Direct Verbs polling with generic verbs arming:

rc_mlx5 progress -> Direct Verbs CQ poll -> UCX cq_ci advances
rc_mlx5 arm      -> ibv_req_notify_cq()   -> provider CQ CI is used

The poll and arm operations use different CQ consumer-index owners. The mlx5
CQ ARM doorbell must use the consumer index maintained by the same Direct
Verbs implementation that performs polling.

This is consistent with commit d68f347dad9afca677a1d88ea32a25529993ce6e,
UCT/DC/TEST: Fix DC arm to use uct_ib_mlx5dv_arm_cq(), which states that
ibv_req_notify_cq() is not appropriate when the mlx5 CQ consumer index is
not updated through the generic verbs path.

Proposed fix

For non-DevX rc_mlx5:

1. Keep uct_ib_iface_pre_arm().
2. Keep uct_rc_iface_arm_cq_check() and UCS_ERR_BUSY handling.
3. Arm each selected verbs CQ with uct_ib_mlx5dv_arm_cq().
4. Keep the DevX arm path unchanged.

if (md->flags & UCT_IB_MLX5_MD_FLAG_DEVX_CQ) {
    return uct_rc_mlx5_iface_devx_arm(iface, events);
}

status = uct_ib_iface_pre_arm(&iface->super.super);
if (status != UCS_OK) {
    return status;
}

dirs = uct_rc_iface_arm_cq_check(&iface->super, events, solicited);
ucs_for_each_bit(dir, dirs) {
    uct_ib_mlx5dv_arm_cq(&iface->cq[dir], solicited[dir]);
}

return UCS_OK;

Validation of the proposed fix

Hardware regression:

rc_mlx5/test_uct_event_ib_non_devx.arm_uses_direct_cq_ci/*: 50/50 passed
rc_mlx5/test_uct_event_ib.tx_cq/0:                         passed
rc_mlx5/test_uct_event_ib.txrx_cq/0:                       passed

Standalone A/B results:

Variant	Max CQ CI	FD timeouts	Productive timeouts
Generic ibv_req_notify_cq() arm	9,025,270	2,046	2,046
uct_ib_mlx5dv_arm_cq() arm	9,025,394	0	0

After applying the direct-arm change, the same workload crossed the same
consumer-index boundary with no productive wakeup timeout.

Additional information

The reproducer uses only UCX upstream source, UCX GTest, UCX ucx_perftest,
libibverbs/rdma-core and one mlx5 HCA. No application-specific framework,
storage system, filesystem or proprietary component is required.

The issue is a mismatch between CQ polling and CQ notification arming in
the non-DevX rc_mlx5 transport.

[roiedanino]

Hi @Zacky-ran,

Thanks for the detailed report.

I attempted to reproduce this on the closest ConnectX-4 Lx setup available:

• UCX commit: e5fbdbb97
• Architecture: x86_64
• HCA: ConnectX-4 Lx PF
• Link layer: Ethernet
• Firmware: 14.32.1010
• Driver: MLNX_OFED 5.1-2.5.8
• UCX_IB_MLX5_DV=y
• UCX_IB_MLX5_DEVX=n

I ran the reported 70,000-iteration workload once, followed by 10 additional repetitions, for 770,000 iterations in total. All runs completed normally in approximately 19 seconds, with no notification-related stall or timeout.

This is not an exact reproduction because I tested a PF rather than a VF, so the result is not conclusive. However, the newer firmware and driver stack may also explain the difference. Could you please update the ConnectX-4 Lx firmware and driver/provider stack and retest?

[Zacky-ran]

Hi @roiedanino ,
Thanks for testing this and for pointing out the PF/VF and software-stack
differences.

I noticed that the original reproduction instructions were incomplete:
UCX_WAKEUP_PROBE_TIMEOUT_MS is part of our temporary ucx_perftest
instrumentation and is not an upstream UCX option. Without that patch, the
standard test can show whether it eventually completes, but it cannot count a
case where the completion fd is not signaled and an immediate subsequent
ucp_worker_progress() consumes work. I will provide the missing probe patch
separately.

We have now repeated the test on the closest physical PF system available to
us. It is not the same setup as yours, so this does not yet resolve the
ConnectX-4 Lx-specific difference.

Test system:

• UCX commit: e5fbdbb97
• Architecture: x86_64
• HCA: four ConnectX-6 Dx PFs
• Link layer: Ethernet/RoCE
• Firmware: 22.41.1000 and 22.36.1010
• Runtime rdma-core/libibverbs: 57.0
• UCX_IB_MLX5_DV=y
• UCX_IB_MLX5_DEVX=n

First, we used a small hardware GTest that performs one real TX completion and
then compares the UCX-maintained CQ consumer index with the consumer index
written to the CQ ARM doorbell:

EXPECT_EQ(tx_cq->cq_ci & 0xffffff,
          be32toh(tx_cq->dbrec[UCT_IB_MLX5_CQ_ARM_DB]) & 0xffffff);

With the unmodified generic arm path, all four PFs failed in the same way:

UCX cq_ci: 1
ARM_DB CI: 0

0 passed, 4 failed

After changing the non-DevX path to uct_ib_mlx5dv_arm_cq(), all four PFs
passed:

UCX cq_ci: 1
ARM_DB CI: 1

4 passed, 0 failed

We also ran the instrumented wakeup test across the observed 2^23 boundary.
On this PF system, a 4 KiB ucp_get workload was used because it advances the
CQ consumer index once per operation.

Both variants reached:

max cq_ci = 8,401,022
2^23      = 8,388,608
difference = 12,414

Results:

Arm path	FD timeouts	Productive timeouts	Runtime
Generic ibv_req_notify_cq()	12,412	12,412	~229 s
Direct uct_ib_mlx5dv_arm_cq()	0	0	~105 s

A productive timeout means that poll(fd, 10ms) timed out, but the immediate
following ucp_worker_progress() returned a non-zero value. The 12,412
productive timeouts are almost exactly the 12,414 CQEs processed beyond the
observed half-range boundary.

This shows that PF alone does not eliminate the consumer-index mismatch or
the notification failure, at least on ConnectX-6 Dx. However, your ConnectX-4
Lx PF, firmware 14.32.1010, and MLNX_OFED 5.1-2.5.8 remain an important
different combination.

Could you please run the one-completion ARM_DB GTest on your ConnectX-4 Lx PF?
It completes in about 100 ms and does not require crossing 2^23:

• If it reports cq_ci=1, ARM_DB=0, the state mismatch exists on that PF as
  well, even if another event prevents the standard perftest from stalling.
• If it reports cq_ci=1, ARM_DB=1, your provider/firmware stack is
  synchronizing the state differently, which would give us a concrete path
  for comparing the two environments.

I have attached two small patches. Both apply directly to e5fbdbb97:

• ucx-11728-minimal-gtest.patch.txt adds only the one-completion ARM_DB
  assertion test.
• ucx-11728-wakeup-probe.patch.txt adds the optional 10 ms productive-timeout
  counter and reports the maximum UCX CQ consumer index.

Apply and run the minimal GTest:

git apply ucx-11728-minimal-gtest.patch.txt
make -C test/gtest -j$(nproc)

UCX_IB_MLX5_DV=y \
UCX_IB_MLX5_DEVX=n \
./test/gtest/gtest \
  --gtest_filter='rc_mlx5/test_uct_event_ib_non_devx.arm_uses_direct_cq_ci/*'

The wakeup probe is optional and can be applied independently:

git apply ucx-11728-wakeup-probe.patch.txt
make -C src/uct/ib/mlx5 -j$(nproc)
make -C src/tools/perf -j$(nproc)

ucx-11728-minimal-gtest.patch.txt
ucx-11728-wakeup-probe.patch.txt