mlx5: ZTR-RTT CC is not enabled by default when DEVX=try (unlike DEVX=no)

[kimullaa]

Describe the bug

Following the fix in #11262, ZTR-RTT CC is now utilized by default when DEVX=no. However, when DEVX=try, ZTR-RTT CC is not enabled by default.

I think it might be better to configure the initial ECE value based on the value obtained from ibv_query_ece even when DEVX=try (with UCX_RC_ECE=auto).

Steps to Reproduce

• ucx: 5290712

$ UCX_RC_ECE=auto  UCX_IB_MLX5_DEVX=try ./src/tools/perf/.libs/lt-ucx_perftest -n 100 -t ucp_am_lat
$ UCX_RC_ECE=auto  UCX_IB_MLX5_DEVX=try ./ucx/src/tools/perf/.libs/lt-ucx_perftest -n 100 -t ucp_am_lat 172.1.1.66
$ tshark -r dump.pcap | grep -i VENDO
[empty]

$ UCX_RC_ECE=auto  UCX_IB_MLX5_DEVX=no ./src/tools/perf/.libs/lt-ucx_perftest -n 100 -t ucp_am_lat
$ UCX_RC_ECE=auto  UCX_IB_MLX5_DEVX=no ./ucx/src/tools/perf/.libs/lt-ucx_perftest -n 100 -t ucp_am_lat 172.1.1.66
$ tshark -r dump.pcap  | grep -i vendor | head
  192   0.316188   172.1.1.66 → 172.1.1.55   RRoCE 130 VENDOR (Unknown Attribute)
  195   0.316189   172.1.1.55 → 172.1.1.66   RRoCE 130 VENDOR (Unknown Attribute)
  197   0.316189   172.1.1.55 → 172.1.1.66   RRoCE 130 VENDOR (Unknown Attribute)
  199   0.316879   172.1.1.66 → 172.1.1.55   RRoCE 130 VENDOR (Unknown Attribute)
  204   0.317561   172.1.1.66 → 172.1.1.55   RRoCE 130 VENDOR (Unknown Attribute)

Setup and versions

• OS version (e.g Linux distro) + CPU architecture (x86_64/aarch64/ppc64le/...)
  • Red Hat Enterprise Linux release 9.7 (Plow)
  • 14.0-611.49.1.el9_7.x86_64
• For RDMA/IB/RoCE related issues:
  • Driver version:
    • rdma-core-2601.0.7-1.el9.x86_64
    • OFED-internal-26.01-1.0.0

CA 'mlx5_0'
       CA type: MT4125
       Number of ports: 1
       Firmware version: 22.48.1000
       Hardware version: 0
       Node GUID: 0x3825f30300c86068
       System image GUID: 0x3825f30300c86068
       Port 1:
               State: Active
               Physical state: LinkUp
               Rate: 100
               Base lid: 0
               LMC: 0
               SM lid: 0
               Capability mask: 0x00010000
               Port GUID: 0x3a25f3fffec86068
               Link layer: Ethernet

Additional information (depending on the issue)

Below is the current analysis of this behavior. By adding print debugging to the macro that sets the ECE field, I observed that -2 (UCS_ULUNITS_AUTO) is being passed:

$ git diff
diff --git a/src/uct/ib/mlx5/dv/ib_mlx5_ifc.h b/src/uct/ib/mlx5/dv/ib_mlx5_ifc.h
index 34dd74bed..f6d0cbc85 100644
--- a/src/uct/ib/mlx5/dv/ib_mlx5_ifc.h
+++ b/src/uct/ib/mlx5/dv/ib_mlx5_ifc.h
@@ -43,6 +43,13 @@
                 (~__uct_dw_mask(_typ, _fld))) | \
                (((___v) & __uct_mask(_typ, _fld)) << \
                 __uct_dw_bit_off(_typ, _fld)); \
+                                                                              \
+        if (#_fld[0] == 'e' && #_fld[1] == 'c' && #_fld[2] == 'e' && #_fld[3] == '\0') { \
+            fprintf(stderr, "[UCT_ECE_SET] Type: %s | Input Val: %d | Written DW: 0x%X\n", \
+                    #_typ, (int)___v, (unsigned int)___h); \
+            fflush(stderr); \
+        } \
         *((__be32 *)(___p) + __uct_dw_off(_typ, _fld)) = htobe32(___h); \
     } while (0)


$  UCX_RC_ECE=auto  UCX_IB_MLX5_DEVX=try ./src/tools/perf/.libs/lt-ucx_perftest -n 100 -t ucp_am_lat 172.1.1.66
[1781577030.470968] [tower20:192004:0]        perftest.c:810  UCX  WARN  CPU affinity is not set (bound to 24 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |        latency (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[UCT_ECE_SET] Type: dctc | Input Val: -2 | Written DW: 0xFFFFFFFE
[UCT_ECE_SET] Type: create_qp_in | Input Val: 268435456 | Written DW: 0x10000000
[UCT_ECE_SET] Type: create_qp_in | Input Val: 268435456 | Written DW: 0x10000000
[UCT_ECE_SET] Type: init2rtr_qp_in | Input Val: -2 | Written DW: 0xFFFFFFFE
Final:                   100    340.766   344.355   344.355        0.02       0.02        2904        2904

Based on the source code, config->ece seems to be initialized here (where UCS_ULUNITS_AUTO evaluates to -2):

ucx/src/uct/ib/rc/base/rc_iface.c

Lines 614 to 621 in f743868

	if (!md->ece_enable && (config->ece != UCS_ULUNITS_AUTO)) {
	ucs_error("%s: cannot set ECE value to 0x%lx since the device does not "
	"support ECE", uct_ib_device_name(dev), config->ece);
	status = UCS_ERR_INVALID_PARAM;
	goto err;
	}
	self->config.ece = config->ece;

set ECE is here

ucx/src/uct/ib/mlx5/rc/rc_mlx5_devx.c

Lines 473 to 475 in f743868

	if (md->super.ece_enable) {
	UCT_IB_MLX5DV_SET(init2rtr_qp_in, in_2rtr, ece, iface->super.config.ece);
	}

I think that processing equivalent to uct_rc_iface_set_ece may be required.

ucx/src/uct/ib/rc/base/rc_iface.c

Lines 876 to 886 in f743868

	if (ibv_query_ece(qp, &ece)) {
	ucs_error("ibv_query_ece(device=%s qpn=0x%x) failed: %m",
	uct_ib_device_name(dev), qp->qp_num);
	return UCS_ERR_IO_ERROR;
	}
	if (ece_val == UCS_ULUNITS_INF) {
	ece.options = UCT_IB_DEVICE_ECE_MAX;
	} else if (ece_val != UCS_ULUNITS_AUTO) {
	ece.options = ece_val;
	}