prov/tcp (xnet): with `FI_TCP_IFACE` unset on a many-interface host, CM/rendezvous hangs indefinitely (no timeout) — same-host peers latch onto a wrong interface

[TSUMUGI-XE]

Summary
On a host with ~20 UP interfaces (loopback + WiFi + tailscale + a libvirt bridge + docker0 + several docker bridges + ~12 veth* + a vnet0), a 2-process job whose ranks both resolve to the same host hangs forever during connection setup when FI_TCP_IFACE is not set. Both processes spin at ~90% CPU in xnet_run_progress() (inside libtcp-fi.so) and never make progress — no timeout, no CM error, no diagnostic. Setting FI_TCP_IFACE=lo fixes it instantly and reliably.

Environment

• libfabric 2.2.0 (2.2.0-impi_2021.17.2, API 2.2), as bundled with Intel oneAPI 2026 / Intel MPI 2021.17.
• Consumer x86-64 Linux host, single net namespace; the collective runs 2 ranks in one container with host networking, so all ~20 host interfaces are visible to the provider.
• Provider: tcp (xnet).
• Consumer of libfabric: oneCCL 2021.17 (ATL/OFI transport) driving a PyTorch/XCCL tensor-parallel job across 2 GPUs. (The GPUs are irrelevant to this hang — it is purely on the TCP control/bootstrap path.)

Backtrace (both ranks, ~90% CPU, identical)

... -> oneCCL atl_ofi_comm::init_transport
    -> atl_base_comm::create_comm_id
    -> atl_ofi_comm::allgatherv
    -> (libfabric) xnet_run_progress        <-- spins here forever, in libtcp-fi.so

The allgather that bootstraps the communicator never converges because the provider selected a non-loopback interface for peers that are actually co-located on the same host.

What I expected
For same-host peers, either (a) the provider prefers/considers loopback, or (b) if a chosen interface can't establish the CM within a bound, it fails with a diagnostic — not an unbounded spin.

What happens
Indefinite hang, no error, both ranks busy-spinning in epoll-driven progress. Indistinguishable from a deadlock.

Workaround
FI_TCP_IFACE=lo (also pinned FI_SOCKETS_IFACE=lo and the consumer's CCL_KVS_IFACE=lo). Instant, 100% reliable.

Requests

1. On a many-interface host, when FI_TCP_IFACE is unset and both endpoints resolve to the same host, prefer loopback (or at least consider it) for the tcp CM — or surface a bounded-timeout CM error instead of spinning indefinitely.
2. Document the unset-FI_TCP_IFACE interface-selection algorithm in fi_tcp(7) (it currently documents only that a specific iface can be requested, not how one is chosen otherwise).

Notes

• #7706 (CLOSED) addressed the inverse — loopback not being filtered out when FI_TCP_IFACE is set. This report is about the unset default selection hanging.
• MPI_Probe freezes on WSL2 network interface microsoft/WSL#13126 is an independent reproduction of the identical symptom (xnet_run_progress/epoll_wait hang in libtcp-fi.so, FI_TCP_IFACE=lo fixes it) filed against a different tracker — and was auto-closed for inactivity without ever being root-caused on the libfabric side. That it reproduces off-WSL too suggests this is provider-level, not app- or WSL-specific.
• Happy to attach the full gstack from both ranks and fi_info output.

[TSUMUGI-XE]

Cross-reference: I filed the oneCCL-layer half of this at uxlfoundation/oneCCL#209 — same many-interface host, same xnet_run_progress hang seen from the oneCCL ATL/OFI bootstrap allgather. That issue asks oneCCL to add a bounded timeout on its side; this one is about the tcp provider's unset-FI_TCP_IFACE selection. Filing both so the two layers can be looked at together.

[ooststep]

Could you attach the fi_info and debug logging? I know there has been a similar issue in the past with the virtual docker bridge combined with FI_DIRECTED_RECV but I believe it was avoided in oneCCL specifically for docker0.

From my recollection, when using the virtual docker bridge as the source interface, the socket would sometimes (frequently) resolve to another interface's address - causing FI_DIRECTED_RECV mismatches.

[TSUMUGI-XE]

Thanks — that matches what I'm seeing exactly, and I can now pin the interface it latches onto.

One note first: since I filed this, the stack moved to Intel oneAPI 2026.1, which bundles libfabric 2.4.0-impi (api 2.4) rather than the 2.2.0-impi in the original report. The behavior below still reproduces there, so this isn't fixed by the version bump. (I can re-capture on the exact 2.2.0 image if that's useful — say the word.)

Topology. Single net namespace, ~20 UP interfaces; the ranks run in one container with --network host, so the tcp provider sees all of them:

fi_info -p tcp  (domains, trimmed to one line per iface; each also appears as FI_EP_MSG and FI_EP_RDM/xnet)
  fabric 172.17.0.0/16      domain docker0              <- docker default bridge
  fabric 172.18.0.0/16      domain br-3b50...           <- docker user-defined bridge
  fabric 172.19.0.0/16      domain br-6a9c...           <- docker user-defined bridge
  fabric 192.168.122.0/24   domain virbr0               <- libvirt
  fabric 192.168.0.0/24     domain enp28s0              <- wired LAN
  fabric 192.168.0.0/24     domain wlp29s0f0            <- WiFi
  fabric <cgnat>/32         domain tailscale0
  fabric fe80::/64          domain vnet0, veth* (x12)
  fabric 127.0.0.1/32       domain lo

Selection trace (FI_LOG_LEVEL=debug FI_LOG_PROV=tcp, FI_TCP_IFACE unset — the hanging condition; minimal fi_pingpong -p tcp -e msg, two procs same host). IPv4 addrs shown; IPv6 link-local + the ~12 veth* omitted for brevity:

tcp:core:ofi_get_list_of_addr(): Available addr: 192.168.0.__ , iface enp28s0 ,   speed 1000    (wired LAN)
tcp:core:ofi_get_list_of_addr(): Available addr: 192.168.0.__ , iface wlp29s0f0 , speed 0       (WiFi)
tcp:core:ofi_get_list_of_addr(): Available addr: <cgnat>      , iface tailscale0, speed 100
tcp:core:ofi_get_list_of_addr(): Available addr: 192.168.122.1, iface virbr0 ,    speed 10000   (libvirt)
tcp:core:ofi_get_list_of_addr(): Available addr: 172.18.0.1   , iface br-3b50... ,speed 10000   (docker user bridge)
tcp:core:ofi_get_list_of_addr(): Available addr: 172.19.0.1   , iface br-6a9c... ,speed 10000   (docker user bridge)
tcp:core:ofi_get_list_of_addr(): Available addr: 172.17.0.1   , iface docker0 ,   speed 10000   (docker default bridge)
   ... (IPv6 link-local for all of the above, then ~12 veth* + vnet0, all speed 10000) ...
tcp:core:ofi_insert_loopback_addr(): available addr: fi_sockaddr_in://127.0.0.1:0
core:core:fi_fabric_(): Opened fabric: 172.17.0.0/16        <-- provider selects docker0

Both the server and client procs select 172.17.0.0/16 (docker0), and the pingpong then never establishes — same signature as the oneCCL bootstrap allgather hanging in xnet_run_progress(). FI_TCP_IFACE=lo fixes it instantly.

On your docker0 question — it did land on docker0 here, but note the enumeration order: virbr0, br-3b50 (172.18), br-6a9c (172.19), then docker0 (172.17) — the two user-defined docker bridges are enumerated before docker0, all at the same reported speed 10000. So an avoidance scoped to docker0 by name would still leave the user-bridge and veth* addresses in play; a different rank/ordering could plausibly latch onto 172.18/172.19 instead. Scoping to RFC1918 docker/bridge subnets generally (or just honoring FI_TCP_IFACE) would be more robust.

Two things that would have made this self-diagnosing:

1. For same-host peers, prefer/fall back to loopback (the loopback addr is enumerated, right after the bridges).
2. Bound the CM attempt and surface a diagnostic instead of spinning unbounded in xnet_run_progress() — the silent infinite spin (no timeout, no error) was the hard part.

Happy to grab the full RDM/xnet fi_info, the IPv6 lines, or a trace from the actual oneCCL 2-rank launch if any of those help.

[ooststep]

1. For same-host peers, prefer/fall back to loopback (the loopback addr is enumerated, right after the bridges).

this seems like a decision for the application (or middleware, CCL, in this case) to make rather than the provider making assumptions. they can likely make a more informed decision on which interface to use based on their use case. Maybe lo fallback can be an opt-in feature - but I'm not sure we want to make it the default behavior.

2. Bound the CM attempt and surface a diagnostic instead of spinning unbounded in xnet_run_progress() — the silent infinite spin (no timeout, no error) was the hard part.

it's a good idea. Feel free to offer a patch for it if you are able. Is it a CM issue, though, or a protocol level src addr mismatch related to FI_DIRECTED_RECV? If it is indeed the same issue I am remembering, I would expect connection is successful, but FI_DIRECTED_RECV matches to fail. I believe we can identify this in the debug logging.

[TSUMUGI-XE]

Answering your question directly: it's the second case — the connection is successful; it's a source-address mismatch, not a CM failure.

Captured the RDM/xnet path (fi_pingpong -p tcp -e rdm, FI_TCP_IFACE unset, two procs same host, libfabric 2.4.0-impi). The connection manager completes cleanly, but the address the peer connects from isn't the address that's in the AV:

tcp:av:ofi_av_insert_addr(): inserting addr: fi_sockaddr_in://172.17.0.1:44555   fi_addr: 0
tcp:av:ofi_av_insert_addr(): inserting addr: fi_sockaddr_in://172.17.0.1:34249   fi_addr: 1
tcp:ep_ctrl:xnet_handle_event_list(): event FI_CONNREQ
tcp:ep_ctrl:xnet_process_connreq(): connreq for 0x...
tcp:ep_ctrl:xnet_handle_event_list(): event FI_CONNECTED
tcp:ep_ctrl:xnet_process_connected(): peer fi_sockaddr_in://192.168.0.x:34249 feature supported: 0   <-- LAN NIC
tcp:ep_ctrl:xnet_handle_event_list(): event FI_CONNECTED
tcp:ep_ctrl:xnet_process_connected(): peer fi_sockaddr_in://172.17.0.1:34249 feature supported: 0    <-- docker0 (matches AV)

So for the endpoint at port 34249 (fi_addr 1): the AV holds it as 172.17.0.1 (docker0) — that's what the OOB bootstrap advertised, since the provider opened the 172.17.0.0/16 (docker0) fabric — but the accepted connection reports the peer source as 192.168.0.x (the wired LAN NIC). Same logical peer, two different source addresses; the one the socket actually used isn't the one in the AV.

That's the failure mode you were remembering: CM succeeds, but a FI_DIRECTED_RECV posted for fi_addr 1 (expecting a message from 172.17.0.1) can't match data that arrives from 192.168.0.x — consistent with the oneCCL allgather's directed receive spinning in xnet_run_progress() in the original report. FI_TCP_IFACE=lo makes every advertised/connected address 127.0.0.1, so the AV and the connection agree and matching succeeds immediately.

Given that, my earlier "bound the CM attempt" suggestion is aimed at the wrong layer — the CM is fine. The useful diagnostic would be at the match layer: when a message arrives whose source address resolves to no AV entry (or a directed recv can't be matched to any connected source), emit a warning rather than silently spinning. I'd be glad to look at a patch for that if it sounds right to you.

If it helps, I can also capture the debug log straight from the oneCCL allgather (the real hang) showing the directed-recv match miss directly, rather than from this minimal pingpong — just say the word.