ICE selects CNI host-side interfaces (cilium_host) as WireGuard endpoints, breaking the mesh it runs on

[dappermint]

Describe the problem

On a Kubernetes node running a CNI, the client gathers the CNI's host-side interface address as an ICE host candidate and then selects it as the WireGuard endpoint for a peer. On Cilium that interface is cilium_host, a /32 inside the node's pod CIDR that is only reachable through the CNI overlay — and the overlay is carried by NetBird. So the client configures a peer endpoint that can only work if the mesh is already working.

The connection fails, ICE re-handshakes, picks another CNI address, and repeats.

On a 4-node cluster (Cilium 1.19.6, VXLAN, NetBird as the node underlay), 445 of 454 endpoint selections in one client's log were pod-CIDR addresses:

 175 configure WireGuard endpoint to: 10.2.1.108   <- cilium_host, node B
 152 configure WireGuard endpoint to: 10.2.2.124   <- cilium_host, node C
 118 configure WireGuard endpoint to: 10.2.3.22    <- cilium_host, node D
   5 configure WireGuard endpoint to: <node C public IP>   <- actually routable
   1 configure WireGuard endpoint to: <load balancer public IP>
   1 configure WireGuard endpoint to: <node B public IP>
   1 configure WireGuard endpoint to: 192.168.122.1
   1 configure WireGuard endpoint to: <relay public IP>

Each node's cilium_host is a /32 from its own pod CIDR:

node-a (control plane)  podCIDR 10.2.2.0/24    cilium_host 10.2.2.124/32
node-b                  podCIDR 10.2.0.0/24    cilium_host 10.2.0.113/32
node-c                  podCIDR 10.2.3.0/24    cilium_host 10.2.3.22/32
node-d                  podCIDR 10.2.1.0/24    cilium_host 10.2.1.108/32

Symptoms

Measured from a host-network pod, same source, same destination machine, back to back:

public IP of node-b : 60 packets, 0% loss,  0.65ms
mesh IP of node-b   : 60 packets, 51% loss, 45ms

Loss is bursty and appears to move between peers, because which peer is broken depends on which candidate ICE most recently picked. Repeat runs of an identical test gave 0%, then 80%, then 0% to the same target, which made this look like a flaky link or an MTU problem for a long time before the endpoint log explained it.

Downstream, this starved every leader-election lease on the affected nodes. csi-attacher, csi-provisioner, barman-cloud, harbor-core and others crashlooped on failed to renew lease: context deadline exceeded with 200+ restarts each, and kubectl logs/exec to those nodes intermittently timed out.

Expected behaviour

Interfaces belonging to a CNI / container network should not be offered as ICE host candidates, or there should be a way to exclude an interface or a CIDR from candidate gathering.

I looked through the full client environment variable reference and could not find an existing knob for this. NB_FORCE_RELAY=true is an effective workaround since it skips ICE entirely, but it gives up direct connectivity for every peer, not just the bad candidates.

Are you using NetBird Cloud?

Self-hosted.

NetBird version

0.74.5 (Talos system extension ghcr.io/siderolabs/netbird:0.74.5, the newest build in the Talos factory for v1.13.7)

Environment

• Talos Linux v1.13.7, 4 nodes, each a libvirt VM
• Cilium 1.19.6, routing-mode: tunnel, tunnel-protocol: vxlan, ipam: kubernetes, pod subnet 10.2.0.0/16
• NetBird is the node underlay: kubelet node IPs, kubeconfig endpoint and inter-node traffic all ride wt0 (a 100.x.0.0/16 mesh range)
• Interfaces present on a node, i.e. the candidate pool:

lo          127.0.0.1/8
enp1s0      192.168.122.12/24     <- libvirt NAT, same /24 on every host
tailscale0  <tailscale /32>
wt0         <netbird mesh /16>    <- netbird itself
cilium_host 10.2.0.113/32         <- CNI, gets selected

Note the libvirt NAT address is also a poor candidate: every hypervisor NATs its own 192.168.122.0/24, so that address is ambiguous across hosts as well.

Relevant log output

peer/conn.go:465: configure WireGuard endpoint to: 10.2.1.108:51820
peer/conn.go:430: set ICE to active connection
peer/conn.go:465: configure WireGuard endpoint to: 10.2.3.22:51820
peer/conn.go:430: set ICE to active connection

174 set ICE to active connection entries in a single client log, reflecting the constant re-handshaking as each unreachable endpoint fails.

[linear-code]

NET-1568

[lixmal]

--extra-iface-blacklist and the equivalent env var