KubeSpan connection broken after network outage

[gjabell]

Bug Report

Description

I have the following setup for a small Talos cluster:

• 3 control-plane/worker nodes on a private home network (i.e. non-public IPs)
• 1 worker node in Hetzner cloud (with a public IP)

I'm using KubeSpan to connect everything together; the cloud node has a firewall with 51820/UDP allowed which satisfies the requirement of one node being publically accessible.

Until recently this setup worked flawlessly, however since a brief network outage one of the controllers cannot contact the worker & vice-versa; running talosctl get kubespanpeerstatuses on both sides shows the other node as "down". The other two controllers connect successfully to the worker, and all controllers connect to each other via KubeSpan.

I've tried the following:

• Restarting each node
• Adding a KubeSpan filter on the controllers to only publish the private IP addresses; since the public IP is not accessible anyways

I imagine this is some strange firewall issue but the fact that two out of three nodes are able to successfully connect is odd.

Logs

Nothing of note in the logs except some warnings, which I assume happen during startup as the peers are established:

user: warning: [2026-04-20T12:01:29.409948029Z]: [talos] reconfigured wireguard link {"component": "controller-runtime", "controller": "network.LinkSpecController", "link": "kubespan", "peers": 3}

Are there other logs which I should check?

Environment

• Talos version: v1.12.6
• Kubernetes version: v1.35.2
• Platform: Bare-metal + Hetzner (cloud)

[smira]

Converted to a discussion, as this is a question.

I would start ensuring that discovery works, that all nodes see each other, e.g. with talosctl get members on each node.

After that you can check kubespanpeerspecs and kubespanpeerstatus and isolate the pair which doesn't work. If it doesn't work, the only meaningful reason is that the connection is blocked, or some other more strange NAT issue.

[gjabell]

All members see each other in both members and kubespanpeerspecs, and everything looks reasonable in kubespanpeerstatus except for the nodes marking each other as down, so I guess discovery isn't the problem.

Must be some firewalling or NAT issue that I'm not seeing; figured I would double check here in case there's something silly I am overlooking 🙂

Thanks for your help, I'll write back here if I end up figuring it out, in case other people run into the same issue.

[gjabell]

This ended up being an issue with my Multi-WAN setup, likely some NAT weirdness where packets would get routed to the wrong WAN interface and dropped if the connections were initialized before the router completely settled.

In case anyone runs into a similar problem, my solution was to add a hook that runs when the WAN falls over to reset open connections to the remote server (e.g. conntrack -D --dst <server ip>), which triggers KubeSpan to reconnect via the new WAN interface.