Bond interface fails during syncLink

[Tenebresus]

I'm currently trying to setup openvswitch, where I want to connect a bond interface to a bridge. But in the kernel logs I see the following error:

[talos] controller failed {"component": "controller-runtime", "controller": "network.LinkSpecController", "error": "1 error occurred:\n\t* error enslaving/unslaving link \"bond1\" under \"\": netlink receive: operation not supported\n\n"}

In the code I found the following function that is responsible for throwing this error:

talos/internal/app/machined/pkg/controllers/network/link_spec.go

Line 200 in 6a01437

func (ctrl *LinkSpecController) syncLink(ctx context.Context, r controller.Runtime, logger *zap.Logger, conn *rtnetlink.Conn, wgClient *wgctrl.Client,

Does anyone have any experience with attaching a bond interface to an openvswitch bridge? And can someone explain to me why Talos wants to enslave the bond interface under an empty master?
This is my network config for the bond interface:

        - interface: bond1
          dhcp: false
          bond:
            mode: 802.3ad
            lacpRate: fast
            xmitHashPolicy: layer3+4
            miimon: 100
            updelay: 200
            downdelay: 200
            interfaces:
              - enp94s0f1
              - enp94s0f0
          mtu: 9000

[smira]

Please run talosctl get linkspecs -o yaml and post it here.

[smira]

I guess Openvswitch can configure it? Some daemonset? Static pod?

I don't have the full picture here, and I don't know how much helpful I would be.

[WarriorXK]

Hi @smira, we are having a similar issue. Would it be possible to tell Talos to create the bond but ignore the enslave status during syncs?

[smira]

Not that I know about.

In order to move this forward, it might be nice to have a minimal reproducer for it, I guess we don't need bonds, just an interface (could be even dummy) which is configured by Talos, but enslaved under a bridge without Talos knowing about it (e.g. via kubectl debug pod and ip command).

Then we can see how to get this going.

[zweizeichen]

@smira any ideas how to proceed here? Talos network config has evolved quite a bit in the meantime and I just ran into this again :P Maybe all that's needed is a mechanism for Talos to create an OVS bridge just like the current linux kernel bridge implementation.

This may look like a single use case but if I interpreted the OVS docs correctly, this will hit anyone trying to attach an OVS bridge to an interface managed by Talos networking.

The repro would be the case outlined in the OVS docs: https://docs.openvswitch.org/en/latest/faq/issues/#common-configuration-issues

1. boot talos
2. using a privileged container containing the respective ovs binaries and mounts to attach the bridge and move the IP (and see it affecting Talos network config)

---

For context: I'm trying to set up https://github.com/ovn-kubernetes/ovn-kubernetes CNI on Talos, however as part of the startup process it attaches an openvswitch bridge to:

• A) either the network interface with the default gateway on it and it then moves the IP to the bridge - probably to avoid this issue -> this breaks connectivity and leads to all kinds of trouble with Talos trying to reconcile the config
• B) ...or a predefined OVS bridge (which we can't create using Talos network config (yet)): Talos only allows you to create linux bridges so far

Option A is probably not the best move as this partially circumvents Talos network management and you'd have to handle DHCP youself in the container etc.

Option B might work if Talos would support managing OVS bridges just like the current implementation of the linux bridges (so you'd be able to configure IPs/DHCP/routes on them etc.), the CNI could then do its thing and attach all kinds of ports to the bridge etc. while Talos manages the DHCP lease.

Ignoring the interface might work for nodes with multiple interfaces but would move interface config management partially out of Talos (a bit like Option A) and would only work for machines with multiple NICs.

I thought a lot about how to resolve this (especially the single interface/bond node use case) and all solutions I came up with would be either brittle (handle network config on ignored interfaces, somehow pass in the config), static (ignoring all interfaces and using kernel parameters for initial config) or complex (again ignoring all interfaces and somehow porting NetworkManager or something similar as system extension).

Maybe adding OVS bridge support to Talos would be the most 'Talos' way to handle this. What do you think?

[smira]

I'm not a big fan of any of these options, but probably "OVS bridge" (whatever that means) sounds like a more Talos-y solution.

If the machine boots, but CNI doesn't come up -> network should still work.

If we stop Kubernetes and CNI on the machine -> network should still work.

So the fact that OVN decides to move addresses, created bridges sounds like the wrong way to approach things in the first place.