[BUG] Support for swapping networks on an existing interface

[frenzyfriday]

Describe the bug
The current implementation does not support changing the network on an existing interface by going directly from:
'[{"name":"nad-first","namespace":"default","mac":"02:87:f5:51:4c:5d","interface":"poda1363d52898"}]'
to
'[{"name":"nad-second","namespace":"default","mac":"02:87:f5:51:4c:5d","interface":"poda1363d52898"}]'

Trying to do this I see the pod interface connected to nad-first disappear, but not reappear connected to nad-second.

But doing a describe on the pod shows that the second reconciliation added the interface:

  Normal   Started                3m23s  kubelet               Started container volumecontainerdisk
  Warning  FailedAddingInterface  10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: failed adding interface poda1363d52898 to network: nad-second(ips: [], mac: 02:41:29:d6:ed:cc, cni-args: <nil>)
  Normal   RemovedInterface       10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: removed interface poda1363d52898 from network: nad-second
  Normal   AddedInterface         10s    multus                Add poda1363d52898 [192.168.222.2/24] from nad-second
  Normal   AddedInterface         10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: added interface poda1363d52898 to network: nad-second(ips: [], mac: 02:41:29:d6:ed:cc, cni-args: <nil>)
  Normal   RemovedInterface       10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: removed interface poda1363d52898 from network: nad-first

Expected behavior
The interface is added back to the pod and is connected to nad-second
The interface shows up in ip a

To Reproduce
Steps to reproduce the behavior:

1. Create nad-first and nad-second. I tried with the following:

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: nad-first
  namespace: default
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "bridge",
      "bridge": "br1",
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.111.0/24"
      }
    }'

2. Create a VM with an interface connected to nad-first. I tried with the following:

spec:
  ...
  template:
    ...
      networks:
        - multus:
            networkName: nad-first
          name: net-first
      domain:
        devices:
          interfaces:
            - bridge: { }
              name: net-first

3. Check the pod corresponding to the VM has the correct annotation

kc get pods virt-launcher-vm-test-vnthp -o yaml
...
  annotations:
...
    k8s.v1.cni.cncf.io/networks: '[{"name":"nad-first","namespace":"default","mac":"02:41:29:d6:ed:cc","interface":"poda1363d52898"}]'

4. Exec into the pod to see the interface

kc exec -it virt-launcher-vm-test-vnthp -- bash
bash-5.1$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue state UP group default qlen 1000
    link/ether 6e:39:97:01:d9:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.32.7/20 brd 10.244.47.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd10:244::200:0:0:7/72 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::6c39:97ff:fe01:d947/64 scope link 
       valid_lft forever preferred_lft forever
3: a1363d52898-nic@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master k6t-a1363d52898 state UP group default qlen 1000
    link/ether fe:87:98:77:fe:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc87:98ff:fe77:fe10/64 scope link 
       valid_lft forever preferred_lft forever
4: k6t-a1363d52898: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:0f:c7:40:aa:1d brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.10/32 scope global k6t-a1363d52898
       valid_lft forever preferred_lft forever
    inet6 fe80::fc87:98ff:fe77:fe10/64 scope link 
       valid_lft forever preferred_lft forever
5: tapa1363d52898: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master k6t-a1363d52898 state UP group default qlen 1000
    link/ether da:0f:c7:40:aa:1d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d80f:c7ff:fe40:aa1d/64 scope link 
       valid_lft forever preferred_lft forever
6: poda1363d52898: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 02:41:29:d6:ed:cc brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.2/24 brd 192.168.111.255 scope global poda1363d52898
       valid_lft forever preferred_lft forever
    inet6 fe80::41:29ff:fed6:edcc/64 scope link 
       valid_lft forever preferred_lft forever

5. Edit the pod and change the annotation manually to k8s.v1.cni.cncf.io/networks: '[{"name":"nad-second","namespace":"default","mac":"02:41:29:d6:ed:cc","interface":"poda1363d52898"}]'
6. Describe the pod to see the removal/addition of interfaces

kubectl describe pods virt-launcher-vm-test-vnthp

  Normal   Started                3m23s  kubelet               Started container volumecontainerdisk
  Warning  FailedAddingInterface  10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: failed adding interface poda1363d52898 to network: nad-second(ips: [], mac: 02:41:29:d6:ed:cc, cni-args: <nil>)
  Normal   RemovedInterface       10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: removed interface poda1363d52898 from network: nad-second
  Normal   AddedInterface         10s    multus                Add poda1363d52898 [192.168.222.2/24] from nad-second
  Normal   AddedInterface         10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: added interface poda1363d52898 to network: nad-second(ips: [], mac: 02:41:29:d6:ed:cc, cni-args: <nil>)
  Normal   RemovedInterface       10s    pod-networks-updates  pod [default/virt-launcher-vm-test-vnthp]: removed interface poda1363d52898 from network: nad-first

8. Exec into the pod to see the missing interface

kc exec -it virt-launcher-vm-test-vnthp -- bash
bash-5.1$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue state UP group default qlen 1000
    link/ether 6e:39:97:01:d9:47 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.32.7/20 brd 10.244.47.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd10:244::200:0:0:7/72 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::6c39:97ff:fe01:d947/64 scope link 
       valid_lft forever preferred_lft forever
3: a1363d52898-nic@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master k6t-a1363d52898 state UP group default qlen 1000
    link/ether fe:87:98:77:fe:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc87:98ff:fe77:fe10/64 scope link 
       valid_lft forever preferred_lft forever
4: k6t-a1363d52898: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:0f:c7:40:aa:1d brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.10/32 scope global k6t-a1363d52898
       valid_lft forever preferred_lft forever
    inet6 fe80::fc87:98ff:fe77:fe10/64 scope link 
       valid_lft forever preferred_lft forever
5: tapa1363d52898: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master k6t-a1363d52898 state UP group default qlen 1000
    link/ether da:0f:c7:40:aa:1d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::d80f:c7ff:fe40:aa1d/64 scope link 
       valid_lft forever preferred_lft forever

Environment:

• multus-dynamic-networks-controller version: v0.3.6 (ghcr.io/k8snetworkplumbingwg/multus-dynamic-networks-controller@sha256:2a2bb32c0ea8b232b3dbe81c0323a107e8b05f8cad06704fca2efd0d993a87be)
• Kubernetes version (use kubectl version):

kc version
Client Version: v1.32.8
Kustomize Version: v5.5.0
Server Version: v1.33.6

• Network-attachment-definition: N/A
• OS (e.g. from /etc/os-release): Fedora 42
• Controller configuration (criSocketPath / multusSocketPath): N/A
• Kernel (e.g. uname -a): N/A
• Others: N/A

Additional info / context
Add any other information / context about the problem here.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!