failed to reset vf and duplicate MAC addresses

[zhonglin6666]

What happened?

    vf 0     link/ether b2:58:37:4a:66:9c brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:23:00:24:8d:13 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether ce:58:b8:93:0c:a4 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 3     link/ether e2:66:c3:15:7a:55 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 4     link/ether 00:23:00:24:8d:13 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

Vf 1 and vf 4 have the same mac addresses, cause network unreachable

What did you expect to happen?

vf should not use the same mac addresses.

What are the minimal steps needed to reproduce the bug?

1. start kubevirt vm-1 use fixed mac addresses sriov network
2. delete multus pod, let it not work
3. stop kubelet vm, found mac not be reset by sriov-cni
4. start multus pod, let it work
5. start kubevirt vm-1, found two same mac address vf

Anything else we need to know?

The kubelet deletes pods, continuing the deletion even if the network fails. However, sriov-cni was not executed, leading to a sriov VF leak.

Component Versions

Please fill in the below table with the version numbers of components used.

Component	Version
SR-IOV Network Device Plugin	v1.3
SR-IOV CNI Plugin
Multus	v4.0.3
Kubernetes	v1.21.5
OS	Ubuntu 22.04/Ubuntu 18.04

Config Files

Config file locations may be config dependent.

Device pool config file location (Try '/etc/pcidp/config.json')

Multus config (Try '/etc/cni/multus/net.d')

CNI config (Try '/etc/cni/net.d/')

Kubernetes deployment type ( Bare Metal, Kubeadm etc.)

Kubeconfig file

SR-IOV Network Custom Resource Definition

Logs

SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)

Multus logs (If enabled. Try '/var/log/multus.log' )

Kubelet logs (journalctl -u kubelet)

[SchSeba]

Hi @zhonglin6666 can you please check the multus logs.
enable the debug mode for the sriov-cni (via the configuration) check the sriov-cni repo for that if you need.

we need to understand why the sriov-cni didn't manage to reset the VF mac address when it gets reset

[zhonglin6666]

Hi @zhonglin6666 can you please check the multus logs. enable the debug mode for the sriov-cni (via the configuration) check the sriov-cni repo for that if you need.

we need to understand why the sriov-cni didn't manage to reset the VF mac address when it gets reset

The pod was evicted, and the network failed to be released due to a failure in invoking multus for deletion.
This causes the SR-IOV VF not to be reset, and when the pod is rescheduled and created, a new VF will be used.

Should a mechanism be added to periodically clean up leaked VFs?