We are using ovs-cni in combination with cilium and multus on the CNI layer, the primary hosted workload are VMs running inside kubevirt. On the network setup we're using VFs (SRIOV).
Performing VM stop or migration operations does randomly leave pods stuck in terminating state.
Dec 23 10:02:07 n5 kubelet[6776]: E1223 10:02:07.632106 6776 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): DelegateDel: error invoking DelegateDel - \"ovs\": error in getting result from DelNetwork: file exists" podSandboxID="3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef"
Dec 23 10:02:07 n5 kubelet[6776]: E1223 10:02:07.632209 6776 kubelet.go:2130] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"8d0c3d18-dc69-4be7-993f-16022fb0a20e\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): DelegateDel: error invoking DelegateDel - \\\"ovs\\\": error in getting result from DelNetwork: file exists\"" logger="UnhandledError"
Dec 23 10:02:07 n5 kubelet[6776]: E1223 10:02:07.633327 6776 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"8d0c3d18-dc69-4be7-993f-16022fb0a20e\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): DelegateDel: error invoking DelegateDel - \\\"ovs\\\": error in getting result from DelNetwork: file exists\"" pod="customer-lab2/virt-launcher-customer-lab2-vms-lab2swyx-mhtml" podUID="8d0c3d18-dc69-4be7-993f-16022fb0a20e"
Dec 23 10:02:22 n5 kubelet[6776]: E1223 10:02:22.634430 6776 log.go:32] "StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox \"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\": plugin type=\"multus\" name=\"multus-cni-network\" failed (delete): DelegateDel: error invoking DelegateDel - \"ovs\": error in getting result from DelNetwork: file exists" podSandboxID="3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef"
Dec 23 10:02:22 n5 kubelet[6776]: E1223 10:02:22.634528 6776 kubelet.go:2130] "Unhandled Error" err="failed to \"KillPodSandbox\" for \"8d0c3d18-dc69-4be7-993f-16022fb0a20e\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): DelegateDel: error invoking DelegateDel - \\\"ovs\\\": error in getting result from DelNetwork: file exists\"" logger="UnhandledError"
Dec 23 10:02:22 n5 kubelet[6776]: E1223 10:02:22.635603 6776 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"KillPodSandbox\" for \"8d0c3d18-dc69-4be7-993f-16022fb0a20e\" with KillPodSandboxError: \"rpc error: code = Unknown desc = failed to destroy network for sandbox \\\"3d92b288e9cae05c8f2d2594e0ae94401df19ce749b794508a8850c3c2a5e2ef\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (delete): DelegateDel: error invoking DelegateDel - \\\"ovs\\\": error in getting result from DelNetwork: file exists\"" pod="customer-lab2/virt-launcher-customer-lab2-vms-lab2swyx-mhtml" podUID="8d0c3d18-dc69-4be7-993f-16022fb0a20e"
Forceful deletion of the pod will recover from this state and does not leave regressions on the host. We did quite some diagnosis on this and have possibly identified a race condition;
Race condition: kubeVirt rename vs. CNI DEL
Thread A (virt-launcher): renames pod37a8eec1ce1 → 37a8eec1ce1-nic
Thread B (CNI DEL): looks up pod37a8eec1ce1 → found
calls renameLink(pod37a8eec1ce1, 37a8eec1ce1-nic)
← Thread A wins first → EEXIST
vs. the timing where DEL wins:
Thread B (CNI DEL): renameLink(pod37a8eec1ce1, 37a8eec1ce1-nic) → success
Thread A (virt-launcher): looks up pod37a8eec1ce1 → not found → skips
This is why it is random — it depends purely on whether Kubernetes triggers DEL before or after KubeVirt's internal network setup runs. No lock coordinates them.
Expected behavior is to always succeed on the termination.
We are using ovs-cni in combination with cilium and multus on the CNI layer, the primary hosted workload are VMs running inside kubevirt. On the network setup we're using VFs (SRIOV).
Performing VM stop or migration operations does randomly leave pods stuck in terminating state.
Forceful deletion of the pod will recover from this state and does not leave regressions on the host. We did quite some diagnosis on this and have possibly identified a race condition;
Race condition: kubeVirt rename vs. CNI DEL
vs. the timing where DEL wins:
This is why it is random — it depends purely on whether Kubernetes triggers DEL before or after KubeVirt's internal network setup runs. No lock coordinates them.
Expected behavior is to always succeed on the termination.