Failed to allocate pod ip even there're plenty available

[ttc0419]

event:

14m                    Warning   FailedCreatePodSandBox   Pod/ingress-nginx-controller-644966f9d8-mmrdb   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "566461a584386191b1cc4dcc05193d30f7120f467ab2c694e2811eda1db34247": plugin type="flannel" failed (add): failed to allocate for range 0: no IP addresses available in range set: 192.168.63.161-192.168.63.190

pods:

NAME                                        READY   STATUS              RESTARTS   AGE   IP               NODE            NOMINATED NODE   READINESS GATES
ingress-nginx-controller-644966f9d8-6l8cg   1/1     Running             0          16m   192.168.63.196   192.168.63.8    <none>           <none>
ingress-nginx-controller-644966f9d8-glqzs   1/1     Running             0          16m   192.168.63.131   192.168.63.10   <none>           <none>
ingress-nginx-controller-644966f9d8-mmrdb   0/1     ContainerCreating   0          16m   <none>           192.168.63.9    <none>           <none>
kube-flannel-ds-5tdbb                       1/1     Running             0          32m   192.168.63.8     192.168.63.8    <none>           <none>
kube-flannel-ds-wxzdt                       1/1     Running             0          32m   192.168.63.10    192.168.63.10   <none>           <none>
kube-flannel-ds-zc9m9                       1/1     Running             0          32m   192.168.63.9     192.168.63.9    <none>           <none>

related: #1416 kubernetes/kubernetes#57280 (comment)

[manuelbuil]

Can you check sudo ls /var/lib/cni/networks/cbr0? I suspect that directory is full of files with the different IPs. You can remove the ones that are not used anymore. Either containerd or flannel had problems when the container was removed and "forgot" to remove that file

[ttc0419]

@manuelbuil Yeah, that's the issue, but I'm wondering if flannel should fix or mitigate this issue, it can be easily reproduced if you reboot the nodes

[manuelbuil]

I rebooted the node but I can't really reproduce it. Can you explain the steps you followed to reproduce it? Note that this is more likely to be a problem of the container runtime than of the CNI plugin. The container runtime should talk to the CNI and tell it to remove everything of an endpoint, if this call does not happen, the CNI plugin can't know that the pod is gone

[ttc0419]

@manuelbuil okay, I think the issue happens when you only have some of the standard CNI plugin installed. In my case, it was bridge, host-local and loopback, the kubelet will complain the portmap plugin is missing but keeps allocating new IPs under /var/lib/cni/networks/cbr0 for some reason. BTW, why does flannel needs these CNI plugins?

[manuelbuil]

That's how Flannel was built. It uses the cbr0 bridge to connect the different interfaces and the portmap to do port mapping

[ttc0419]

@manuelbuil Is there a list in the doc of which CNIs does flannel need to work?

[oshoval]

Hi, would it make sense to create a daemonSet which cleanup IP zombies?

iirc, by design, DEL might be lost, hence having GC makes sense

We also see this issue, reasoning below.
Long tests have this problem after a while (using /24 per node)
The job runs lot of upgrades
https://prow.ci.kubevirt.io/view/gs/kubevirt-prow/pr-logs/pull/kubevirt_cluster-network-addons-operator/2428/pull-e2e-cluster-network-addons-operator-lifecycle-k8s/1978786400497045504
it didnt happen with Calico for example

14m         Warning   FailedCreatePodSandBox   pod/secondary-dns-5fc6686967-t97vf   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_secondary-dns-5fc6686967-t97vf_cluster-network-addons_9b165a5c-539f-41d4-9781-1f5ae2cb3311_0(5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402): error adding pod cluster-network-addons_secondary-dns-5fc6686967-t97vf to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402" Netns:"/var/run/netns/b30d651d-76fb-4e01-bcae-b162095eeba8" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=cluster-network-addons;K8S_POD_NAME=secondary-dns-5fc6686967-t97vf;K8S_POD_INFRA_CONTAINER_ID=5c39f1e51a02a48e2d99e0670c449379f2d45c9c631971d96f078b55931c3402;K8S_POD_UID=9b165a5c-539f-41d4-9781-1f5ae2cb3311" Path:"" ERRORED: error configuring pod [cluster-network-addons/secondary-dns-5fc6686967-t97vf] networking: [cluster-network-addons/secondary-dns-5fc6686967-t97vf/9b165a5c-539f-41d4-9781-1f5ae2cb3311:cbr0]: error adding container to network "cbr0": plugin type="flannel" failed (add): failed to allocate for range 0: no IP addresses available in range set: 10.244.0.1-10.244.0.254...