[BUG] No Address Allocated to Pod

[ade-iqbal]

Kube-OVN Version

v1.12.19

Kubernetes Version

v1.28.11

Operation-system/Kernel Version

Ubuntu 22.04.4 LTS 5.15.0-117-generic

Description

I create a workload on my cluster, but the pod gets stuck on creating the container. When I check the event of the pod, this is because there is no allocated IP for the pod. But my subnet still has many IPs to allocate for the workload. The following are details:

Events:
Type      Reason                  Age      From                   Message
Warning   AcquireAddressFailed    7m28s    kube-ovn-controller    AddressOutOfRange 
Warning   FailedCreatePodSandBox  9s       kubelet                (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "8d8xxx": plugin type="multus" name="multus-cni-network" failed (add): [default/my-pod-xxx:generic-veth]: error adding container to network "generic-veth": plugin type="kube-ovn" failed (add): RPC failed; request ip return 500 no address allocated to pod default/my-pod-xxx provider ovn, please see kube-ovn-controller logs to find errors

On my kube-ovn-cni logs, I just got this information:

wait address for pod default/my-pod-xxx provider ovn
no address allocated to pod default/my-pod-xxx provider ovn, please see kube-ovn-controller logs to find errors
del port request: {kube-ovn my-pod-xxx default xxx /var/run/netns/cni-xxx eth0 ovn [] {[] [] []} }

Steps To Reproduce

1. Creating a subnet with CIDR /24
2. Create a pod

Current Behavior

Pod can't have an IP and is stuck on creating the container

Expected Behavior

Pod gets an IP and status is Running

[dosubot]

Halo @ade-iqbal! Saya adalah bot yang siap membantu Anda menyelesaikan bug, menjawab pertanyaan, dan menjadi kontributor. Saya di sini untuk membantu Anda sambil menunggu pemelihara manusia. Mari kita atasi masalah ini bersama! 🚀

I found a similar closed issue titled "request ip return 500 no address allocated to pod $POD_NAME provider ovn, please see kube-ovn-controller logs to find errors" which was resolved by a pull request that replaced StrategicMergePatchType with MergePatchType ^[1].

Additionally, there is another closed issue titled "[BUG] wrong ippool allocation](#4687)" which was resolved by supporting multiple IPPools in the namespace ^[2].

You might want to check if the solution from these issues applies to your case, such as verifying the patch type used or considering the configuration of IPPools in your namespace.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[zhangzujian]

no address allocated to pod default/my-pod-xxx provider ovn, please see kube-ovn-controller logs to find errors

Please provide logs of the kube-ovn-controller pods.

[ade-iqbal]

There aren't logs kube-ovn-controller pods about this issue, only have logs for another pods

[zhangzujian]

There aren't logs kube-ovn-controller pods about this issue, only have logs for another pods

kube-ovn-controller is responsible for subnet management and pod IPAM. We need its logs to see what happened. You can also check the subnet's status to see whether it is ready for use.

[ade-iqbal]

My subnet is already for use

NAME              PROVIDER    VPC           PROTOCOL    CIDR              PRIVATE   NAT     DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE    V6USED    V6AVAILABLE   EXCLUDEIPS         U2OINTERCONNECTIONIP
my-subnet         ovn         ovn-cluster   IPv4        192.168.1.0/24    false     true    false     distributed   18       235            0         0             ["192.168.1.1"]

I try to get logs from kube-ovn-controller, but nothing found

vm@host:~$ kubectl logs -n kube-system -l app=kube-ovn-controller | grep my-pod-xxx
vm@host:~$

[zhangzujian]

Probably there are several kube-ovn-controller pods and the one selected by app=kube-ovn-controller is not working as leader. Please attach ALL the logs of the leader pod.

[ade-iqbal]

I have 3 pods of kube-ovn-controller and check one by one, but there are any logs found

vm@host:~$ kubectl get pod -n kube-system | grep kube-ovn-controler
kube-ovn-controller-xyz123abc-7cgcj    1/1     Running   0               7d
kube-ovn-controller-xyz123abc-nncgp    1/1     Running   0               7d
kube-ovn-controller-xyz123abc-zhgrp    1/1     Running   0               7d
vm@host:~$
vm@host:~$ kubectl logs -n kube-system kube-ovn-controller-xyz123abc-7cgcj | grep my-pod-xxx
vm@host:~$ kubectl logs -n kube-system kube-ovn-controller-xyz123abc-nncgp | grep my-pod-xxx
vm@host:~$ kubectl logs -n kube-system kube-ovn-controller-xyz123abc-zhgrp | grep my-pod-xxx

[ade-iqbal]

Probably there are several kube-ovn-controller pods and the one selected by app=kube-ovn-controller is not working as leader. Please attach ALL the logs of the leader pod.

About this issue, the pod gets an IP address after several minutes to hours. And now I get the same problem, but still there aren't logs on kube-ovn-controller about the pod

[oilbeater]

Can you run kubectl get lease -n kube-system kube-ovn-controller to see which pod hold the leader. I'm afraid something weird happened that some legacy pod hold the leader.

[ade-iqbal]

The leader exists, but there aren't logs about the pod in the leader controller

NAME                  HOLDER                                AGE
kube-ovn-controller   kube-ovn-controller-764f66f79-nncgp   14d