What happend:

Kubernetes cluster with switched L2 pod network. Each host has static /24 pod network and uses Calico IPAM in host-local mode.

Newly scheduled pod stuck in ContainerCreating. Logs indicate IP addresses are exhausted. Checking /var/lib/cni/networks/k8s-pod-network/, I see all IPs in range seem in used. Only roughly 20 IPs should be assigned though.

Checking multus logs, I see messages like

2025-10-11 15:25:28.892 [ERROR][38370] plugin.go 580: Final result of CNI DEL was an error. error=invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

This is response of the calico CNI plugin. Trying to further understand, I injected a small shell script tracing input/output of the CNI calls and some env variables. Then, I deleted a daemonset pod.

The delete output, observe the third line (input to the Calico CNI plugin) and the error message we already know:

CNI_ARGS=IgnoreUnknown=true;K8S_POD_NAMESPACE=backup;K8S_POD_NAME=node-agent-d97p8;K8S_POD_INFRA_CONTAINER_ID=a173930f520b46882cb2eb983ed593
bcaccb65e35a3e305c304aa1bb8f894fea;K8S_POD_UID=774f021f-d09a-4e62-ab2b-add3adb6ca82
CNI_COMMAND=DEL
{"cniVersion":"0.3.1","ipam":{"type":"host-local"},"name":"k8s-pod-network","type":"calico"}
{
    "code": 999,
    "msg": "invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable"
}

Checking an add output, we see multus passes the full output of /etc/cni/net.d/10-calico.conflist:

CNI_ARGS=IgnoreUnknown=true;K8S_POD_NAMESPACE=backup;K8S_POD_NAME=node-agent-d97p8;K8S_POD_INFRA_CONTAINER_ID=a173930f520b46882cb2eb983ed593
bcaccb65e35a3e305c304aa1bb8f894fea;K8S_POD_UID=774f021f-d09a-4e62-ab2b-add3adb6ca82
CNI_COMMAND=DEL
{"cniVersion":"0.3.1","datastore_type":"kubernetes","ipam":{"ranges":[[{"subnet":"usePodCidr"}]],"routes":[{"dst":"0.0.0.0/0"}],"type":"host-local"},"kubernetes":{"kubeconfig":"/etc/cni/net.d/calico-kubeconfig"},"log_file_path":"/var/log/calico/cni/cni.log","log_level":"info","mtu":1440,"name":"k8s-pod-network","nodename":"<redacted>","policy":{"type":"k8s"},"type":"calico"}
{
    "cniVersion": "0.3.1",
    "interfaces": [
        {
            "name": "cali109d890e582"
        }
    ],
    "ips": [
        {
            "version": "4",
            "address": "172.24.15.16/32"
        }
    ],
    "routes": [
        {
            "dst": "0.0.0.0/0"
        }
    ],
    "dns": {}
}

What you expected to happen:

Full content of /etc/cni/net.d/10-calico.conflist also passed to the calico CNI plugin for del requests, such that these run successfully and IPs are released.

How to reproduce it (as minimally and precisely as possible):

Trace delegated plugin calls.

Anything else we need to know?:

Checking the code, I see #1419 changed the behavior of the add call. Dropping to neither 4.1.4 nor 4.2.0 which should not have the new code did not resolve the issue though. Porting back the changes to the add function to the del function resolved the issue, see #1450.

I support #1433 -- which was hiding the actual error until the IP pool exhausted.

The cluster ran sucecssfully for some time with a Multus release 4.1.4 based on bitnami's charts. I cannot tell for sure whether this was stable, but must be assumed (the cluster runs a bunch of cronjobs). I assume the bitnami chart was deploying the "thin" version of multus?

Environment:

• Multus version 4.2.2
  image path and image ID (from 'docker images') v4.2.2-thick
• Kubernetes version (use kubectl version): ghcr.io/k8snetworkplumbingwg/multus-cni:v4.2.2-thick
• Primary CNI for Kubernetes cluster: Calico
• OS (e.g. from /etc/os-release): Ubuntu 24.04
• File of '/etc/cni/net.d/'
• File of '/etc/cni/multus/net.d'
• NetworkAttachment info (use kubectl get net-attach-def -o yaml)
• Target pod yaml info (with annotation, use kubectl get pod <podname> -o yaml)
• Other log outputs (if you use multus logging)