Complete Cilium configuration annotations not supported

[vishu2498]

Summary

According to the documentation, there are very specifc annotations which are supported by k8s configuration to customize different parameters of metrics-server, cilium etc. The same can be configured from CAPI CK8s too.

However, these annotations aren't enough for cilium configuration when a machine (specifically MAAS machine) being used has different networking configurations. Currently, these cilium configurations work from annotations:

k8sd/v1alpha1/cilium/cni-exclusive
k8sd/v1alpha1/cilium/devices
k8sd/v1alpha1/cilium/direct-routing-device
k8sd/v1alpha1/cilium/vlan-bpf-bypass
k8sd/v1alpha1/cilium/tunnel-port

The cilium daemonset uses a cilium-config configmap for all of its configurations. Checking through the configmap, it has a variety of options available which can't be defined through annotations.

In our scenario, we have a MAAS machine that has a br0 interface. Directly providing direct-routing-device and devices didn't work and cilium went to CrashLoopBackOff state with error:

time="2025-08-12T08:08:06Z" level=fatal msg="failed to start: daemon creation failed: failed to detect devices: unable to determine direct routing device. Use --direct-routing-device to specify it\nfailed to stop: unable to find controller ipcache-inject-labels" subsys=daemon

To make it work, we had to adjust some more parameters in the configmap and restart the daemonset with these options:

"routing-mode": "native"
"ipv4-native-routing-cidr": "192.168.0.0/16"
"auto-direct-node-routes": "true"

These options aren't available via annotations from cluster-config section stored in /capi/etc/config.yaml . This results in cluster being stuck in creation state while waiting for cilium to get up and running.

What Should Happen Instead?

Since k8s directly provides an option to set annotations in cluster-config section, user should be able to use all of the cilium configurations from annotations instead of just the options present in the documentation.

This provides a granular apporach towards creating a cluster from k8s where the setup of machine has different network configurations.

Reproduction Steps

(example values are provided for explaination)

1. Have a MAAS machine with br0 interface as its default networking interface.
2. Create a config file with annotations in cluster-config section:

cluster-config:
  annotations:
    k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: "true"
    k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: "true"
    k8sd/v1alpha1/cilium/auto-direct-node-routes: "true"
    k8sd/v1alpha1/cilium/devices: "br0,enp2s0f0"
    k8sd/v1alpha1/cilium/direct-routing-device: "br0"
    k8sd/v1alpha1/cilium/routing-mode: native
    k8sd/v1alpha1/cilium/ipv4-native-routing-cidr: "192.168.0.0/16"
    k8sd/v1alpha1/cilium/enable-ip-masq-agent": "false"

2. Create a cluster from k8s bootstrap --name "${name}" --file "${config_file}"
3. When cluster bootstrapping starts, connect to the cluster and observe the cilium pod. It would go into CrashLoopBackOff state.
4. Check the logs of cilium pod. It shows the configurations at the start. Also check the cilium-config configmap.
5. Compare the values set in the annotations sections with the ones you see in cluster. The annotations not present in the document don't get configured in the cilium's setup.

System information

A MAAS machine with br0 interface as its default networking interface.

Can you suggest a fix?

The annotations section of cluster-config should be able to receive all the parameters which are present in cilium-config configmap for customized setup.

Are you interested in contributing with a fix?

No response

[rapour]

Hi @vishu2498

Currently a limited subset of configuration options are exposed via annotations so adding arbitrary keys like k8sd/v1alpha/lifecycle/skip-stop-services-on-remove won't work. But this shouldn't prevent you from deploying your cluster over a specific network. Could you please bootstrap your cluster using the node IP address attached to br0 and let me know if that solves the issue?

sudo k8s bootstrap --name "${name}" --file "${config_file}" --address=<NODE_IP_ATTACHED_TO_BR0>