node-local-dns with Cilium LRP fails after ~10 minutes due to BPF backend programming issue #9448
Description
Environmental Info:
RKE2 Version:
v1.33.7+rke2r1
Node(s) CPU architecture, OS, and Version:
Linux prodmaster01 6.8.0-90-generic #100-Ubuntu SMP PREEMPT_DYNAMIC x86_64 GNU/Linux
Ubuntu 24.04.3 LTS
Cluster Configuration:
3 control-plane/etcd/master nodes, 4 worker nodes (7 total)
CNI: Cilium 1.18.4 with kubeProxyReplacement: true
Describe the bug:
When using node-local-dns with Cilium Local Redirect Policy (LRP) as documented, DNS resolution fails for all pods after approximately 10-15 minutes. The LRP successfully claims the frontend IP (10.43.0.10:53) but fails to program backends into the BPF map, causing all DNS queries to return "Operation not permitted".
This appears to be related to upstream Cilium bugs with skipRedirectFromBackend functionality (cilium/cilium#40450, cilium/cilium#36740).
Steps To Reproduce:
- Installed RKE2 v1.33.7+rke2r1 on Ubuntu 24.04.3 LTS with default configuration
- Create Cilium HelmChartConfig with LRP enabled:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-cilium
namespace: kube-system
spec:
valuesContent: |-
kubeProxyReplacement: true
k8sServiceHost: 127.0.0.1
k8sServicePort: 6443
localRedirectPolicy: true
- Create rke2-coredns HelmChartConfig with node-local-dns and Cilium LRP:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: rke2-coredns
namespace: kube-system
spec:
valuesContent: |-
nodelocal:
enabled: true
use_cilium_lrp: true
- Wait 10-15 minutes after all pods are running
- Test DNS resolution:
kubectl run dns-test --image=busybox --rm -it --restart=Never -- nslookup kubernetes.default
Expected behavior:
DNS resolution should work continuously. The node-local-dns pods should cache DNS queries locally using Cilium LRP to redirect traffic from the CoreDNS service IP (10.43.0.10) to the local node-local-dns pod.
Actual behavior:
DNS resolution fails with:
nslookup: can't connect to remote host (10.43.0.10): Operation not permitted
Cilium logs show:
level=error msg="LocalRedirectPolicy matches an address owned by an existing service => refusing to override" address=10.43.0.10:53/TCP serviceName=kube-system/rke2-coredns-rke2-coredns
level=warn msg="Failure processing services" error="frontend already owned by another service: 10.43.0.10:53/TCP is owned by kube-system/lrp-nodelocal:local-redirect"
Cilium service list shows LocalRedirect type:
51 10.43.0.10:53/TCP LocalRedirect
52 10.43.0.10:53/UDP LocalRedirect
But BPF LB map shows no backends (0.0.0.0:0 instead of actual pod IPs):
10.43.0.10:53/TCP (0) 0.0.0.0:0 (51) (0) [LocalRedirect]
10.43.0.10:53/UDP (0) 0.0.0.0:0 (52) (0) [LocalRedirect]
Additional context / logs:
Related Issues:
- [Release-1.33] - node-local-dns pod in Init state with Cilium CNI #9334 - Backport fix addresses init container issue but not this BPF programming bug
- node-local-dns pod in Init state with Cilium CNI #9056 - Original node-local-dns with Cilium issue
- CiliumLocalRedirectPolicy doesnt work with skipRedirectFromBackend cilium/cilium#40450 - LRP services disappear after ~10 minutes
- CiliumLocalRedirectPolicy skipRedirectFromBackend does not work as expected cilium/cilium#36740 - skipRedirectFromBackend doesn't work as expected
Workaround:
Disable LRP and node-local-dns:
rke2-cilium HelmChartConfig
localRedirectPolicy: false
rke2-coredns HelmChartConfig
nodelocal:
enabled: false
Note: According to RKE2 documentation, when using Cilium with kubeProxyReplacement: true, iptables-based node-local-dns (use_cilium_lrp: false) will not work because Cilium eBPF bypasses iptables. This means node-local-dns is currently unusable with Cilium in kube-proxy replacement mode due to these LRP bugs.
Recommendation:
The documentation should be updated to warn users that Cilium LRP with node-local-dns may be unstable in Cilium 1.17.x - 1.18.x. Consider adding a note to https://docs.rke2.io/networking/networking_services about this limitation.