Cilium bug in v1.32.2+rke2r1

[xhejtman]

Hello,

I tried minimal setup of RKE v1.32.2+rke2r1 with Cilium CNI. I customized Cilium config as described here:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    annotateK8sNode: true
    kubeProxyReplacement: true
    k8sServiceHost: "localhost"
    k8sServicePort: "6443"

have this configuration of rke2:

server: https://storage-elixir2:9345
token:  xxx

tls-san:
  - storage-elixir2a
  - storage-elixir2

tls-san-security: true

disable:
  - rke2-canal

disable-kube-proxy: true

cni: cilium

cluster-cidr: 10.42.0.0/16,xxx:9:2::/96

service-cidr: 10.43.0.0/16,xxx:9:3::/112

kube-controller-manager-arg:
  - "feature-gates=InPlacePodVerticalScaling=true"
  - "node-cidr-mask-size-ipv6=108"

audit-policy-file: "/etc/rancher/rke2/audit-policy.yaml"

kube-apiserver-arg:
  - "feature-gates=InPlacePodVerticalScaling=true"
  - "audit-log-maxage=30"
  - "audit-log-maxbackup=10"
  - "audit-log-path=/var/log/kube-audit/audit.log"
  - "audit-log-maxsize=100"
  - "audit-log-format=json"
  - "audit-policy-file=/etc/rancher/rke2/audit-policy.yaml"

profile: cis

kube-scheduler-arg:
  - "feature-gates=InPlacePodVerticalScaling=true"

kubelet-arg:
  - "v=5"
  - "feature-gates=InPlacePodVerticalScaling=true"
  - "node-ip=10.16.48.2,xxx:0:1:48:2"
  - "kube-api-qps=100"
  - "kube-api-burst=100"

node-ip: '10.16.48.2,xxx:0:1:48:2'

control-plane-resource-requests: "kube-apiserver-memory=3G,etcd-memory=32G"

pod-security-admission-config-file: /etc/rancher/rke2/rke2-pss-cerit.yaml

it can start on two servers, however, the third is unable to join, because the peer etcd is not reachable.

I can see that the etcd is running on the second server:

tcp        0      0 10.16.48.3:2379         0.0.0.0:*               LISTEN      11745/etcd

10.16.48.3 == storage-elixir2b

It also accepts connections:

[root@storage-elixir2b ~]# telnet 10.16.48.3 2380
Trying 10.16.48.3...
Connected to 10.16.48.3.
Escape character is '^]'.
^CConnection closed by foreign host.

But I cannot connect to it from the first (or third) server:

[root@storage-elixir2a ~]# telnet 10.16.48.3 2380
Trying 10.16.48.3...
telnet: connect to address 10.16.48.3: No route to host
[root@storage-elixir2a ~]# telnet 10.16.48.3 2380
Trying 10.16.48.3...
^C
[root@storage-elixir2a ~]#

Ping or ssh is indeed working between the servers.

[root@storage-elixir2a ~]# telnet 10.16.48.3 22
Trying 10.16.48.3...
Connected to 10.16.48.3.
Escape character is '^]'.
SSH-2.0-OpenSSH_8.7
^]
telnet>

Is that so, that cilium is broken in this version or some fundamental part of configuration is missing?

I have similar issue with connecting to 10.43.0.1:443 from a pod (kubeapi server).

[root@storage-elixir2a ~]# kubectl get pods -A -o wide
NAMESPACE     NAME                                                         READY   STATUS      RESTARTS   AGE   IP            NODE
kube-system   cilium-hgz9j                                                 1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   cilium-operator-658cc7f59-gk9gs                              1/1     Running     0          15m   10.16.48.3    storage-elixir2b
kube-system   cilium-operator-658cc7f59-qn5l8                              1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   cilium-vzd9c                                                 1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   cloud-controller-manager-storage-elixir2a                    1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   cloud-controller-manager-storage-elixir2b                    1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   etcd-storage-elixir2a                                        1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   etcd-storage-elixir2b                                        1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   helm-install-rke2-cilium-ndmjg                               0/1     Completed   0          15m   10.16.48.2    storage-elixir2a
kube-system   helm-install-rke2-coredns-6wpcj                              0/1     Completed   0          15m   10.16.48.2    storage-elixir2a
kube-system   helm-install-rke2-ingress-nginx-v8d7d                        0/1     Completed   0          15m   <none>        storage-elixir2a
kube-system   helm-install-rke2-metrics-server-vwrxm                       0/1     Completed   0          15m   <none>        storage-elixir2a
kube-system   helm-install-rke2-runtimeclasses-4tqlg                       0/1     Completed   0          15m   <none>        storage-elixir2a
kube-system   helm-install-rke2-snapshot-controller-b5bq8                  0/1     Completed   1          15m   <none>        storage-elixir2a
kube-system   helm-install-rke2-snapshot-controller-crd-7gj6x              0/1     Completed   0          15m   <none>        storage-elixir2a
kube-system   kube-apiserver-storage-elixir2a                              1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   kube-apiserver-storage-elixir2b                              1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   kube-controller-manager-storage-elixir2a                     1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   kube-controller-manager-storage-elixir2b                     1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   kube-scheduler-storage-elixir2a                              1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   kube-scheduler-storage-elixir2b                              1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   kube-vip-ds-4vbj6                                            1/1     Running     0          15m   10.16.48.2    storage-elixir2a
kube-system   kube-vip-ds-pbwmr                                            1/1     Running     0          10m   10.16.48.3    storage-elixir2b
kube-system   rke2-coredns-rke2-coredns-7895c6f5d8-9cp5q                   1/1     Running     0          15m   10.42.0.8     storage-elixir2a
kube-system   rke2-coredns-rke2-coredns-7895c6f5d8-bqxwc                   1/1     Running     0          10m   10.42.1.179   storage-elixir2b
kube-system   rke2-coredns-rke2-coredns-autoscaler-5868d76f68-299m5        1/1     Running     0          15m   10.42.0.192   storage-elixir2a
kube-system   rke2-ingress-nginx-controller-nwbjw                          1/1     Running     0          10m   10.42.1.12    storage-elixir2b
kube-system   rke2-ingress-nginx-controller-x7hrb                          1/1     Running     0          14m   10.42.0.76    storage-elixir2a
kube-system   rke2-metrics-server-85479b695c-zjjsz                         1/1     Running     0          15m   10.42.0.72    storage-elixir2a
kube-system   rke2-snapshot-controller-696989ffdd-gw2h7                    1/1     Running     0          14m   10.42.0.30    storage-elixir2a

[brandond]

Are you sure that you've opened the correct ports between nodes? Cilium would not be involved in either providing or interfering with connectivity between nodes. It only provides the cluster network overlay.

[xhejtman]

ah, you are correct, sorry. There was nft firewalld artefact :(