After trying to join a node: dqlite: proxy: first: remote -> local: remote error: tls: bad certificate

[nomis-it]

Summary

I try to setup a Microk8s 1.32 Cluster on Rocky 9.5
When I try to join a node the syslog is full of the following message:
microk8s.daemon-k8s-dqlite[8430]: 2025/02/12 07:14:36 [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate

on both nodes and even the node join command tells that everything worked cluster I'm not getting ha up.
the microk8s status only shows the first node:

[root@kn5 ~]# microk8s status --wait-ready
microk8s is running
high-availability: no
datastore master nodes: 10.10.2.55:19001
datastore standby nodes: none
addons:
enabled:
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
disabled:

I run a 1.23 Cluster on Ubuntu 22.04 LTS just fine.

Selinux is disabled, also tried with disabled firewall and various versions: 1.30, 1.31, 1.32

the issues starts as soon as dqlite starts to listen on the public interface and not on loopback

I identified the following cert:

openssl x509 -in /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
34:5a:4c:8e:6f:40:5f:c4:41:18:c9:66:8e:2d:2b:25:ef:c3:b2:73
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=k8s
Validity
Not Before: Feb 12 06:51:08 2025 GMT
Not After : Feb 10 06:51:08 2035 GMT
Subject: CN=k8s
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (4096 bit)
Modulus:
00:b5:ae:10:bd:4c:1f:21:3b:02:e2:00:85:a3:51:
a5:b4:2e:03:c0:c2:d2:5e:42:f0:27:42:d8:b4:74:
00:d6:6c:c1:c6:11:89:7a:f2:c4:97:23:8f:90:1b:
00:4e:9f:1c:20:b0:01:07:dd:c6:fe:f5:ec:f9:61:
15:f6:94:1b:6e:59:da:5d:05:f2:78:00:ea:b3:26:
c1:ec:bb:d1:53:81:c5:b3:7c:9e:80:7d:a8:d2:e9:
d2:d4:e5:0e:1e:7d:6e:e5:67:90:14:d0:ba:0a:09:
76:a7:47:d2:23:59:0a:8a:37:80:a4:7c:18:28:77:
22:ce:f5:aa:99:05:31:a2:32:d0:88:2c:a4:24:06:
52:8d:04:d4:c3:20:8f:38:e0:b2:48:1f:2f:39:a0:
92:73:d4:51:3d:e2:2c:3b:14:db:85:18:97:ea:92:
c7:3f:cd:61:52:52:5e:8a:49:26:1f:7a:20:3c:97:
4f:a3:7b:72:8e:c6:93:04:87:73:6c:c8:b2:cf:19:
04:f4:4f:34:c6:7f:43:f6:1e:4f:77:cc:1e:ae:02:
fe:e2:7e:de:a4:3e:a6:89:fa:dd:f3:a9:50:da:47:
37:25:c5:b6:ba:75:07:15:0f:07:44:f3:ae:7c:a1:
b6:36:2d:db:bc:0a:68:c6:53:7b:0d:57:15:fd:00:
a2:40:79:3a:e3:34:db:79:90:be:38:b1:4f:58:ce:
62:9b:08:4d:7e:0e:35:5b:bd:3c:a1:53:a6:83:d1:
82:10:07:c2:72:bc:fe:30:48:14:aa:e2:b0:d6:74:
8e:88:74:23:bd:b1:c3:ef:a1:01:eb:fa:fe:14:32:
fb:9d:7b:d2:d6:f0:b8:5a:e6:69:af:ba:f2:0a:32:
dc:e0:0e:10:d4:5a:30:61:f4:bb:d5:37:3f:cb:36:
65:14:95:6b:99:40:7a:33:88:b1:b4:56:74:b9:2b:
1f:97:d2:d3:88:64:4b:e7:70:73:2b:0e:30:da:43:
2d:05:20:b5:f3:da:a5:98:b9:d5:aa:cd:b4:42:ec:
1d:42:03:96:b9:c7:6b:14:87:ac:0f:b3:04:37:3c:
73:43:39:0c:80:98:21:72:93:f6:e5:9c:48:d0:4e:
4a:7b:d2:e7:b1:1b:de:0d:1b:86:aa:3d:6c:1e:4d:
0b:20:d6:bb:7d:b5:d5:23:86:3c:be:a3:f0:57:6d:
e7:85:a6:61:b1:de:a4:8d:5c:24:67:21:83:d4:c4:
d4:ff:15:04:3e:33:b8:08:ee:c3:17:da:0a:d0:18:
23:08:ac:a3:ef:0b:22:82:a8:9e:33:91:dc:50:4d:
fd:ca:f1:7b:b6:ca:aa:48:68:dc:56:65:85:c5:1b:
7f:f4:a1
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Authority Key Identifier:
DirName:/CN=k8s
serial:34:5A:4C:8E:6F:40:5F:C4:41:18:C9:66:8E:2D:2B:25:EF:C3:B2:73
X509v3 Basic Constraints:
CA:FALSE
X509v3 Key Usage:
Digital Signature, Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Subject Alternative Name:
DNS:kn5, IP Address:127.0.0.1

which dqlite is using.

[nomis-it]

addition:
I also tried to create other cert with that csr file without success

[nomis-it]

also tested my cfg on ubuntu 24.04 LTS, and there everything is fine, looks like it has somehow to do with rocky specific stuff

[louiseschmidtgen]

Hi @nomis-it,

Thank you for reporting your issue with us.

Could you please generate an inspection report as described in these docs, this will help us look into your issue. In the meantime you could try setting the listen address of k8s-dqlite as described in these k8s-dqlite configuration docs.

For an HA cluster you will need at least three nodes, please find more information on high-availability in these docs.

Best regards,
Louise

[nomis-it]

dqlite listens to the correct interface after adding a node, but produces the error, what i found out:

it looks like it has to do with multi ip/interface setups.
while testing the same occured on an ubuntu setup.

i atteched the ubuntu inspect file

root@kn-06:~# microk8s inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/7665/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball
Report tarball is at /var/snap/microk8s/7665/inspection-report-20250217_054237.tar.gz

[nomis-it]

i also disabled all other interfaces and then try to install and i face the same issues.
my poc on a VM did btw work well, so im really confussed if i miss sth else

[louiseschmidtgen]

Hi @nomis-it,

Thank you for sending the inspection report.

can you please tell me how you created this cluster? Are you trying to use a specific network interface?

Best regards,
Louise

[nomis-it]

On one node i just run

microk8s add-node

and used the provided string to join cluster on other node. i tried over various interfaces with the same result.
i also tried to disable all other interfaces so that i only get one join command from the Designated Master Node with the same result.

[nomis-it]

okay, im feeling very awkward, after trying around its everything fine with microk8s. Looks like a Layer 8 issue...

I inspected the traffic and found out that another cluster is sending requests to the new clusters ip.
It looks like that the IPs used before as part of the cluster and where not cleanly removed there.

im so sorry and also thankful for your support

[HodeiG]

I also was seeing the same error log after joining a node and in my case I realized that the servers were out of sync. Once I fixed this issue, I had no issues. Essentially, check that systemctl status systemd-timesyncd doesn't show any NTP timeouts and update /etc/systemd/timesyncd.conf in all the nodes accordingly.

Perhaps this is documented somewhere, but if it's not it would be great to do so. Ideally, the join command should also detect that the nodes are not synced.