dqlite: Bad certificate

[tomoddershede]

I am doing a plain installation on 3 brand new 20.04 vms. (k8w1, k8w2, k8w3)

ufw is disabled.

Installation:
sudo snap install microk8s --classic

k8w2$ sudo microk8s join 192.168.xx.15:25000/xxxxxxxxxxxxxxxxxxxxxxxxxxxx
k8w2$ sudo microk8s join 192.168.xx.15:25000/xxxxxxxxxxxxxxxxxxxxxxxxxxxx

k8w1$ microk8s status
microk8s is running
high-availability: yes
datastore master nodes: 192.168.xx.15:19001 192.168.xx.16:19001 192.168.xx.17:19001
datastore standby nodes: none
addons:
enabled:
ha-cluster # Configure high availability on the current node
disabled:
ambassador # Ambassador API Gateway and Ingress
cilium # SDN, fast with full network policy
dashboard # The Kubernetes dashboard
dns # CoreDNS
fluentd # Elasticsearch-Fluentd-Kibana logging and monitoring
gpu # Automatic enablement of Nvidia CUDA
helm # Helm 2 - the package manager for Kubernetes
helm3 # Helm 3 - Kubernetes package manager
host-access # Allow Pods connecting to Host services smoothly
ingress # Ingress controller for external access
istio # Core Istio service mesh services
jaeger # Kubernetes Jaeger operator with its simple config
keda # Kubernetes-based Event Driven Autoscaling
knative # The Knative framework on Kubernetes.
kubeflow # Kubeflow for easy ML deployments
linkerd # Linkerd is a service mesh for Kubernetes and other frameworks
metallb # Loadbalancer for your Kubernetes cluster
metrics-server # K8s Metrics Server for API access to service metrics
multus # Multus CNI enables attaching multiple network interfaces to pods
portainer # Portainer UI for your Kubernetes cluster
prometheus # Prometheus operator for monitoring and logging
rbac # Role-Based Access Control for authorisation
registry # Private image registry exposed on localhost:32000
storage # Storage class; allocates storage from host directory
traefik # traefik Ingress controller for external access

snap info microk8s
...
snap-id: EaXqgt1lyCaxKaQCU349mlodBkDCXRcg
tracking: 1.20/stable
refresh-date: today at 19:28 UTC
...

After the third node is joined the HA-Status is enabled. But I experience that the /var/log/syslog then get full of the following log entries:

Jan 3 19:47:13 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:13.830882 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:15 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:15.221227 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:15 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:15.226065 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:16 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:16.604562 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:16 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:16.618007 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:17 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:17.735550 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:18 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:18.025754 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:18 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:18.035640 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:19 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:19.435944 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:19 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:19.449481 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:20 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:20.834551 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:20 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:20.847067 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate
Jan 3 19:47:21 k8w1 microk8s.daemon-apiserver[41678]: I0103 19:47:21.265046 41678 log.go:181] [ERROR] dqlite: proxy: first: remote -> local: remote error: tls: bad certificate

appreaciate your input

inspection-report-20210103_200737.tar.gz

[ktsakalozos]

Hi @tomoddershede

Could you help me reproduce the error? Can you describe in more detail how the cluster is formed? You need to call microk8s add-node on k8w1 then microk8s join on k8w2 then microk8s add-node again on k8w1 and finally microk8s join on k8w3. Just to be sure you need to allow time for each node to register.

The error you see comes from [1] indicating that a request to the data store could not be replicated to a node.

[1] https://github.com/canonical/go-dqlite/blob/88e5f725af8b2dd82d353d82e28fa78deb0be8ad/app/proxy.go#L14

[tomoddershede]

@ktsakalozos

I am doing exact that (execp that I use the same join command)

#microk8s add-node -l 3600 (on k8w1 to get the the join command)
I wait 4 minutes and checks that there is no nodes that is in NotReady state (using kubectl get nodes)

Then I run the join command on k8w2

I wait 4 minutes and checks that there is no nodes that is in NotReady state (using kubectl get nodes)
Then I run the join command k8w3

---

update:

I now tried to do the add-node once for each node (exactly as you described), and I still get the same error

[ktsakalozos]

There could be a chance the failing nodes had registered in the past and now they are trying to re-join with a different certificate? Could you share the output of cat /var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml ?

We may want to run a microk8s.remove-node k8w2 --force and microk8s.remove-node k8w3 --force to clean-up anything that is left over from previous join attempts.

[tomoddershede]

It seems something changed. Now doing exactly the same thing (ansible), a week after, it does not give me the error anymore. Thank you for your help. Case can be closed.

[halilbulentorhon]

I have the same problem here is the related question:
stackoverflow

[HodeiG]

I had exactly the same issue and I realized that the servers were out of sync. Once I fixed this issue, I had no issues. Check that systemctl status systemd-timesyncd doesn't show any timeouts and update /etc/systemd/timesyncd.conf in all the nodes accordingly.

Perhaps this is documented somewhere, but if it's not it would be great to do so. Ideally, the join command should also detect that the nodes are not synced.