tls: failed to verify certificate 509 after enabling host-access 1.29/stable

[cancerian0684]

Summary

Failed to check container logs when host-access is enabled on microk8s 1.29/stable on ubuntu 22 due to below error

Error from server: Get "https://10.0.1.1:10250/containerLogs/container-registry/registry-6c9fcc695f-bkrbl/registry?follow=true": tls: failed to verify certificate: x509: certificate is valid for 192.168.1.14 not 10.0.1.1

Process finished with exit code 1

What Should Happen Instead?

I should be able to check logs. The same thing works on microk8s snap 1.27/stable channel.

Reproduction Steps

1. Install microk8s snap 1.29/stable on ubuntu 22
2. Enable host-access (10.0.1.1)
3. Check logs for any pods, command will throw an exception

Error from server: Get "https://10.0.1.1:10250/containerLogs/container-registry/registry-6c9fcc695f-bkrbl/registry?follow=true": tls: failed to verify certificate: x509: certificate is valid for 192.168.1.14, 172.29.0.1, 172.18.0.1, 172.20.0.1, 172.21.0.1, 172.17.0.1, 2401:4900:1c71:9f73:1bb0:83dd:e889:c0be, 2401:4900:1c71:9f73:911e:b5e7:aaff:782e, not 10.0.1.1

Process finished with exit code 1

Introspection Report

Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy asnycio usage and limits to the final report tarball
Copy inotify max_user_instances and max_user_watches to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/6364/var/kubernetes/backend/localnode.yaml': No such file or directory

WARNING: Maximum number of inotify user watches is less than the recommended value of 1048576.
Increase the limit with:
echo fs.inotify.max_user_watches=1048576 | sudo tee -a /etc/sysctl.conf
sudo sysctl --system
Building the report tarball
Report tarball is at /var/snap/microk8s/6364/inspection-report-20231217_102515.tar.gz

Can you suggest a fix?

Are you interested in contributing with a fix?

[neoaggelos]

Hi @cancerian0684

Looks like the kubelet picks up 10.0.1.1 as the node address instead. Can you check the following:

microk8s kubectl get node -o wide

If that is the case, you could specify the IP by doing:

echo '
--node-ip=192.168.1.14
' | sudo tee -a /var/snap/microk8s/current/args/kubelet
sudo snap restart microk8s.daemon-kubelite

I am not able to reproduce the issue locally, so it looks like kubelet sometimes picks up the wrong address?

[ajaydevtron]

Hello Team,

Do you have update on this issue ?

I am also facing the same for this microk8s version.

[SphtKr]

Having this issue also. Fresh install of 22.04 + microk8s 1.29/stable from snap... here are the odd bits: removing the microk8s snap and reinstalling it resulted in the same problem before installing the host-access addon, but notably I did not explicitly disable the add-on before removing the snap--therefore I am inferring this is doing something somewhere to the host networking stack? Further, after I figured out what was going on, I disabled host-access, removed 1.29/stable, and then installed 1.28/stable... now when I enable host-access it is doing the same thing on 1.28/stable.

If this is helpful:

user@myhost:~$ microk8s kubectl get node -o wide
NAME       STATUS   ROLES    AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
myhost     Ready    <none>   174m   v1.28.3   10.0.2.1      <none>        Ubuntu 22.04.3 LTS   5.15.0-92-generic   containerd://1.6.15

EDIT: should have mentioned this was a after sudo microk8s enable host-access:ip=10.0.2.1

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[torgeirl]

Reproduction Steps

1. Install microk8s snap 1.29/stable on ubuntu 22
2. Enable host-access (10.0.1.1)
3. Check logs for any pods, command will throw an exception

This is still an issue with v.1.32 on Ubuntu 24.04.

[torgeirl]

For anyone else looking, here is a workaround (from #4477) that at least allows you to use run microk8s.kubectl logs:

$ sudo microk8s.kubectl logs --insecure-skip-tls-verify-backend=true podname

[sabpamdev1]

Still an issue and getting by with workaround.

[SphtKr]

Anyone stumbling on this, check out my additional workaround in canonical/microk8s-core-addons/issues/362