kubelet certificate expired, renewing the certificate by deleting it and restarting microk8s results in an error

[davidnich]

Summary

Related to: #2489
Partially duplicates: #4561

Kubelet certificates are created for a duration of one year, after which they must be renewed. Bugs related to this are:

• lack of documentation of how to renew kubelet certificates
• error regerating kubelet certificate with miucrok8s 1.34.1

With invalid kubelet certificates, retrieving pod logs running on the node fails

• ex before renewing:: Error from server: Get "https://192.168.x.x:10250/containerLogs/xxx/xxx/xxx": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2025-11-10T07:52:51+01:00 is after 2024-08-11T16:25:06Z
• ex after renewing: Error from server: Get "https://192.168.x.x:10250/containerLogs/xxx/xxx/xxx": tls: failed to verify certificate: x509: cannot validate certificate for 192.168.x.x because it doesn't contain any IP SANs

See also: kubernetes/kubernetes#62939

What Should Happen Instead?

Ideally microk8s should be set up to automatically renew certs. microk8s refresh-certs should support the kubelet cert. The kubelet cert creation should add the IP SAN

Reproduction Steps

I regenerated the kubelet certificate using info from the referenced microk8s issue above - ex:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data 
microk8s stop 
sudo rm /var/snap/microk8s/current/certs/kubelet.* 
microk8s start 
kubectl uncordon <node> 

Note that according to #4561 the certificate generation issues may be related to nodes added with earlier versions of microk8s

Introspection Report

inspection-report-20251110_083602.tar.gz

Can you suggest a fix?

To address these issues:

• microk8s should automatically renew kubelet certificates or at least add this option to microk8s refresh-certs
• the documentation should be updated to cover kubelet certs alongside the other certs (https://microk8s.io/docs/command-reference#heading--microk8s-refresh-certs)
• the regenerating the kubelet certicates with the correct CA and including the IP SAN should be resolved

Are you interested in contributing with a fix?

Happy to help if I can

[davidnich]

Note that the certificate generation issue was first reported here: #4561

The workaround that worked for me regenerating the cert is documented there:

sudo microk8s stop

sudo openssl genrsa -out /var/snap/microk8s/current/certs/kubelet.key 2048

sudo openssl req -new \
    -key /var/snap/microk8s/current/certs/kubelet.key \
    -out /var/snap/microk8s/current/certs/kubelet.csr \
    -config /var/snap/microk8s/current/certs/csr.conf

sudo openssl x509 -req \
    -in /var/snap/microk8s/current/certs/kubelet.csr \
    -CA /var/snap/microk8s/current/certs/ca.crt \
    -CAkey /var/snap/microk8s/current/certs/ca.key \
    -CAcreateserial \
    -out /var/snap/microk8s/current/certs/kubelet.crt \
    -days 365 \
    -extensions req_ext \
    -extfile /var/snap/microk8s/current/certs/csr.conf

sudo microk8s start