kubelet.crt doesn't contain any IP SANs when the cluster is upgraded

[mshlain]

Summary

kubelet.crt doesn't contain any IP SANs when the cluster is upgraded from 1.27.8 to 1.28.7 to 1.30

When the cluster is clean installed with 1.30, the kubelet.crt contains the IP SANs.

What Should Happen Instead?

Same as clean install, the kubelet.crt should contain the IP SANs on upgraded clusters.

Reproduction Steps

I have few microk8s clusters.

One fresh cluster 'A' that was recently installed with 1.30.

When installed metrics-server helmchart ont cluster 'A', everything worked fine.

Then moved to cluster 'B' that was installed as 1.27.8 then upgraded to 1.28.7 and finally to 1.30.

When tried to install metrics-server helmchart on cluster 'B', it fails with the following error:

```
1 scraper.go:149] \"Failed to scrape node\" err=\"Get \\\"https://10.171.16.216:10250/metrics/resource\\\": tls: failed to verify certificate: x509: cannot validate certificate for 10.171.16.216 because it doesn't contain any IP SANs\" 
```

Indeed when I checked the kubelet.crt on cluster 'B', it doesn't contain any IP SANs.

In contrast, the kubelet.crt on cluster 'A' contains the IP SANs.

Tried to reset certs on cluster 'B' but it didn't help.

What helped was to regenerate the certificate by open ssl:

sudo microk8s stop

sudo openssl genrsa -out /var/snap/microk8s/current/certs/kubelet.key 2048

sudo openssl req -new \
    -key /var/snap/microk8s/current/certs/kubelet.key \
    -out /var/snap/microk8s/current/certs/kubelet.csr \
    -config /var/snap/microk8s/current/certs/csr.conf

sudo openssl x509 -req \
    -in /var/snap/microk8s/current/certs/kubelet.csr \
    -CA /var/snap/microk8s/current/certs/ca.crt \
    -CAkey /var/snap/microk8s/current/certs/ca.key \
    -CAcreateserial \
    -out /var/snap/microk8s/current/certs/kubelet.crt \
    -days 365 \
    -extensions req_ext \
    -extfile /var/snap/microk8s/current/certs/csr.conf

sudo microk8s start

Introspection Report

bad_cert.txt
bad_inspection-report-20240703_133230.tar.gz

good_cert.txt
good_inspection-report-20240703_150340.tar.gz

[sbidoul]

I'm noticing this too, on nodes that joined with older versions of microk8s before the cluster was upgraded.

Doing a microk8s leave + join resolves the issue on worker nodes.

On some control plane nodes, kubelet.crt is expired.

Is there an official method to regenerate invalid or expired kubelet.crt?

[macintoshme]

I was heavily-handed playing with my lab cluster and found that the kubelet cert didn't seem to be involved in any of the cert refreshes, unlike other certs.

I eventually removed it and it was successfully regenerated, but this time, like the issue says, without any IP SANs which was problematic.

I really appreciate the explicit directions @mshlain, you saved me at least 30min of research.