Document requirements/recommended process for updating cluster TLS certs/keys

[jimmycuadra]

If you're running Kubernetes with the master components secured with TLS, eventually you will need to update the certificate and key, and possibly even the CA cert. Right now there is no documentation about how this should be approached. What services need to be restarted when the CA cert, endpoint cert, or private key are changed on disk? If all the master components are running via the kubelet's static manifest directory, is it sufficient to just restart kubelet on the host? Or is it necessary to somehow manually restart each containerized master component that reads those files?

[pwittrock]

@kelseyhightower Do you know how folks normally figure this out?

[jimmycuadra]

I just had to go through this process more or less manually, and learned that restarting the kubelet does not restart any k8s master components that were launched by static manifests the kubelet is observing. In order to get kube-apiserver to restart and pickup the new TLS credentials, I had to move the kube-apiserver manifest out of the directory kubelet was watching, restart kubelet, then move the manifest back in and restart the kubelet again.

This definitely needs to be documented. Hopefully there is a better way of telling the kubelet to restart the master components, too. If not, there really should be.

[jimmycuadra]

Would a maintainer please bring a Kubernetes developer who can answer this into the conversation? Not having a way to restart the kube-system components to pick up TLS credential changes is a blocking issue for my team to roll out Kubernetes in production. Thanks!

[jimmycuadra]

Comment from chancez on Slack which helps with a workaround for the time being:

ive only tried once/twice to test a solution for an issue a user was having, but i basically did what you did, but I did docker kill $container instead of messing with the manifest files

[eugene-chow]

I just had to re-make my cert and deploy it to all the nodes. These are my notes:

1. Create the new certs and deploy it to all the nodes
2. Restart each and every node one-by-one. Alternatively, restart every kubernetes service if you don't want to bring down the node completely.
3. Delete the default service account token, which is linked to the old TLS certs, in every namespace (eg. default-token-b7scp). A new default token will be automatically created.
4. Restart every pod that uses the default token, so that it reads the new token.

[liggitt]

steps 3 and 4 are not required if you keep the old key as an additional valid public key (can pass multiple --service-account-key-file args to the apiserver), or use a dedicated service account token signing key separate from the apiserver's private tls key

[eugene-chow]

Thanks for the heads up. I forgot to mention that my setup isn't a production system but Kelsey's https://github.com/kelseyhightower/kubernetes-the-hard-way. The cluster in this tutorial uses one cert to rule them all.

[xiangpengzhao]

/sig docs

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[ankon]

Clearly this is a still-existing problem, and it is really something that needs to be addressed minimally in documentation, ideally even in code.

[emilhdiaz]

This is major blocker for us to run Kubernetes into production. Any advice would be much appreciated!

I already tried to simulate a certificate rotation in a test environment and couldn't do so without causing downtime for running applications.

/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[ankon]

/remove-lifecycle rotten