Skip to content

Script to renew cluster certificates #9709

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
---
title: "Manually renew cluster certificates"
linkTitle: "Script To Renew Certificates"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am curious if these instructions should just be in the README of the repo and we can just refer the readme in the official docs. But I would defer to docs writer on that

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed pattern of updating vsphere credentials but I am okay with either options.
@chrisnegus please let us know your thoughts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with having it here. But I don't have a strong opinion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
linkTitle: "Script To Renew Certificates"
linkTitle: "Script to renew certificates"

description: >
Step-by-step guide to renew Kubernetes certificates on EKS Anywhere clusters using a script.
weight: 20
---

Certificates for external etcd and control plane nodes expire after 1 year in EKS Anywhere. EKS Anywhere automatically rotates these certificates when new machines are rolled out in the cluster. New machines are rolled out during cluster lifecycle operations such as `upgrade`. If you upgrade your cluster at least once a year, as recommended, manual rotation of cluster certificates will not be necessary.

This page shows the process for manually rotating certificates if you have not upgraded your cluster in 1 year.

The following table lists the cluster certificate files:

| etcd node | control plane node |
|-----------------------|--------------------------|
| apiserver-etcd-client | apiserver-etcd-client |
| ca | ca |
| etcdctl-etcd-client | front-proxy-ca |
| peer | sa |
| server | etcd/ca.crt |
| | apiserver-kubelet-client |
| | apiserver |
| | front-proxy-client |

Commands below can be used for quickly checking your certificates' expiration dates:

```bash
# The expiry time of api-server certificate on you cp node
echo | openssl s_client -connect ${CONTROL_PLANE_IP}:6443 2>/dev/null | openssl x509 -noout -dates

# The expiry time of certificate used by your external etcd server, if you configured one
echo | openssl s_client -connect ${EXTERNAL_ETCD_IP}:2379 2>/dev/null | openssl x509 -noout -dates
```

{{% alert title="Note" color="primary" %}}
You can rotate certificates by following the steps given below. You cannot rotate the `ca` certificate because it is the root certificate. Note that the commands used for Bottlerocket nodes are different than those for Ubuntu and RHEL nodes.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this a Note or something

{{% /alert %}}

## Overview
This script automates:

- Certificate renewal for etcd and control plane nodes
- Cleanup of temporary files if certificates are renewed and cluster is healthy

## Prerequisites

- Admin machine with:
- `kubectl`, `yq`, `jq`, `scp`, `ssh`, and `sudo` installed
- SSH access to all control plane and etcd nodes

## Steps

### 1. Setup environment variable:

```bash
export KUBECONFIG=<path-to-management-cluster-kubeconfig>
```

### 2. Prepare below config yaml file by adding node and private key information of your control plane and/or external etcd to a file, such as `keys-config.yaml`:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### 2. Prepare below config yaml file by adding node and private key information of your control plane and/or external etcd to a file, such as `keys-config.yaml`:
### 2. Prepare a keys-config.yaml file
Add node and private key information of your control plane and/or external etcd to a file, such as `keys-config.yaml`:

I was thinking of something like this.


```bash
clusterName: <cluster-name>
controlPlane:
nodes:
- <control-plane-1-ip>
- <control-plane-2-ip>
- <control-plane-3-ip>
sshKey: <complete-path-to-private-ssh-key>
sshUser: <ssh-user>
etcd:
nodes:
- <external-etcd-1-ip>
- <external-etcd-2-ip>
- <external-etcd-3-ip>
sshKey: <complete-path-to-private-ssh-key>
sshUser: <ssh-user>
```

### 3. Download the Script

{{< tabpane >}}
{{< tab header="Ubuntu or RHEL" lang="bash" >}}
```bash
curl -O https://raw.githubusercontent.com/aws/eks-anywhere/refs/heads/main/scripts/renew_certificates.sh
chmod +x renew_certificates.sh
```
{{< /tab >}}
{{< tab header="Bottlerocket" lang="bash" >}}
```bash
curl -O https://raw.githubusercontent.com/aws/eks-anywhere/refs/heads/main/scripts/renew_certificates_bottlerocket.sh
chmod +x renew_certificates_bottlerocket.sh
```
{{< /tab >}}
{{< /tabpane >}}

### 4. Run the Script

```bash
./renew_certificates.sh -f keys-config.yaml
```


### What the Script Does

- Backs up:
- All etcd certificates (in case of external etcd)
- Control plane certificates
- Renews external etcd certificates
- Updates the Kubernetes secret apiserver-etcd-client
- Renews all kubeadm certificates
- Restart static control plane pods
- Cleans up temporary certs and backup folders (only if certificates are renewed successfully and cluster is healthy)
4 changes: 2 additions & 2 deletions docs/content/en/docs/troubleshooting/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -332,7 +332,7 @@ ${IMAGE_ID} tmp-certs-check \
{{< /tab >}}
{{< /tabpane >}}

EKS Anywhere typically renews certificates when upgrading a cluster. However, if a cluster has not been upgraded for over a year, then it is necessary to manually renew these certificates. Please see [Certificate rotation]({{< relref "../clustermgmt/security/manually-renew-certs.md" >}}) to manually rotate expired certificates.
EKS Anywhere typically renews certificates when upgrading a cluster. However, if a cluster has not been upgraded for over a year, then it is necessary to manually renew these certificates. Please see [Certificate rotation]({{< relref "../clustermgmt/security/certificate-management/manually-renew-certs.md" >}}) to manually rotate expired certificates.

### Bootstrap cluster fails to come up

Expand Down Expand Up @@ -630,7 +630,7 @@ kubectl get nodes --no-headers -l '!node-role.kubernetes.io/control-plane' -o js
```

### No static pods is running in container runtime on control plane nodes
When the kubelet systemd service is running on a control plane node and you notice that no [static pods](https://kubernetes.io/docs/concepts/workloads/pods/#static-pods) are running, this could be due to a certificate issue. Check the kubelet service logs. If you see error messages like the ones below, renew the kubelet-client-current.pem certificate by following [these steps]({{< relref "../clustermgmt/security/manually-renew-certs/#kubelet" >}}):
When the kubelet systemd service is running on a control plane node and you notice that no [static pods](https://kubernetes.io/docs/concepts/workloads/pods/#static-pods) are running, this could be due to a certificate issue. Check the kubelet service logs. If you see error messages like the ones below, renew the kubelet-client-current.pem certificate by following [these steps]({{< relref "../clustermgmt/security/certificate-management/manually-renew-certs/#kubelet" >}}):
```
part of the existing bootstrap client certificate in /etc/kubernetes/kubelet/kubeconfig
Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem
Expand Down
Loading
Loading