Skip to content

Commit 5c55e00

Browse files
lahinsonlahinson
committed
[OSDOCS-12351]: Dividing etcd backup assembly
1 parent 7bb886f commit 5c55e00

31 files changed

+225
-168
lines changed

‎_topic_maps/_topic_map.yml

+8-1
Original file line numberDiff line numberDiff line change
@@ -2520,7 +2520,14 @@ Topics:
25202520
- Name: Performance considerations for etcd
25212521
File: etcd-performance
25222522
- Name: Backing up and restoring etcd data
2523-
File: etcd-backup
2523+
Dir: etcd-backup-restore
2524+
Topics:
2525+
- Name: Backing up etcd
2526+
File: etcd-backup
2527+
- Name: Replacing an unhealthy etcd member
2528+
File: replace-unhealthy-etcd-member
2529+
- Name: Disaster recovery
2530+
File: etcd-disaster-recovery
25242531
- Name: Encrypting etcd data
25252532
File: etcd-encrypt
25262533
- Name: Setting up fault-tolerant control planes that span data centers

‎backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following file: backup_and_restore/control_plane_backup_and_restore/etcd-backup.adoc.
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="backup-etcd"]
37
= Backing up etcd

‎backup_and_restore/control_plane_backup_and_restore/disaster_recovery/about-disaster-recovery.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: etcd/etcd-backup-restore/etcd-disaster-recovery.adoc.
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="about-dr"]
37
= About disaster recovery

‎backup_and_restore/control_plane_backup_and_restore/disaster_recovery/quorum-restoration.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: etcd/etcd-backup-restore/etcd-disaster-recovery.adoc.
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="dr-quorum-restoration"]
37
= Quorum restoration

‎backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: etcd/etcd-backup-restore/etcd-disaster-recovery.adoc.
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="dr-restoring-cluster-state"]
37
= Restoring to a previous cluster state

‎backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-3-expired-certs.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: etcd/etcd-backup-restore/etcd-disaster-recovery.adoc.
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="dr-recovering-expired-certs"]
37
= Recovering from expired control plane certificates

‎backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.adoc

+4
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following file: etcd/etcd-backup-and-restore/replace-unhealthy-etcd-member.adoc
4+
15
:_mod-docs-content-type: ASSEMBLY
26
[id="replacing-unhealthy-etcd-member"]
37
= Replacing an unhealthy etcd member

‎etcd/etcd-backup-restore/_attributes

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../_attributes/

‎etcd/etcd-backup-restore/etcd-backup.adoc

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following file: backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.adoc.
4+
5+
:_mod-docs-content-type: ASSEMBLY
6+
[id="etcd-backup"]
7+
include::_attributes/common-attributes.adoc[]
8+
= Backing up and restoring etcd data
9+
:context: etcd-backup
10+
11+
toc::[]
12+
13+
As the key-value store for {product-title}, etcd persists the state of all resource objects.
14+
15+
Back up the etcd data for your cluster regularly and store it in a secure location, ideally outside the {product-title} environment. Do not take an etcd backup before the first certificate rotation completes, which occurs 24 hours after installation, otherwise the backup will contain expired certificates. It is also recommended to take etcd backups during non-peak usage hours because the etcd snapshot has a high I/O cost.
16+
17+
Be sure to take an etcd backup before you update your cluster. Taking a backup before you update is important because when you restore your cluster, you must use an etcd backup that was taken from the same z-stream release. For example, an {product-title} 4.17.5 cluster must use an etcd backup that was taken from 4.17.5.
18+
19+
[IMPORTANT]
20+
====
21+
Back up your cluster's etcd data by performing a single invocation of the backup script on a control plane host. Do not take a backup for each control plane host.
22+
====
23+
24+
After you have an etcd backup, you can xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.adoc#dr-restoring-cluster-state[restore to a previous cluster state].
25+
26+
// Backing up etcd data
27+
include::modules/backup-etcd.adoc[leveloffset=+1]
28+
29+
[role="_additional-resources"]
30+
.Additional resources
31+
* xref:../../hosted_control_planes/hcp_high_availability/hcp-recovering-etcd-cluster.adoc#hcp-recovering-etcd-cluster[Recovering an unhealthy etcd cluster]
32+
33+
// Creating automated etcd backups
34+
include::modules/etcd-creating-automated-backups.adoc[leveloffset=+1]
35+
include::modules/creating-single-etcd-backup.adoc[leveloffset=+2]
36+
include::modules/creating-recurring-etcd-backups.adoc[leveloffset=+2]

‎etcd/etcd-backup-restore/etcd-disaster-recovery.adoc

+81
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following directory: backup_and_restore/control_plane_backup_and_restore/disaster-recovery/.
4+
5+
:_mod-docs-content-type: ASSEMBLY
6+
[id="etcd-disaster-recovery"]
7+
include::_attributes/common-attributes.adoc[]
8+
= Disaster recovery
9+
:context: etcd-disaster-recovery
10+
11+
toc::[]
12+
13+
The disaster recovery documentation provides information for administrators on how to recover from several disaster situations that might occur with their {product-title} cluster. As an administrator, you might need to follow one or more of the following procedures to return your cluster to a working state.
14+
15+
[IMPORTANT]
16+
====
17+
Disaster recovery requires you to have at least one healthy control plane host.
18+
====
19+
20+
[id="etcd-dr-quorum"]
21+
== Quorum restoration
22+
23+
You can use the `quorum-restore.sh` script to restore etcd quorum on clusters that are offline due to quorum loss. When quorum is lost, the {product-title} API becomes read-only. After quorum is restored, the {product-title} API returns to read/write mode.
24+
25+
// Restoring etcd quorum for high availability clusters
26+
include::modules/dr-restoring-etcd-quorum-ha.adoc[leveloffset=+2]
27+
28+
[role="_additional-resources"]
29+
.Additional resources
30+
* xref:../../installing/installing_bare_metal/upi/installing-bare-metal.adoc#installing-bare-metal[Installing a user-provisioned cluster on bare metal]
31+
* xref:../../installing/installing_bare_metal/bare-metal-expanding-the-cluster.adoc#replacing-a-bare-metal-control-plane-node_bare-metal-expanding[Replacing a bare-metal control plane node]
32+
33+
[NOTE]
34+
====
35+
If you have a majority of your control plane nodes still available and have an etcd quorum, xref:../../backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.adoc#replacing-unhealthy-etcd-member[replace a single unhealthy etcd member].
36+
====
37+
38+
[id="etcd-dr-restore"]
39+
== Restoring to a previous cluster state
40+
41+
To restore the cluster to a previous state, you must have previously backed up the `etcd` data by creating a snapshot. You will use this snapshot to restore the cluster state. For more information, see "Backing up etcd data".
42+
43+
If applicable, you might also need to xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-3-expired-certs.adoc#dr-recovering-expired-certs[recover from expired control plane certificates].
44+
45+
[WARNING]
46+
====
47+
Restoring to a previous cluster state is a destructive and destablizing action to take on a running cluster. This procedure should only be used as a last resort.
48+
49+
Before performing a restore, see "About restoring to a previous cluster state" for more information on the impact to the cluster.
50+
====
51+
52+
// About restoring to a previous cluster state
53+
include::modules/dr-restoring-cluster-state-about.adoc[leveloffset=+2]
54+
55+
// Restoring to a previous cluster state for a single node
56+
include::modules/dr-restoring-cluster-state-sno.adoc[leveloffset=+2]
57+
58+
// Restoring to a previous cluster state
59+
include::modules/dr-restoring-cluster-state.adoc[leveloffset=+2]
60+
61+
// Restoring a cluster from etcd backup manually
62+
include::modules/manually-restoring-cluster-etcd-backup.adoc[leveloffset=+2]
63+
64+
[role="_additional-resources"]
65+
.Additional resources
66+
* xref:../../backup_and_restore/control_plane_backup_and_restore/backing-up-etcd.adoc#backing-up-etcd-data_backup-etcd[Backing up etcd data]
67+
* xref:../../installing/installing_bare_metal/upi/installing-bare-metal.adoc#installing-bare-metal[Installing a user-provisioned cluster on bare metal]
68+
* xref:../../networking/accessing-hosts.adoc#accessing-hosts-on-aws_accessing-hosts[Accessing hosts on Amazon Web Services in an installer-provisioned infrastructure cluster]
69+
* xref:../../installing/installing_bare_metal/bare-metal-expanding-the-cluster.adoc#replacing-a-bare-metal-control-plane-node_bare-metal-expanding[Replacing a bare-metal control plane node]
70+
71+
include::modules/dr-scenario-cluster-state-issues.adoc[leveloffset=+2]
72+
73+
// Recovering from expired control plane certificates
74+
include::modules/dr-recover-expired-control-plane-certs.adoc[leveloffset=+1]
75+
76+
//Testing restore procedures
77+
include::modules/dr-testing-restore-procedures.adoc[leveloffset=+1]
78+
79+
[role="_additional-resources"]
80+
.Additional resources
81+
* xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.adoc#dr-restoring-cluster-state[Restoring to a previous cluster state]

‎etcd/etcd-backup-restore/images

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../images/

‎etcd/etcd-backup-restore/modules

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../modules/

‎etcd/etcd-backup-restore/replace-unhealthy-etcd-member.adoc

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
//NOTE TO CONTRIBUTORS:
2+
//
3+
//If you update any of the content in this assembly file, be sure to also make the same changes in the assemblies in the following file: backup_and_restore/control_plane_backup_and_restore/replacing-unhealthy-etcd-member.adoc.
4+
5+
:_mod-docs-content-type: ASSEMBLY
6+
[id="replace-unhealthy-etcd-member"]
7+
include::_attributes/common-attributes.adoc[]
8+
= Replacing an unhealthy etcd member
9+
:context: replace-unhealthy-etcd-member
10+
11+
toc::[]
12+
13+
The process to replace a single unhealthy etcd member depends on whether the etcd member is unhealthy because the machine is not running or the node is not ready, or because the etcd pod is crashlooping.
14+
15+
[NOTE]
16+
====
17+
If you have lost the majority of your control plane hosts, follow the disaster recovery procedure to xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-2-restoring-cluster-state.adoc#dr-restoring-cluster-state[restore to a previous cluster state] instead of this procedure.
18+
19+
If the control plane certificates are not valid on the member being replaced, then you must follow the procedure to xref:../../backup_and_restore/control_plane_backup_and_restore/disaster_recovery/scenario-3-expired-certs.adoc#dr-recovering-expired-certs[recover from expired control plane certificates] instead of this procedure.
20+
21+
If a control plane node is lost and a new one is created, the etcd cluster Operator handles generating the new TLS certificates and adding the node as an etcd member.
22+
====
23+
24+
// Identifying an unhealthy etcd member
25+
include::modules/restore-identify-unhealthy-etcd-member.adoc[leveloffset=+1]
26+
27+
// Determining the state of the unhealthy etcd member
28+
include::modules/restore-determine-state-etcd-member.adoc[leveloffset=+1]
29+
30+
== Replacing the unhealthy etcd member
31+
32+
Depending on the state of your unhealthy etcd member, use one of the following procedures:
33+
34+
* Replacing an unhealthy etcd member whose machine is not running or whose node is not ready
35+
* Installing a primary control plane node on an unhealthy cluster
36+
* Replacing an unhealthy etcd member whose etcd pod is crashlooping
37+
* Replacing an unhealthy stopped baremetal etcd member
38+
39+
// Replacing an unhealthy etcd member whose machine is not running or whose node is not ready
40+
include::modules/restore-replace-stopped-etcd-member.adoc[leveloffset=+2]
41+
42+
[role="_additional-resources"]
43+
.Additional resources
44+
* xref:../../machine_management/control_plane_machine_management/cpmso-troubleshooting.adoc#cpmso-ts-etcd-degraded_cpmso-troubleshooting[Recovering a degraded etcd Operator]
45+
* link:https://docs.redhat.com/en/documentation/assisted_installer_for_openshift_container_platform/2024/html/installing_openshift_container_platform_with_the_assisted_installer/expanding-the-cluster#installing-primary-control-plane-node-unhealthy-cluster_expanding-the-cluster[Installing a primary control plane node on an unhealthy cluster]
46+
47+
// Replacing an unhealthy etcd member whose etcd pod is crashlooping
48+
include::modules/restore-replace-crashlooping-etcd-member.adoc[leveloffset=+2]
49+
50+
// Replacing an unhealthy baremetal stopped etcd member
51+
include::modules/restore-replace-stopped-baremetal-etcd-member.adoc[leveloffset=+2]
52+
53+
[role="_additional-resources"]
54+
.Additional resources
55+
* xref:../../machine_management/deleting-machine.adoc#machine-lifecycle-hook-deletion-etcd_deleting-machine[Quorum protection with machine lifecycle hooks]

‎etcd/etcd-backup-restore/snippets

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../snippets

0 commit comments

Comments
 (0)