Skip to content

Commit 71450d2

Browse files
scylla-operator-bot[bot]scylla-operator-bot[bot]
authored
Merge pull request #2955 from mflendrich/ep-failed-replace-recovery
Add an enhancement proposal for a failed replace recovery guide
2 parents 79750d3 + dc2edb5 commit 71450d2

1 file changed

Lines changed: 215 additions & 0 deletions

File tree

‎enhancements/proposals/2955-failed-replace-recovery/failed-replace-recovery.md‎

Lines changed: 215 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,215 @@
1+
# Procedure to Recover from Failed Node Replace
2+
3+
## Summary
4+
5+
Node replace operations can fail, leaving the ScyllaDB cluster in an inconsistent state in which Operator cannot apply any further configuration updates.
6+
The [failed membership change](https://docs.scylladb.com/manual/branch-2025.1/operating-scylla/procedures/cluster-management/handling-membership-change-failures.html#cleaning-up-after-a-failed-membership-change) tutorial in the ScyllaDB docs is not correct for Operator, because there is a need to also keep the Kubernetes state (the `StatefulSet`, the node service, the storage) consistent.
7+
8+
## Motivation
9+
10+
No well-defined recovery path in scenarios where Operator gets stuck for unhandled reasons.
11+
12+
### Goals
13+
14+
- Enable users and CX with a well-defined procedure to perform when the node replace is stuck.
15+
- Put this procedure in Operator's public docs.
16+
- Understand if this procedure is reusable for different classes of failures.
17+
18+
### Non-Goals
19+
20+
- Add more automation to Operator
21+
- Try to solve for situations where other nodes are unhealthy or there are unrelated topology changes ongoing in the cluster.
22+
23+
## Proposal
24+
25+
The proposal is to add a guide that builds upon the [failed membership change guide](https://docs.scylladb.com/manual/branch-2025.1/operating-scylla/procedures/cluster-management/handling-membership-change-failures.html#cleaning-up-after-a-failed-membership-change) that would employ the following steps:
26+
27+
_**Note Box**: To recover from failed node replace for multi-datacenter ScyllaDB deployments, simply perform the steps outlined in this guide in the worker cluster where the failed node is located._
28+
29+
#### Verify that there is indeed a node that failed to join the cluster
30+
31+
Perform `nodetool status` on a functioning node of the cluster (different than the culprit node). You should see a node with status different than `UN` (for example `DN`, `?N`).
32+
33+
_**Warning Box** The guide assumes that the rest of the cluster is healthy._
34+
35+
```console
36+
$ kubectl exec -n examplens scylla-exampledc-somehealthynode-5 -- nodetool status
37+
38+
Datacenter: exampledc
39+
=====================
40+
Status=Up/Down
41+
|/ State=Normal/Leaving/Joining/Moving
42+
-- Address Load Tokens Owns Host ID Rack
43+
UN 10.152.183.112 491.57 KB 256 ? e7478c73-07a9-4fb2-a435-6603ccc9e6bd examplerack
44+
DN 10.152.183.214 466.12 KB 256 ? 09d815de-6f6d-4394-8439-bd8d34231835 examplerack
45+
UN 10.152.183.43 456.25 KB 256 ? ac4e578d-cc82-4b71-9ba1-0f40aede9e8d examplerack
46+
```
47+
48+
Ensure that the culprit entry matches the old (replaced) node ID, or the new (attempting to replace) node ID, based on the logs of the failing node.
49+
50+
#### Back up your data
51+
52+
This is a dangerous operation. It is recommended to perform a data backup before proceeding.
53+
54+
#### Capture a must-gather archive
55+
56+
Since recovery by manual removal of a node is a destructive operation, please collect a [must-gather](https://operator.docs.scylladb.com/stable/support/must-gather.html) archive before changing any Kubernetes state. Do not skip this step - the archive may be useful in configuration recovery if something goes wrong.
57+
58+
#### Pause the components that might interfere with the procedure
59+
60+
This assumes that the failed node's name is `scylla-exampledc-examplerack-1` in the namespace `examplens`
61+
62+
Stop the Operator (keep the ScyllaDB instances running, but prevent reconciliation). In a future improvement we can add a field to `ScyllaCluster` and/or `ScyllaDBDatacenter` to prevent Operator from "seeing" it, instead of scaling down to 0.
63+
64+
```console
65+
$ # Before performing this, take note of the number of replicas in the deployment before this operation (typically 2).
66+
$ # You will need to scale back to that number later.
67+
$ kubectl scale -n scylla-operator deploy/scylla-operator --replicas=0 --timeout=5m
68+
```
69+
Prevent the StatefulSet from recreating the pod instantly. We achieve this by orphan deleting the StatefulSet for that specific rack,
70+
because Operator will recreate the StatefulSet in its exact form when Operator is scaled back up at the end of the procedure.
71+
72+
_**Warning Box** Execute this step carefully - it can have destructive effects._
73+
74+
```console
75+
$ # WARNING: Do not forget the --cascade=orphan parameter.
76+
$ # Doing otherwise will cause downtime.
77+
$ kubectl delete statefulset -n examplens scylla-exampledc-examplerack --cascade=orphan
78+
```
79+
80+
#### Obtain the Host ID of the culprit node, and the Host IDs of potential ghost nodes.
81+
82+
Follow the [_Step One: Determining Host IDs of Ghost Members_](https://docs.scylladb.com/manual/branch-2025.1/operating-scylla/procedures/cluster-management/handling-membership-change-failures.html#step-one-determining-host-ids-of-ghost-members) guide and note down the Host IDs of any potential _ghost nodes_.
83+
84+
#### Stop the culprit node
85+
86+
**WARNING**: This deletes the node's data.
87+
88+
```console
89+
$ # Stop the node that is failing to join the cluster.
90+
$ # The StatefulSet does not exist, so it will not recreate the Pod.
91+
$ kubectl delete pod -n examplens scylla-exampledc-examplerack-1
92+
93+
$ # WARNING: This causes data loss.
94+
$ # Delete the PersistentVolumeClaim for the data volume held by the culprit node.
95+
$ kubectl delete persistentvolumeclaim -n scylla scylla-exampledc-examplerack-1
96+
97+
$ # Delete the service associated with the node.
98+
$ # Operator interprets this as a need to provision a brand new node instead of attempting to replace the old one in the rack.
99+
$ kubectl delete service -n examplens scylla-exampledc-examplerack-1
100+
```
101+
102+
#### Remove the culprit node and any possible _ghost nodes_ from the ScyllaDB cluster
103+
104+
Follow the [_Step Two: Removing the Ghost Members_](https://docs.scylladb.com/manual/branch-2025.1/operating-scylla/procedures/cluster-management/handling-membership-change-failures.html#step-two-removing-the-ghost-members) part of the failed membership change guide to `nodetool removenode` the node that failed to join the cluster and any _ghost members_.
105+
106+
Repeat as necessary:
107+
108+
```console
109+
$ kubectl exec -n examplens scylla-exampledc-somehealthynode-5 -- nodetool removenode HOST_ID_OF_THE_NODE_TO_REMOVE
110+
```
111+
112+
Verify that the culprit node and/or the ghost nodes are no longer present in `nodetool status`:
113+
114+
```console
115+
$ kubectl exec -n examplens scylla-exampledc-somehealthynode-5 -- nodetool status
116+
117+
Datacenter: exampledc
118+
=====================
119+
Status=Up/Down
120+
|/ State=Normal/Leaving/Joining/Moving
121+
-- Address Load Tokens Owns Host ID Rack
122+
UN 10.152.183.112 491.57 KB 256 ? e7478c73-07a9-4fb2-a435-6603ccc9e6bd examplerack
123+
UN 10.152.183.43 456.25 KB 256 ? ac4e578d-cc82-4b71-9ba1-0f40aede9e8d examplerack
124+
```
125+
126+
#### Resume Operator and let it heal the cluster
127+
128+
Resume Operator by scaling back to the original number of replicas.
129+
130+
```console
131+
$ # Replace NNN with the number of replicas from before the initial scale-down-to-0.
132+
$ # That number is typically 2.
133+
$ kubectl scale -n scylla-operator deploy/scylla-operator --replicas=NNN --timeout=5m
134+
```
135+
136+
#### Verify healing results
137+
138+
Sit back and see Operator:
139+
- recreate the (identical) `StatefulSet` for the rack under repair (`scylla-exampledc-examplerack`),
140+
- recreate the (identical) `Service` for the node that failed to replace previously (`scylla-exampledc-examplerack-1`),
141+
- create new `Pod` and `PersistentVolumeClaim` for a net new node in place of the culprit node (each named `scylla-exampledc-examplerack-1`),
142+
143+
After that happens, run `nodetool status` to see that the new node has joined the cluster and is `UN`:
144+
145+
```console
146+
$ kubectl exec -n examplens scylla-exampledc-somehealthynode-5 -- nodetool status
147+
148+
Datacenter: exampledc
149+
=====================
150+
Status=Up/Down
151+
|/ State=Normal/Leaving/Joining/Moving
152+
-- Address Load Tokens Owns Host ID Rack
153+
UN 10.152.183.112 491.57 KB 256 ? e7478c73-07a9-4fb2-a435-6603ccc9e6bd examplerack
154+
UN 10.152.183.234 493.66 KB 256 ? NEW-UUID-DIFFERENT-THAN-BEFORE-abcde examplerack
155+
UN 10.152.183.43 456.25 KB 256 ? ac4e578d-cc82-4b71-9ba1-0f40aede9e8d examplerack
156+
```
157+
158+
### Notes/Constraints/Caveats [Optional]
159+
160+
Question to reviewers.
161+
- Should we make this tutorial target other use cases as well (failed bootstrap in general?)
162+
- Can we expect in the general case that this will guarantee a return of the data integrity conditions one would expect from a successful node replace?
163+
164+
### Risks and Mitigations
165+
166+
#### Recovery of the Declarative State
167+
168+
As a general precaution, the guide includes a step to capture a must-gather snapshot of the declarative state of the Kubernetes cluster. It can be used for further debugging or for recovery of the declarative state in the event of some unhandled failure.
169+
170+
#### User mistake - cascading deletion of the StatefulSet
171+
172+
The guide builds upon the [promises of `StatefulSet` semantics](https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/):
173+
174+
> If you want to delete only the StatefulSet and not the Pods, use `--cascade=orphan`.
175+
176+
> Deleting the Pods in a StatefulSet will not delete the associated volumes. This is to ensure that you have the chance to copy data off the volume before deleting it. Deleting the PVC after the pods have terminated might trigger deletion of the backing Persistent Volumes depending on the storage class and reclaim policy.
177+
178+
If the user forgets to `--cascade=orphan` when deleting the `StatefulSet` as part of the procedure, it will cause downtime: the pods of the rack will be deleted, resulting in (reversible) unavailability.
179+
180+
Since deletion of a `StatefulSet` [does not cause the deletion of the associated PVCs](https://kubernetes.io/docs/tasks/run-application/delete-stateful-set/#persistent-volumes) by itself, this operation should not cause data loss by itself.
181+
182+
Reversing this situation is possible by recreating the `StatefulSet` with identical configuration as the original one, that will recreate the deleted pods. These pods inherit the PVCs, so their state should be fully preserved. In a typical case, Operator will perform this automatically when scaled back to a positive number of replicas.
183+
184+
As an additional (proactive) mitigation, the guide includes a warning box as part of the "orphan delete the StatefulSet" step.
185+
186+
#### Operator fails to start up
187+
188+
Once scaled up to a positive number of replicas, Operator will restore the original declarative state (recreate the `StatefulSet`) automatically. In the event that Operator is non-functional for whatever reason (environmental, bug, unhandled case), the StatefulSet can be recreated manually from its configuration in the must-gather archive collected.
189+
190+
## Design Details
191+
192+
### Test Plan
193+
194+
#### Acceptance Test
195+
196+
Verify that the procedure yields the desired result of deleting a non-functioning node and creating one that is up, **in a cluster that has all other nodes up**.
197+
Verify that the failure modes: accidental cascading deletion of the StatefulSet, recovery without Operator from the must-gather archive, actually yield the expected results, and do not result in data loss.
198+
199+
#### Regression Test - Optional
200+
201+
Create an SCT or E2E case covering this procedure and expecting it to result in a healed cluster without data loss.
202+
203+
### Upgrade/Downgrade Strategy
204+
205+
Not relevant. This procedure is tied to the version of our API and the semantics of the `StatefulSet` underneath.
206+
207+
### Version Skew Strategy
208+
209+
Same as above. As long as the APIs stay the same, there should be no impact.
210+
211+
## Alternatives
212+
213+
- Implement a "freeze" switch in Operator to mitigate the need to scale it down to zero.
214+
- Implement this sequence of operations in Operator directly.
215+

0 commit comments

Comments
 (0)