Forced bootstrap from new leader

[sebadob]

Hey,

I am not sure how exactly I got into this situation, but during testing / local dev, I was able to get the cluster in a state, where it would not come up, if one node was missing, that was the leader before.

Let me try to explain the situation:

Assuming I have 3 nodes which have successfuly formed an openraft cluster and everything is running smoothly. Then, let's assume the whole network goes down for whatever reason and because of K8s health checks or similar, pods are getting restarted. And now let's assume as well, that something else happened at the same time and the node that was the leader before ends up with a corrupted volume and cannot restart afterwards and would need to be rebuilt from the other nodes.

To break it down a bit better, this was (kind of) the situation I got into during testing:

1. Running Cluster:

• Node 1: Follower
• Node 2: Follower
• Node 3: Leader

2. The network crashes and pods are getting shut down or restarted automatically while all Nodes "think" Node 3 is still the leader.
3. Node 3 (old Leader) had a major issue and lost it's volume, therefore it cannot come up again, when the network comes back
4. Network comes back, Node 1 and 2 are starting up again
5. Node 3 cannot come up because of a lost / corrupted volume
6. Node 1 and 2 had Node 3 saved as leader, before they lost connection to the rest of the cluster and are stuck in a reconnect loop until Node 3 comes up (which it can't anymore).

It of course makes a lot of sense that Node 1 and 2 would try to connect to Node 3 to reach a consistent state and not potentially loose any data. But, what if Node 3 got destroyed and it just cannot come back? In this case, I want to make it possible to have something like a forced bootstrap or basically reset the last known leader to None, so the healthy Nodes can elect a new one again.
Yes, this means you could potentially loose some data (I guess?), but at least it would be way less than if I would apply a backup.

What I was thinking about is to add an option during startup that would delete the last saved openraft::vote::vote::Vote.
Now my problem is, that I don't know how exactly I got into that state I described above, but I got there already, somehow. Since I don't know a way how to reproduce this consistently, I cannot test if deleting the saved Vote before starting up the logs store would fix the issue.

Is this something that could work, or can I do this in some other way?

[drmingdrmer]

Node 1 and node 2 should not try to connect to node 3. Could you share logs showing the network traffic?

Usually, if nodes 1 and 2 don't receive any messages from node 3 for a while, they will start an election to determine a leader. Can you provide logs showing what these nodes did during startup? Typically the saved vote will be updated to a candidate vote.

Regarding data recovery, OpenRaft does support recovering a node after wiping its data, to enable this feature:

• In OpenRaft 0.9:
  This requires enabling the loosen-follower-log-revert feature flag. Without this flag, wiping a node's data and restarting it will cause the leader to panic. https://docs.rs/openraft/latest/openraft/docs/faq/index.html#can-i-wipe-out-the-data-of-one-node-and-wait-for-the-leader-to-replicate-all-data-to-it-again

• In OpenRaft 0.10 (main branch):
  It is enabled by a runtime config, instead of a compilation time feature flag.
  https://github.com/databendlabs/openraft/blob/main/openraft/src/docs/faq/faq.md#can-i-wipe-out-the-data-of-one-node-and-wait-for-the-leader-to-replicate-all-data-to-it-again

[sebadob]

Node 1 and node 2 should not try to connect to node 3.

Yeah, that's the behavior I observed lately when trying to replicate this issue. I think I saw it last a few minor versions ago (openraft 0.9), but I could not find a way to reliably reproduce it, I am still trying.

I am currently making use of the loosen-follower-log-revert with some safeguards under certain conditions, just to have a way to at least recover a node easily, if this is requested via config.

In OpenRaft 0.10 (main branch):
It is enabled by a runtime config, instead of a compilation time feature flag.

This is a very nice change.

---

Regarding my main issue, as said I saw it last a few minor versions back. I thought that it could maybe have something to do with a case where for instance the last known leader had a higher commited log id, and all the nodes that are then up after a cold start have lower values. I am not sure if it would even be possible to construct such a situation.

In my latest tests when fiddling around with it and trying to break it, is always worked in a way that the new nodes would elect a new leader, not matter if I cleaned up the last saved vote or not.

If I ever get into that situation again, I will try to extract logs, but for now, I have not seen this issue in quite some time. I will start testing hiqlite against openraft 0.10 on an unstable branch, as soon as I have some spare time to do that.

Thank you!

[drmingdrmer]

Regarding my main issue, as said I saw it last a few minor versions back.

As far as I know, there hasn't been such an issue with any history version of OpenRaft. The reelection of a leader after crashes has always worked reliably.

Please send me logs when you reproduce this behavior. Any log would help determine what's happening.

I am not sure but this might be related to Kubernetes networking. K8S can sometimes introduce networking issues like hostname resolution problems. I'm not a K8S expert, but my colleagues and I have encountered similar issues before. 🤔

[sebadob]

I am not sure but this might be related to Kubernetes networking.

Afaik, this was on localhost during "trying to break it". If have only seen this twice since I started my hiqlite project with it.

I just remember, that a node got stuck in an endless "trying to connect to last known leader" loop.
I will provide logs, if I ever get into that situation again.