Failover Design

[bkolad]

Sequencer Role:

In the sequencer, the following components depend on the Sequencer role:

• BlobSender
• SeqDB
• CacheWarmUp
• TimeOracle
• Inner/sync_state
• ReplicaSyncTask

Goal

The goal of this issue is to define the steps required to transition the Sequencer from the Replica role to the Leader role.

Challenges

1. Atomicity

The transition must be atomic from the perspective of the external world. The Sequencer must never appear to be in a partially transitioned state—it should be either fully in the Replica role or fully in the Leader role.

2. Ordering

A strict ordering for switching components must be established. For example, switching Inner/sync_state to Leader mode before switching BlobSender would result in an invalid sequencer state. During the transition, the replica could start producing empty batches while BlobSender is still inactive and unable to send them, pushing the sequencer into a bad state.

3. Lifecycle of async tasks

Many of the components above (e.g., BlobSender, CacheWarmUp, TimeOracle) spawn asynchronous actors that are registered when the sequencer starts. Replicas do not start these actors. However, the system currently does not support starting these tasks after the sequencer has already been initialized.

If 2 and 3 are well designed, atomicity should follow automatically.

2. Ordering should be enforced by a dedicated component, ReplicaToLeaderTransitionManager.

During the Replica → Leader transition, the following sequence should occur:

1. ReplicaToLeaderTransitionManager receives a database notification indicating that the node has become the new Leader.

2. It sends a shutdown request to ReplicaSyncTask and waits for confirmation that the task has fully stopped.

3. It sends a request to Inner/sync_state to transition to Leader mode.

4. Inner/sync_state:

   • Sends a request via SideEffectsTask instructing BlobSender and SeqDB to upgrade to the Leader role.
   • Updates its local role flag to Leader.
   • To support this, we will introduce a new ExecutorEvent::UpgradeToLeader message in SideEffectsTask.

5. ReplicaToLeaderTransitionManager must provide a mechanism to wait until step 4 has fully completed.

6. ReplicaToLeaderTransitionManager sends a request to CacheWarmUp, notifying it to start.

7. ReplicaToLeaderTransitionManager sends a request to TimeOracle, notifying it to start.