feat(conductor): fix leader loop switch by adding round-robin leadership transfer #18697
+113
−1
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds an optional round-robin leader transfer mechanism to op-conductor, configurable via the
--raft.round-robin-leader-transferflag.Problem
The current leader transfer in op-conductor relies on Raft's default
LeadershipTransfer(), which selects the next leader based on log completeness (the follower with the most up-to-date log). However, this selection mechanism is unaware of application-level health status (op-node & execution layer).This can cause a problematic scenario where unhealthy nodes keep being elected as leader in a loop:
Solution
Introduce a deterministic round-robin leader transfer that cycles through all voters in sorted order (by ServerID):
This ensures that even if only one node in the cluster is healthy, it will eventually become the leader after at most N-1 transfers.
Changes
--raft.round-robin-leader-transferflag (default: false, backward compatible)Tests
Manual testing was performed with a 4-node conductor cluster where 2 nodes were intentionally made unhealthy. With round-robin enabled, leadership eventually transferred to a healthy node. Without round-robin, leadership kept bouncing between the unhealthy nodes.
Additional context
This feature is opt-in and disabled by default to maintain backward compatibility. When disabled, the behavior is identical to the current implementation.
The round-robin approach trades off Raft's "most up-to-date log" optimization for guaranteed leader rotation. In practice, this is acceptable because:
All voters in a healthy cluster should have similar log states
The primary goal of leader transfer in op-conductor is to find a healthy sequencer, not to optimize for log completeness
If a target node's log is too stale, Raft will reject the transfer and we try the next node
Metadata