Why does strimzi-operator add PreferredLeaderElectionGoal (PLE) to goals?

[avoidik]

Hello,

The question is quite simple, but there's no rationale behind it. Why?

[avoidik]

If you'd check in the CC source-code, there's not PLE in the default.goals: https://github.com/linkedin/cruise-control/blob/3.0.3/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/config/constants/AnalyzerConfig.java#L311-L327

However, strimzi-operator adds it here: https://github.com/strimzi/strimzi-kafka-operator/blob/0.50.1/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/cruisecontrol/CruiseControlConfiguration.java#L53

As a result, the PLE is getting executed with every rebalancing.

[avoidik]

What's the theoretical impact? During the rebalancing the PLE may have impact to a few other goals, it also creates a skew by prioritising the first broker from the list.

https://github.com/linkedin/cruise-control/blob/3.0.3/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/analyzer/goals/PreferredLeaderElectionGoal.java#L136-L139

[kyguy]

@avoidik yes, to my understanding you are correct. Running the PreferredLeaderElectionGoal together with other goals can lead to unintended side effects. In particular, it may undo the work of other goals by reassigning partition leadership back to the preferred replica, thereby overriding prior optimizations.

[scholzj]

@kyguy @ppatierno Do you remember by any chance why this is included?

[ppatierno]

Looking at the code it was added as default goal since the beginning, If I saw it correctly since 2020. I can't see any specific reason why it was added if on purpose or not. I can totally understand the reasoning and its impact. @kyguy wdyt?

[kyguy]

This was due to an oversight and a misunderstanding of the PreferredLeaderElectionGoal on my part when I initially integrated Cruise Control with Strimzi.

Taking a closer look at the code and this GitHub issue comment here [1] it appears that the current implementation of the PreferredLeaderElectionGoal does not support running alongside other goals. It was designed to be used independently of other goals, specifically for rebalances focused solely on Preferred Leader Election (PLE) and broker demotion operations.

As @avoidik pointed out, running the PreferredLeaderElectionGoal together with other goals can lead to unintended side effects. In particular, it may undo the work of other goals by reassigning partition leadership back to the preferred replica, thereby overriding prior optimizations.

Ideally, the PreferredLeaderElectionGoal would support running alongside other goals. However, since it currently does not, Cruise Control should at least include safeguards to prevent it from being used in combination with other goals, along with documentation explaining the why. I will open an issue for this in the upstream Cruise Control project, link it here, and see to it that it gets addressed.

We should also remove the PreferredLeaderElectionGoal goal from Strimzi's default.goals list.

[1] linkedin/cruise-control#1326 (comment)

[kyguy]

Created an issue for this in the upstream Cruise Control project here [1]

[1] linkedin/cruise-control#2360

[ppatierno]

@kyguy so are you going to open a PR to remove the PreferredLeaderElectionGoal as default from Strimzi?

[kyguy]

Just opened a PR here [1] to remove the PreferredLeaderElectionGoal from the default.goals list in the Strimzi-configured Cruise Control configuration.

[1] #12554