Support generate rebalance plan

[swuferhong]

Search before asking

• I searched in the issues and found nothing similar.

Description

Support generate rebalance plan

Willingness to contribute

• I'm willing to submit a PR!

[gyang94]

Hi @swuferhong , I am interested in this topic and may I take this task?
In my understanding, fluss rebalance is something similar to kafka's reassign process, which also allow users to submit a plan, and then executing the plan to rebalance the cluster.
For this ticket, we need to implement the GoalOptimizer, which combines the user requests and the bucket load to generate a rebalance plan. But I think the there are too many details need to be discuss (e.g. what metrics we will take to generate plan, what is the algorithm to generate this plan based on those metrics, how to handle the conflicts between user's plan and metrics... etc).
If there is a chance to do this task together, we can discuss further. Looking forward your reply.

[swuferhong]

Hi, @gyang94. Apologies, I'm currently finalizing the POC for this task (80%). The POC implementation details are substantially similar to the FIP proposal:
https://cwiki.apache.org/confluence/display/FLUSS/FIP-8%3A+Support+Cluster+Reblance. However, the initial implementation is intentionally minimal, currently we only support LeaderReplicaDistributionGoal , ReplicaDistributionGoal , PreferredLeaderElectionGoal , and will not introduce a metrics collection system. It will simply count the current bucket distribution of each tabletServer and will not involve rack location. So, if you're interested, there will be many sub-tasks to work on in the future.

Additionally, regarding the implementation, the.FIP-8's idea is not to expose the reassignment interface to users, as user-defined reassignment plans are impractical for large clusters—especially for Fluss with Partitioned tables. Therefore, the rebalance plan generation will be managed by Fluss self. User can use the serverTag interface to mark machines that need to be decommissioned or require special handling, and the RebalanceManager can collect these serverTags to generate the plan.

For the specific goal implementation, we will refer to LinkedIn's open-source Cruise Control (https://github.com/linkedin/cruise-control), which has many excellent design choices.

If you'd like to participate in this large rebalance effort, you could start by reviewing my API PR (#1380) and the upcoming PR supporting rebalance. Later, you could expand on the foundational capabilities, such as collecting richer metrics or supporting more advanced goals.

[swuferhong]

FIP-8 also doesn't plan to expose the PreferredLeaderElection interface to users, as they can achieve the same goal by passing the PreferredLeaderElectionGoal via the rebalance interface.

[gyang94]

@swuferhong Thanks for your detailed introduction about the current progress and future work. I will take a look at the PR first.
I understand that we start this feature with minimal implementation. And in the future if we need to expand more advanced capabilities, maybe I can help a little bit. :)

[swuferhong]

@swuferhong Thanks for your detailed introduction about the current progress and future work. I will take a look at the PR first. I understand that we start this feature with minimal implementation. And in the future if we need to expand more advanced capabilities, maybe I can help a little bit. :)

Certainly, during testing, we identified some high-priority advanced features in stage two. I'll CC you when I create the issue.