KAFKA-13555: Use input partitions for StickyTaskAssignor [WIP] #19670
Conversation
|
The assignment logic for active tasks now primarily uses the number of input partitions as a proxy for individual task weight. The trigger for rebalancing active tasks considers the weight of the task to be added to the current weight of the client. A average weight buffer has been introduced to make the system less aggressive in breaking stickiness due to minor weight imbalances, aiming to reduce unnecessary task movements while still correcting significant imbalances. Standby task assignment continues to use the traditional task count-based logic. This was done due to a lack of understanding as to whether the input partitions should be considered for these assignments. This divergence in logic for active vs. standby tasks has introduced some awkwardness in the codebase, particularly around function signatures (e.g., findBestClientForTask needing different evaluation criteria). The findLeastLoadedClient method has been refactored to remove an unnecessary loop, improving its efficiency by computing the required information within the initial loop using additional variables. Some issues I’m aware of: • Standby Assignment Strategy: The current change makes standby client selection use the current task count-based metrics. Is this approach acceptable (which has resulted in the current awkward approach of passing in separate functions) or should it use the input partition based approach, or something else? • Unassigned tasks sorting: Should the sorting of remaining unassigned active tasks be changed from TaskId to descending weight to potentially further improve weight balance? • Should some input partitions be excluded from the calculation of a task’s weight? |
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
Addresses: KAFKA-13555