feat(shard-distributor): load-based rebalancing and placement

[AndreasHolt]

What changed?

We added load balancing to the leader’s periodic reconciliation (inside rebalanceShardsImpl). It now moves shards when executors fall outside hysteresis bands around the cluster’s mean load. Moves are limited by a per-pass budget that scales with the total number of shards, and the loop stops when the budget is used up or things are back within the bands. For each candidate move we compute a “benefit”, and we only move a shard if that benefit is positive. We also added/updated metrics around load balance and shard movement. Finally, ephemeral shard placement was changed to assign new shards to the ACTIVE executor with the lowest load (instead of the one with the fewest shards).

Why?

• Prior to this change, the shard distributor did not do load-aware placement of new ephemeral shards, nor periodic rebalancing of shards based on load.
• To make shard creation/ephemeral assignment less likely to create future imbalance by placing new shards on the lowest-load executor.
• We need observability to see how well we load balance, therefore we emit the metrics on load imbalance, churn etc.

How did you test it?

• Ran the shard distributor unit tests locally, including new/updated load-balancing tests in service/sharddistributor/leader/process/loadbalance_test.go and handler tests under service/sharddistributor/handler

• We have also tested it with running it with multiple canary instances, and then logging the moves. The logs showed shards being moved from executors with high load to executors with low load.

Potential risks