There are ongoing intra-broker partition movements

[ccrootdeeplay]

Good afternoon, we are using strimzi version 0.30 (upgraded from 0.27.1) we have disks in jbod so we use rebalance " Rebalance Disk: true " at the moment it successfully generates proposal but it returns the error "Error for request: kafka-cruise-control. kafka-prod.svc:9090/kafkacruisecontrol/rebalance?json=true&dryrun=false&verbose=true&skip_hard_goal_check=false&rebalance_disk=true. Server │
│ returned: Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.OngoingExecutionException: There are ongoing intra-broker partition movements.'." in bin/kafka-log-dirs.sh some partitions hang with status isFuture: true I tried using manual balancing for such partitions, but after a while it "gets stuck" is there any way to automate this?

[scholzj]

I guess the error suggests that you have some rebalance in progress. Either manual or automated. It sound like you need to resolve it first before doing the next one.

[ccrootdeeplay]

Yep sounds like, but kafka-reassign-partitions.sh says "No partition reassignments found." looks like their stuck on zk node( have no idea how to fix this...

[ccrootdeeplay]

All i can do -> find isfuture: true partitions with kafka-log-dirs.sh -> start rebalance for it with hands -> it's ok and after i can start rebalance with cc, but it becomes zombie again and next rebalance be with "ongoing intra-broker partition movements"

[imans777]

We ran into similar issues lately. Though we use kafka baremetal and cruise control along side it, no operator/k8s. I can briefly tell you what we did and what we observed to overcome the problem:

• limit cc's max partition movement. default is 1250. we used 500 and still got stucked. So we set it to 50. Also limit concurrent movements and throttles.
• stopping disk rebalancing, from what we've seen, only prevents new assignments. Ongoing ones will still continue so we had to wait for them.
• remove /kafka/controller zk node and let another controller take control.
• restarting all non-controller brokers.
• restarting zookeeper nodes one by one.
• possibly take out a bunch of brokers out of cluster, wait for some time, switch controller, and them bring them back.
• checking the kafka metrics of "average idle time" and "queued requests" and "kafka handlers" and seeing the anomaly of being stuck.
• setting logs of kafka to DEBUG and see anomalies.
• possibly from the two items above, we saw that the handles were fulled out and requests were stuck. If you are unable to run "kafka-topics.sh --describe" and see "error Disconnected node ... timed out waiting for ListPartitionReassignment" then this is definitely the problem.
• make sure the kafka configs are tuned enough. look for number of threads and timeouts.
• you can also do some dirty things though will full caution. check zk node /admin/reassign_partition, that shows if anything is ongoing. to force stop, you can remove that, and then we also had to remove "adding_replicas" and "removing_replicas" from "/brokers/topics/..." node and make sure other partitions are correct (remove "adding_replicas" from main partition list). This was very tiresome and tricky and risky, but did worked out for us.

Hope this helps.

[urton]

Adding a comment here since I just ran into this with Strimzi (CC pod restarted a day into a multi-TB rebalance). This was the only solution that worked for me:

• Delete the KafkaRebalance CR
• Remove the cruiseControl: block from the Kafka CR (brokers will roll)
• Add the cruiseControl: block back to the Kafka CR (brokers will roll)
• Recreate the KafkaRebalance CR; annotate as needed

Strimzi version: 0.45.1
K8s platform|version: OpenShift (ROSA HCP)|4.19.21