[Bug]: Rebalance does not happen at all using AutoRebalance feature

[pnagy-cldr]

Bug Description

We test the AutoRebalance feature and in rare cases it does not do anything.
The issue is very similar to: #11195
The only difference is that the old issue talks about manual rebalance, and the current one is about the automatic one.

Steps to reproduce

1. Create Kafka where autorebalance is configured for add/remove brokers:

spec:
cruiseControl:
autoRebalance:
- mode: add-brokers
- mode: remove-brokers

2. Scale Kafka up a Kafka cluster.
3. Strimzi will detect that add_broker should be called, and it calls CC
4. CC is still in rolling restart phase, so in an unlucky situation the old CC gets the request and it does not know any capacity info about new broker, response will be 500:
   'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: java.lang.NullPointerException: Cannot invoke "com.linkedin.kafka.cruisecontrol.config.BrokerCapacityInfo.capacity()"
5. The KafkaRebalance resource will be in NotReady state, updated by Strimzi after the error:
   ERROR KafkaRebalanceAssemblyOperator:403 - Reconciliation #1287(watch) KafkaRebalance(...):Status updated to [NotReady] due to error:...
6. In case the KafkaRebalance is in NotReady state and the scalingNodes does not change in the meantime (So you do not scale further), your KafkaRebalance will simply be deleted by KafkaAutoRebalancingReconciler, and the rebalance will not happen.

Expected behavior

This issue is not permanent, as I mentioned in the steps it seems rare.
Should Strimzi retry this kind of issues? In automatic rebalance case, it is very weird that my rebalance will not happen ever because of a timing issue. Obviously I can create my rebalance manually but that is why I use the automatic one.

Strimzi version

0.45

Kubernetes version

1.28

Installation method

Helm chart

Infrastructure

k3s

Configuration files and logs

No response

Additional context

No response

[egyedt]

Some additional details (based on code review and with Cruise Control knowledge):
From Cruise Control side, the problem is that, CC needs to know about broker capacities, but without a restart the new file with the new broker is not populated.

Since CC is in the rolling restart phase, the "old CC" manages the request (when the above timing issue happens), but this CC will be not familiar with the new broker(s). Only the "new CC" knows about the new broker(s). Because of the lack of retry attempts (the request will be sent only one time), the "new CC" which would able to handle the request will never get any queries and the KafkaRebalance resource will result in failed (NotReady) KafkaRebalanceStatus, so the operator will remove this CR in the next reconciliation loop.

The result will be the state in the title: "Rebalance may not happen at all using AutoRebalance feature".

The issue is very similar to: #11195
In contrast, we have to do here something for sure (e.g. better timing - send query after CC restarted -, retry or something else), since this is a fully automated process (operator recognize the new replica count, creates new broker(s), create KafkaRebalance resource, handles it automatically, then delete KafkaRebalance and finish the procedure related to scaling) and it is not able to trigger "refresh" value for "strimzi.io/rebalance" annotation manually, the problem should be handled in operator code in my opinion.

@scholzj Could one of you take a look at this issue please?
Thanks in advance!

[scholzj]

I have seen this few times, but it was pretty rare. Is it happening all the time to you?

I think the main issue is that this method is not fully reliable:

strimzi-kafka-operator/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/resource/kubernetes/DeploymentOperator.java

Line 110 in 35e3d84

public Future<Void> waitForObserved(Reconciliation reconciliation, String namespace, String name, long pollIntervalMs, long timeoutMs) {

... but TBH I do not know how to make it more reliable. We update the Deployment and try to see if a rolling update is happening and if it does wait for it. But as things are eventually consistent, there seem to be some races were this method just passes immediately. How often it happens might depend on many factors I guess such as how fast your Kubernetes core compoenents are etc.

I think there are several ways how to solve it that come to my mind:

1. The best would be to improve the reliability of this method - but as I said, I do not know how.
2. The easiest would be to see if using Recreate strategy for the rollout helps with this particular problems. If you are able to reproduce this problem, you can easily set it through the Deployment template for the Cruise Control Deployment and test if it helps. If it does help, we can easily change the default in the code.
3. We can move away from using Deployments and move to manage the Pod directly or through StrimziPodSet. That would allow us to better handle the rolling (as we would manage it). But this seems pretty heavy solution for a this kind of issue.
4. Being able to dynamically reconfigure Cruise Control would be also great and would help with other things as well. But that would need to be done in the Cruise Control code and be accepted by Linkedin.

I'm afraid I'm currently on leave, so I will not have time to work on this right now. Not sure if @ppatierno will have some spare cycles to look at it. But in any case, if you can reliably reproduce it, the second point is something you can easily try your self to see if this would help. I cannot try that easily because as I said, while I saw this issue, it was pretty rare so it would be hard to test for me.

[ppatierno]

I will find some time to investigate options 1 and 2 suggested by Jakub.
I think 3 is too much and 4 could take longer time (in terms of engaging with the CC community and having dynamic configuration supported).
A 5th option could be setting up a retry mechanism.

[scholzj]

Option 3 would certainly not be trivial. But it would likely help more issues such as #8589 etc.