[Bug]: AutoRebalance to remove brokers failed because of kafkaRebalanceState is null

[acgtun]

Bug Description

2025-10-12 04:08:10 INFO  KafkaAutoRebalancingReconciler:148 - Reconciliation #107345(watch) Kafka(kafka/preprod): Auto-rebalance on scaling down with KafkaRebalance kafka/preprod-auto-rebalancing- │
│ 2025-10-12 04:08:10 ERROR AbstractOperator:285 - Reconciliation #107345(watch) Kafka(kafka/preprod): createOrUpdate failed                                                                            │
│ java.lang.NullPointerException: Cannot invoke "io.strimzi.api.kafka.model.rebalance.KafkaRebalanceState.ordinal()" because "kafkaRebalanceState" is null                                              │
│     at io.strimzi.operator.cluster.operator.assembly.KafkaAutoRebalancingReconciler.lambda$onRebalanceOnScaleDown$8(KafkaAutoRebalancingReconciler.java:152) ~[io.strimzi.cluster-operator-0.47.0.jar │
│     at io.vertx.core.impl.future.Composition.complete(Composition.java:40) ~[io.vertx.vertx-core-5.0.1.jar:5.0.1]                                                                                     │
│     at io.vertx.core.impl.future.FutureBase.lambda$emitResult$0(FutureBase.java:59) ~[io.vertx.vertx-core-5.0.1.jar:5.0.1]                                                                            │
│     at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                    │
│     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                │
│     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:507) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                        │
│     at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:182) ~[io.netty.netty-transport-4.2.2.Final.jar:4.2.2.Final]                                                         │
│     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                             │
│     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                                 │
│     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                     │
│     at java.lang.Thread.run(Thread.java:840) ~[?:?]                                                                                                                                                   │
│ 2025-10-12 04:08:10 WARN  AbstractOperator:566 - Reconciliation #107345(watch) Kafka(kafka/preprod): Failed to reconcile                                                                              │
│ java.lang.NullPointerException: Cannot invoke "io.strimzi.api.kafka.model.rebalance.KafkaRebalanceState.ordinal()" because "kafkaRebalanceState" is null                                              │
│     at io.strimzi.operator.cluster.operator.assembly.KafkaAutoRebalancingReconciler.lambda$onRebalanceOnScaleDown$8(KafkaAutoRebalancingReconciler.java:152) ~[io.strimzi.cluster-operator-0.47.0.jar │
│     at io.vertx.core.impl.future.Composition.complete(Composition.java:40) ~[io.vertx.vertx-core-5.0.1.jar:5.0.1]                                                                                     │
│     at io.vertx.core.impl.future.FutureBase.lambda$emitResult$0(FutureBase.java:59) ~[io.vertx.vertx-core-5.0.1.jar:5.0.1]                                                                            │
│     at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:148) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                    │
│     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                │
│     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:507) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                        │
│     at io.netty.channel.SingleThreadIoEventLoop.run(SingleThreadIoEventLoop.java:182) ~[io.netty.netty-transport-4.2.2.Final.jar:4.2.2.Final]                                                         │
│     at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:1073) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                             │
│     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                                 │
│     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty.netty-common-4.2.2.Final.jar:4.2.2.Final]                                                     │
│     at java.lang.Thread.run(Thread.java:840) ~[?:?]

Steps to reproduce

1. enable autoRealance for remove-broker
2. reduce kafkanodepool broker from 20 nodes to 10 nodes

Expected behavior

No response

Strimzi version

0.47.0

Kubernetes version

Client Version: v1.29.12 Server Version: v1.30.3

Installation method

argoCD

Infrastructure

kubernetes cluster

Configuration files and logs

No response

Additional context

Actually, the data was already moved out by cruise-control, but the partitions were not moved out. Later, I manually run a reassignment command to assign replicas to other brokers (0 to 9), after that, the AutoRebalance immediately remove broker 10 to 19.

[scholzj]

A good start would be to share the full logs from the operator, the full configuration of your custom resources including their .status section, etc.

[acgtun]

The full log is too large. Basically it repeated the foloowing

2025-10-12 01:02:08 WARN  KafkaClusterCreator:191 - Reconciliation #106272(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:02:10 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106272(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:04:02 WARN  KafkaClusterCreator:191 - Reconciliation #106274(timer) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:04:04 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106274(timer) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:04:05 WARN  KafkaClusterCreator:191 - Reconciliation #106278(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:04:06 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106278(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [Idle] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:04:06 WARN  KafkaClusterCreator:191 - Reconciliation #106279(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:04:08 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106279(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:04:08 WARN  KafkaClusterCreator:191 - Reconciliation #106280(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:04:10 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106280(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:06:02 WARN  KafkaClusterCreator:191 - Reconciliation #106287(timer) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas


2025-10-12 01:06:04 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106287(timer) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:06:05 WARN  KafkaClusterCreator:191 - Reconciliation #106291(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:06:06 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106291(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [Idle] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:06:06 WARN  KafkaClusterCreator:191 - Reconciliation #106292(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:06:08 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106292(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:06:08 WARN  KafkaClusterCreator:191 - Reconciliation #106293(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:06:10 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106293(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:08:02 WARN  KafkaClusterCreator:191 - Reconciliation #106295(timer) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:08:04 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106295(timer) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:08:05 WARN  KafkaClusterCreator:191 - Reconciliation #106299(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:08:06 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106299(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [Idle] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:08:06 WARN  KafkaClusterCreator:191 - Reconciliation #106300(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:08:08 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106300(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:08:08 WARN  KafkaClusterCreator:191 - Reconciliation #106301(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:08:10 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106301(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:10:02 WARN  KafkaClusterCreator:191 - Reconciliation #106312(timer) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:10:04 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106312(timer) Kafka(kafka/preprod): Reconciling auto-rebalance in the [RebalanceOnScaleDown] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []
2025-10-12 01:10:05 WARN  KafkaClusterCreator:191 - Reconciliation #106316(watch) Kafka(kafka/preprod): Cannot scale down brokers [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] because [16, 17, 18, 19, 10, 11, 12, 13, 14, 15] have assigned partition-replicas
2025-10-12 01:10:06 INFO  KafkaAutoRebalancingReconciler:98 - Reconciliation #106316(watch) Kafka(kafka/preprod): Reconciling auto-rebalance in the [Idle] state with scaling nodes: blocked scale down = [16, 17, 18, 19, 10, 11, 12, 13, 14, 15], added scale up = []

[scholzj]

I don't think that this has much value.

[acgtun]

yeah, just report this and see anyone else has the similar issues or not.
If we encounter the same issue next time, I will get more info

[acgtun]

likely an issue in cruise-control when removing brokers

[acgtun]

kafka CR status

│ Status:                                                                                                                                                                                                                                                                    │
│   Auto Rebalance:                                                                                                                                                                                                                                                          │
│     Last Transition Time:  2025-10-12T04:48:31.131377313Z                                                                                                                                                                                                                  │
│     State:                 Idle                                                                                                                                                                                                                                            │
│   Cluster Id:              EuR-rbY9RSusefXcID0WVA

it has the remove_broker list at that moment

[Hayajiro]

We happened to stumble upon this issue, as well. In our case, it was caused by over-restrictive network policies, so it might be worth a shot to check for this, first.