[Bug]: Brokers take too long to become ready

[Haustorius]

Bug Description

Hello,

I have Strimzi kafka setup with 3 brokers running on Azure Kubernetes Service. I am running 0.47.0 operator version with 3.9.0 kafka, previously I used 0.38.0 - 3.6.0 with zookeeper.

I have only one kafkaNodepool resource, with 3 nodes with brokers being also quorum controllers.
When running AKS upgrades for example, some brokers take way too long to become ready. And I need to restart them manually. For example one broker was taking like 40 minutes before I lost patience and restarted it.

The thing is, even with INFO logs turned on, I did not notice any actual problem, no indication of what is happening, just reconciling every so often like normal kafka broker. But it remains inactive...

Can you please help? this is a big issue for me. It was happening even on older zookeeper kafka I had.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.47.0

Kubernetes version

Kubernetes 1.33.6

Installation method

YAML files

Infrastructure

Azure Kubernetes Service

Configuration files and logs

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    version: 3.9.0
    metadataVersion: 3.9-IV0
    # The replicas field is required by the Kafka CRD schema while the KafkaNodePools feature gate is in alpha phase.
    # But it will be ignored when Kafka Node Pools are used
    resources:
      requests:
        memory: 30Gi
        cpu: "6"
      limits:
        memory: 50Gi
        cpu: "6"
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: WARN
        log4j.logger.io.strimzi: WARN
        log4j.logger.kafka: WARN
        log4j.logger.org.apache.kafka: WARN
        log4j.logger.org.apache.kafka.controller: WARN
        log4j.logger.kafka.log.LogCleaner: WARN
        log4j.appender.cleanerAppender.Threshold: WARN
    authorization:
      type: simple
      superUsers:
        - admin
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: scram-sha-512
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: scram-sha-512
        configuration:
          bootstrap
            annotations:
              service.beta.kubernetes.io/azure-load-balancer-internal: "true"
              service.beta.kubernetes.io/azure-load-balancer-internal-subnet: <redacted>
              service.beta.kubernetes.io/azure-load-balancer-ipv4: <redacted>
          brokers:
            - broker: 0
              annotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
                service.beta.kubernetes.io/azure-load-balancer-internal-subnet: <redacted>
                service.beta.kubernetes.io/azure-load-balancer-ipv4: <redacted>
              advertisedHost: <redacted>
            - broker: 1
              annotations:
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
                service.beta.kubernetes.io/azure-load-balancer-internal-subnet: <redacted>
                service.beta.kubernetes.io/azure-load-balancer-ipv4: <redacted>
              advertisedHost: kafka-broker-1-prod.cml.cetin
            - broker: 2
                service.beta.kubernetes.io/azure-load-balancer-internal: "true"
                service.beta.kubernetes.io/azure-load-balancer-internal-subnet: <redacted>
                service.beta.kubernetes.io/azure-load-balancer-ipv4: <redacted>
              advertisedHost: kafka-broker-2-prod.cml.cetin
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.9"
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  entityOperator:
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: agentpool
                      operator: In
                      values:
                        - kafka
                        - tempkafka
    # tlsSidecar: # (28)
    #   resources:
    #     requests:
    #       cpu: 200m
    #       memory: 64Mi
    #     limits:
    #       cpu: 500m
    #       memory: 128Mi
    topicOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      logging: # (29)
        type: inline
        loggers:
          rootLogger.level: INFO
      # resources:
      #   requests:
      #     memory: 512Mi
      #     cpu: "1"
      #   limits:
      #     memory: 512Mi
      #     cpu: "1"
    userOperator:
      watchedNamespace: kafka
      reconciliationIntervalMs: 60000
      logging: # (30)
        type: inline
        loggers:
          rootLogger.level: INFO
      # resources:
      #   requests:
      #     memory: 512Mi
      #     cpu: "1"
      #   limits:
      #     memory: 512Mi
      #     cpu: "1"

  kafkaExporter:
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: agentpool
                      operator: In
                      values:
                        - kafka
                        - tempkafka
  # cruiseControl:
  #   metricsConfig:
  #     type: jmxPrometheusExporter
  #     valueFrom:
  #       configMapKeyRef:
  #         name: cruise-control-metrics
  #         key: metrics-config.yml
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    # See https://github.com/prometheus/jmx_exporter for more info about JMX Prometheus Exporter metrics
    lowercaseOutputName: true
    rules:
    # Special cases and very specific rules
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    # Some percent metrics use MeanRate attribute
    # Ex) kafka.server<type=(KafkaRequestHandlerPool), name=(RequestHandlerAvgIdlePercent)><>MeanRate
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    # Generic gauges for percents
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
      labels:
        "$4": "$5"
    # Generic per-second counters with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
    # Generic gauges with 0-2 key/value pairs
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
    # Emulate Prometheus 'Summary' metrics for the exported 'Histogram's.
    # Note that these are missing the '_sum' metric!
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
        quantile: "0.$8"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        quantile: "0.$6"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        quantile: "0.$4"
    # KRaft mode: uncomment the following lines to export KRaft related metrics
    # KRaft overall related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    - pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
      name: kafka_server_raftmetrics_$1
      type: COUNTER
    - pattern: "kafka.server<type=raft-metrics><>(.+):"
      name: kafka_server_raftmetrics_$1
      type: GAUGE
    # KRaft "low level" channels related metrics
    # distinguish between always increasing COUNTER (total and max) and variable GAUGE (all others) metrics
    - pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
      name: kafka_server_raftchannelmetrics_$1
      type: COUNTER
    - pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
      name: kafka_server_raftchannelmetrics_$1
      type: GAUGE
    # Broker metrics related to fetching metadata topic records in KRaft mode
    - pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
      name: kafka_server_brokermetadatametrics_$1
      type: GAUGE
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka-pool
  namespace: kafka
  labels:
    strimzi.io/cluster: strimzi
spec:
  replicas: 3
  roles:
    - broker
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 1Ti # old kafka was 800Gi, kraft needs a bit more
        deleteClaim: false
  resources:
    requests:
      memory: 30Gi
      cpu: "6"
    limits:
      memory: 50Gi
      cpu: "6"
  template:
    pod:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: agentpool
                    operator: In
                    values:
                      - kafka
                      - tempkafka

events from strimzi-kafka-pool-1:
`Events:
  Type     Reason                  Age                 From                     Message
  ----     ------                  ----                ----                     -------
  Normal   Scheduled               32m                 default-scheduler        Successfully assigned kafka/strimzi-kafka-pool-1 to aks-kafka-27046106-vmss000003
  Warning  FailedAttachVolume      32m                 attachdetach-controller  Multi-Attach error for volume "pvc-db657150-a816-4669-b32d-2797b5220d22" Volume is already exclusively attached to one node and can't be attached to another
  Normal   SuccessfulAttachVolume  30m                 attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-db657150-a816-4669-b32d-2797b5220d22"
  Normal   Pulling                 30m                 kubelet                  spec.containers{kafka}: Pulling image "quay.io/strimzi/kafka:0.47.0-kafka-3.9.0"
  Normal   Pulled                  30m                 kubelet                  spec.containers{kafka}: Successfully pulled image "quay.io/strimzi/kafka:0.47.0-kafka-3.9.0" in 7.954s (7.954s including waiting). Image size: 390908086 bytes.
  Normal   Created                 30m                 kubelet                  spec.containers{kafka}: Created container: kafka
  Normal   Started                 30m                 kubelet                  spec.containers{kafka}: Started container kafka
  Warning  Unhealthy               6s (x194 over 30m)  kubelet                  spec.containers{kafka}: Readiness probe failed:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0    34    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 503
`

`2026-02-03 10:22:39 INFO  KafkaRoller:383 - Reconciliation #1(watch) Kafka(kafka/strimzi): Will temporarily skip verifying pod strimzi-kafka-pool-1/1 is up-to-date due to io.strimzi.operator.cluster.operator.resource.KafkaRoller$UnforceableProblem: Pod strimzi-kafka-pool-1 is not ready because the Kafka node is performing log recovery. There are 255 logs and 1 segments left to recover., retrying after at least 32000ms
2026-02-03 10:22:39 ERROR VertxUtil:134 - Reconciliation #1(watch) Kafka(kafka/strimzi): Exceeded timeout of 300000ms while waiting for Pods resource strimzi-kafka-pool-1 in namespace kafka to be ready
2026-02-03 10:22:39 WARN  KafkaRoller:982 - Reconciliation #1(watch) Kafka(kafka/strimzi): Error waiting for pod kafka/strimzi-kafka-pool-1 to become ready: io.strimzi.operator.common.TimeoutException: Exceeded timeout of 300000ms while waiting for Pods resource strimzi-kafka-pool-1 in namespace kafka to be ready`

only warnings/errors I get on the brokers themselves are:
`java.util.concurrent.CompletionException: org.apache.kafka.common.errors.NotControllerException: The active controller appears to be node 2.
    at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
    at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
    at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:636)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
    at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
    at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:884)
    at org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:875)
    at org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:153)
    at org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:142)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:215)
    at org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:186)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.errors.NotControllerException: The active controller appears to be node 2.
2026-02-03 10:21:33,549 WARN [QuorumController id=1] Performing controller activation. Loaded ZK migration state of NONE. This is expected because this is a de-novo KRaft cluster. (org.apache.kafka.controller.QuorumController) [quorum-controller-1-event-handler]

KafkaConfig) [main]
2026-02-03 10:16:38,351 WARN inter.broker.protocol.version is deprecated in KRaft mode as of 3.3 and will only be read when first upgrading from a KRaft prior to 3.3. See kafka-storage.sh help for details on setting the metadata.version for a new KRaft cluster. (kafka.server.KafkaConfig) [main]`

Additional context

No response

[scholzj]

You should start by sharing the full logs (i.e., from the beginning) from the operator as well as the Kafka nodes, and try to describe the steps properly when it happens. If the Pods take 40 minutes to get ready, it is often because they were not correctly terminated in the first place and need to recover the data / rebuild the indexes at startup. So if that is the case, you would need to investigate that as well.

[Haustorius]

Unfortunately it is from my production cluster with sensitive information and I am uncomfortable sharing full logs, but I can better explain the steps..

I run Strimzi on Azure Kubernetes Service cluster. When running orchestrator version upgrade of control plane and nodepools, a temporary node is created by ARM and old node is drained and all pods redeployed to the new node with higher version. Pods are terminated by webhooks handled by Azure. There are 4 cycles of these drains for my 3 nodes.
I ran this upgrade procedure multiple times and from my observation:
first three redeploys are OK most of the time, 4th redeploy of broker (last one) gets stuck. Today first broker took about 6 minutes to be up, second got stuck.
Obviously for clusters with less data on disks, this is not a problem and it cannot be reproduced. I have around 500GB of data on each disk.
When the node gets drained, there is the multiattach PVC error, but resolves itself in around 5-10 minutes maximum.
I tried to deploy strimzi drain cleaner utility tool on my other environment but that is much smaller in disk size unfortunately, so don't think I can reproduce this issue easily.

[scholzj]

I'm afraid that without the logs, nobody will be able to give you exact answers. Just some options and maybes. While it is understandable that you cannot share logs from your production environment, you might need to setup a test environment from which you can share all the logs and reproduce the problems there.

Some generic comments on what you shared:

• Drain Cleaner is something you should definitely use from how you describe it. Without it, you can suffer from availability issues. But without the logs, it is not obvious if it explains the problem or not.
• Issues with attaching/detaching the PVCs again sound like something that might be causing the brokers to be slow at start because they suggest there might not have been a clean shutdown. However, that is infrastructure issue, so you would need to investigate that in your infra to understand the cause and implications of it..

[Haustorius]

Thank you for your response, I understand without logs it is impossible to help. I will try to reproduce on my dev environment and get back here with logs.