[Bug]: CruiseControlMetricsReporter cannot connect to broker after upgrading operater to 0.47.0

[kyleli666]

Bug Description

CruiseControlMetricsReporter cannot connect to broker after upgrading operater to 0.47.0

2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:08 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1072 - [Producer clientId=CruiseControlMetricsReporter] Node -1 disconnected.
2025-09-15 11:19:08 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:899 - [Producer clientId=CruiseControlMetricsReporter] Connection to node -1 (kafka-sgp-kafka-brokers/10.4.12.228:9091) could not be established. Node may not be available.
2025-09-15 11:19:08 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1255 - [Producer clientId=CruiseControlMetricsReporter] Bootstrap broker kafka-sgp-kafka-brokers:9091 (id: -1 rack: null isFenced: false) disconnected
2025-09-15 11:19:09 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:09 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] Metadata:314 - [Producer clientId=CruiseControlMetricsReporter] Rebootstrapping with [kafka-sgp-kafka-brokers/10.4.12.228:9091]
2025-09-15 11:19:09 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1072 - [Producer clientId=CruiseControlMetricsReporter] Node -1 disconnected.
2025-09-15 11:19:09 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:899 - [Producer clientId=CruiseControlMetricsReporter] Connection to node -1 (kafka-sgp-kafka-brokers/10.4.16.150:9091) could not be established. Node may not be available.
2025-09-15 11:19:09 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1255 - [Producer clientId=CruiseControlMetricsReporter] Bootstrap broker kafka-sgp-kafka-brokers:9091 (id: -1 rack: null isFenced: false) disconnected
2025-09-15 11:19:10 INFO  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1072 - [Producer clientId=CruiseControlMetricsReporter] Node -1 disconnected.
2025-09-15 11:19:10 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:899 - [Producer clientId=CruiseControlMetricsReporter] Connection to node -1 (kafka-sgp-kafka-brokers/10.4.19.165:9091) could not be established. Node may not be available.
2025-09-15 11:19:10 WARN  [kafka-producer-network-thread | CruiseControlMetricsReporter] NetworkClient:1255 - [Producer clientId=CruiseControlMetricsReporter] Bootstrap broker kafka-sgp-kafka-brokers:9091 (id: -1 rack: null isFenced: false) disconnected

Steps to reproduce

1. Install strimzi operator 0.46.1
2. create kafka resources, we can see CruiseControlMetrics metrics produced in kafka topics.
3. upgrade strimzi operator 0.47.0 and wait for kafka cluster to be finally stable
4. CruiseControlMetrics stop producing in kafka and we see above errors.

Expected behavior

CruiseControlMetrics metrics produced in kafka topics

Strimzi version

0.47.0

Kubernetes version

Kubernetes 1.31

Installation method

Helm chart

Infrastructure

Amazon EKS

Configuration files and logs

my resouces: (reposrt.sh failed with error: Could not find any processes matching : 'Thread.print')

---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: controller
  namespace: middleware
  labels:
    strimzi.io/cluster: kafka-sgp
  annotations:
    strimzi.io/next-node-ids: "[991-999]"
spec:
  replicas: 3
  roles:
    - controller
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 1Gi
        class: gp3
        deleteClaim: true
  resources:
    requests:
      cpu: "0.5"
      memory: 1Gi
    limits:
      cpu: "1"
      memory: 2Gi
  template:
    pod:
      metadata:
        labels:
          sidecar.istio.io/inject: "true"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nodeGroup
                    operator: In
                    values:
                      - Test
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/cluster: kafka-sgp
                topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/cluster: kafka-sgp
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: broker
  namespace: middleware
  labels:
    strimzi.io/cluster: kafka-sgp
  annotations:
    strimzi.io/next-node-ids: "[1-888]"
spec:
  replicas: 3
  roles:
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 30Gi
        class: gp3
        deleteClaim: true
  resources:
    requests:
      cpu: 1000m
      memory: 6Gi
    limits:
      cpu: 2000m
      memory: 8Gi
  template:
    pod:
      metadata:
        labels:
          sidecar.istio.io/inject: "true"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: nodeGroup
                    operator: In
                    values:
                      - Test
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 50
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    strimzi.io/cluster: kafka-sgp
                topologyKey: kubernetes.io/hostname
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              strimzi.io/cluster: kafka-sgp
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-sgp
  namespace: middleware
  annotations:
    strimzi.io/kraft: enabled
    strimzi.io/node-pools: enabled
spec:
  kafka:
#    version: 3.9.1
    #metadataVersion: 3.9-IV0
    version: 4.0.0
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
    config:
      # https://kafka.apache.org/documentation/#brokerconfigs
      auto.create.topics.enable: "false"
      num.partitions: 1
      # offsets.topic.replication.factor is for the offsets topic __consumer_offsets
#      offsets.topic.replication.factor: 1
      # default.replication.factor is for auto created topics
      default.replication.factor: 3
      min.insync.replicas: 1
      # max(8, vCPUs)
      num.io.threads: 8
      # max(5, vCPUs / 2)
      num.network.threads: 5
      # max(2, vCPUs / 4)
      num.replica.fetchers: 2
      connections.max.idle.ms: 1800000
      replica.lag.time.max.ms: 30000
      # socket.request.max.bytes < Java heap size, 104857600=100MiB
      socket.request.max.bytes: 104857600
      socket.receive.buffer.bytes: 102400
      socket.send.buffer.bytes: 102400
#      unclean.leader.election.enable: true
      # The minimum age of a log file to be eligible for deletion due to age
      log.retention.hours: 3
#      log.retention.minutes: 30
      log.retention.bytes: 800000000
      # The maximum size of a log segment file. When this size is reached a new log segment will be created.
      log.segment.bytes: 134217728
    rack:
      topologyKey: kubernetes.io/hostname
    metricsConfig:
      type: strimziMetricsReporter
      values:
        allowList:
          - "kafka_.*"
  cruiseControl:
    brokerCapacity:
      inboundNetwork: 125MiB/s
      outboundNetwork: 125MiB/s
    autoRebalance:
      - mode: add-brokers
      - mode: remove-brokers
    resources:
      requests:
        cpu: "0.5"
        memory: 512Mi
      limits:
        cpu: "2"
        memory: 1Gi
  kafkaExporter:
    resources:
      requests:
        cpu: 200m
        memory: 64Mi
      limits:
        cpu: 500m
        memory: 128Mi
---
# Help istio get cross-cluster pod and service dns records, and work as bootstrap while 9092 port excluded in istio.
apiVersion: v1
kind: Service
metadata:
  name: kafka-sgp-broker-list
  namespace: middleware
spec:
  selector:
    strimzi.io/broker-role: 'true'
    strimzi.io/cluster: kafka-sgp
  ports:
    - name: tcp-clients
      protocol: TCP
      port: 9092
      targetPort: tcp-clients
  clusterIP: None
  type: ClusterIP

Additional context

No response

[ppatierno]

I see you are using Istio, or am I wrong? Strimzi doesn't have any support for Istio, I am not sure it could cause troubles of any nature here.

[kyguy]

But it looks like a networking (and maybe DNS) related issue

+1

@kyleli666 Two questions:

1. Can you connect and produce messages to the cluster with other clients after the upgrade?
2. Can the clients connect to the cluster after the upgrade when Istio is disabled?

[kyleli666]

1. Can you connect and produce messages to the cluster with other clients after the upgrade?

Yes, I can. I excluded 9091 and 9092 in istio side-car.

2. Can the clients connect to the cluster after the upgrade when Istio is disabled?

I hope Istio can be utilized since it's easier and cheaper on AWS in a cross-region service mesh scenario.

But yes, I've tested that, cruse control can work in 0.47.0 without istio side-car.
I feel disappointed, but I understand the situation, thanks guys.

[kyleli666]

So there is nothing related to DNS and/or Istio (because not supported at all) with 0.47.0. You can take a look here https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.47.0 But it looks like a networking (and maybe DNS) related issue, if the DNS names in the log make sense to you.

The DNS names and IPs in the logs look correct to me.

[ppatierno]

Supporting Istio within Strimzi is not a simple thing. Istio is going to create a mesh handling encryption (with TLS) for you AFAIK (I am not Istio expert) which is the same you get with Apache Kafka deployed by Strimzi (so is it going to be TLS within TLS??). Anyway no support for Istio and no plan for it right now. I am sorry for that, but I was pretty sure that it was Istio causing the issue here.

[scholzj]

Even without the TLS in TLS it would be pretty long way to make Istio useful as it has AFAIK no support for Kafka. So it would make it pretty insecure.