This server is not the leader for that topic-partition .

[L040639]

{"level":"ERROR","timestamp":"2025-09-03T17:42:32.658Z","logger":"kafkajs","message":"[Connection] Response ListOffsets(key: 2, version: 3)","broker":"XXXXX","clientId":"XXXXX","error":"This server is not the leader for that topic-partition","correlationId":651829,"size":113}
{"level":"ERROR","timestamp":"2025-09-03T17:42:32.658Z","logger":"kafkajs","message":"[Connection] Response ListOffsets(key: 2, version: 3)","broker":"XXXXX","clientId":"XXXXX","error":"This server is not the leader for that topic-partition","correlationId":651829,"size":113}

we are facing above client side issue with one topic in the production we have 3 brokers and the issue topic has 3 replicas and 1 partiton , so every broker should has already a replica of the topic and the topic was deployed and configured correctly . producing and consuming is working for other topic not only one topic .

below are some broker logs getting continuously i find, but these are not errors. Need some help here .

2025-09-03 16:50:59,746 INFO [GroupCoordinator 0]: Assignment received from leader f01b6126-2681-41ce-a48d-31af52c25d3b-84f9fc9e-95a0-41c9-873f-3c8737f9ebdc for group XXXX for generation 783447. The group has 2 members, 0 of which are static. (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-4]
2025-09-03 16:50:59,753 INFO [GroupCoordinator 0]: Preparing to rebalance group XXXX in state PreparingRebalance with old generation 783447 (__consumer_offsets-4) (reason: Leader f01b6126-2681-41ce-a48d-31af52c25d3b-84f9fc9e-95a0-41c9-873f-3c8737f9ebdc re-joining group during Stable; client reason: not provided) (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-1]
2025-09-03 16:50:59,757 INFO [GroupCoordinator 0]: Stabilized group f01b6126-2681-41ce-a48d-31af52c25d3b.consent-listener-vv generation 783448 (__consumer_offsets-4) with 2 members (kafka.coordinator.group.GroupCoordinator) [data-plane-kafka-request-handler-0]

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-cluster
  namespace: <XXX>
spec:
  kafka:
    authorization:
      type: opa
      url: https://<XXX>/v1/data/kafka/allow
      allowOnError: false
      enableMetrics: true
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.9'
      connections.max.idle.ms: 1800000
    storage:
      type: persistent-claim
      size: 500Gi
      class: gp3
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: '<XXX>'
          maxSecondsWithoutReauthentication: 3600
          enableMetrics: true
          type: oauth
          validIssuerUri: '<XXX>'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-prd-cert-secret
          bootstrap:
            alternativeNames:
            - <XX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XX>.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: <XXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: <XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.9.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-streams-prd-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
            service.beta.kubernetes.io/aws-load-balancer-listener-attributes.TCP-9094: tcp.idle_timeout.seconds=5000
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>   ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XXX>
                      operator: Exists
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
  zookeeper:
    storage:
      type: persistent-claim
      class: gp3
      size: 50Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-streams-prd-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: <XX>  ##dedicated nodegroup for kafka
                      operator: Exists
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        tolerations:
          - key: "<XX>"
            operator: "Exists"
            effect: "NoSchedule"
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"
    template:
      pod:
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: node-role.kubernetes.io/worker
                      operator: Exists
        tolerations:
          - key: "<XXX>"
            operator: "Exists"
            effect: "NoSchedule"

Can someone respond on this please ?

[katheris]

@L040639 to me this indicates that the client connecting to that topic is misbehaving. So perhaps investigate the logs of that client and how it works compared to the other clients that are working.