tls connection issue .

[L040639]

Below is the kafka cluster configuration, so i have enabled 2 ports both are for internal purpose, one is internal at 9093 port and second internal is type of loadbalancer at 9094 port to replicate the dat using mirrormaker2 betwen the clusters .

So the applications are accessing inside the cluster only, using the internal kafka bootstrap address i.e namespace-cluster-kafka-bootstrap.namespace.svc:9093 , previously this is working fine now we are getting below error and message deliverie are got failed .

%3|1743150489.062|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://-cluster-kafka-bootstrap.kafka-stre]: sasl_ssl://-cluster-kafka-bootstrap..svc:9093/bootstrap: SSL handshake failed: Disconnected: connecting to a PLAINTEXT broker listener? (after 9999ms in state SSL_HANDSHAKE)
ERROR:producer.main:Message delivery failed: KafkaError{code=_MSG_TIMED_OUT,val=-192,str="Local: Message timed out"}

If i use the another bootstrap address which is exposed using a private NLB, it is working fine , but sometimes for few applications they are facing below issue with the private nlb route .

Connection error: Client network socket disconnected before secure TLS connection was established","broker":":9094","clientId":"","stack":"Error: Client network socket disconnected before secure TLS connection was established\n at connResetException (node:internal/errors:720:14)\n at TLSSocket.onConnectEnd (node:_tls_wrap:1714:19)\n at TLSSocket.emit (node:events:529:35)\n at TLSSocket.emit (node:domain:489:12)\n at endReadableNT (node:internal/streams/readable:1400:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)"}

Duly checked my tls certs , everything was fine, i am not really sure why the internal bootstrap url not working at 9093 port as 9094 port bootstrap is working .

Can you help me here @scholzj

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-streams-dev-cluster
  namespace: kafka-streams-dev
spec:
  kafka:
    authorization:
      type: opa
      url: https://opa.<XXXX>:8181/v1/data/kafka/allow
      allowOnError: false
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.9'
    storage:
      type: persistent-claim
      size: 100Gi
      class: gp3
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XXXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-dev-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: kafka-streams-dev-cert-secret
          bootstrap:
            alternativeNames:
            - dev.<XXXX>
            annotations:
              external-dns.alpha.kubernetes.io/hostname: dev.<XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          brokers:
          - broker: 0
            advertisedHost: broker-0.kafka.kubed.internal
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker-0.<XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 1
            advertisedHost: broker-1.kafka.kubed.internal
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker-1.<XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
          - broker: 2
            advertisedHost: broker-2.kafka.kubed.internal
            annotations:
              external-dns.alpha.kubernetes.io/hostname: broker-2.<XXX>.
              external-dns.alpha.kubernetes.io/access: "private"
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.9.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-streams-dev-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-scheme: internal
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-streams-dev-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-streams-dev-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      class: gp3
      size: 10Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-streams-dev-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - kafka-streams-dev-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: kafka-streams-dev-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[see-quick]

At first glance, tt seems to me like some networking issue (disruption). Could you please share logs from the components that would help us identify a potential issue? With the current description, is it hard to decide what might be the cause. Thanks