[Bug]: TLS handshake failures on internal listeners after upgrade to 0.49.x with externally managed CA – brokers present only leaf certificate, omitting full chain

[Taffarel]

Bug Description

After upgrading the Strimzi operator from 0.47.0 to 0.49.1 with an externally managed cluster CA (clusterCa.generateCertificateAuthority: false), internal TLS listeners (e.g., plaintext over TLS on port 9093) stop working for clients.

• Brokers now present only the leaf (broker) certificate during TLS handshake.

• In 0.47.0, the broker keystore included the full certificate chain (leaf + intermediate(s) + root, concatenated from the provided cluster-ca Secret).

• Clients trusting only the root/intermediate CA (standard for custom/multi-level PKI) fail validation → SSLHandshakeException / PKIX path building failed / certificate_unknown.

• No changes to Secrets, CRs, certificates, SANs, or keys (PKCS#8 confirmed).

• Rollback to 0.47.0 restores full chain presentation and fixes connectivity immediately.

• Operator reconciles normally with no TLS-related errors in logs.

• External listeners (e.g., routes/NodePort) may be similarly affected if using the same custom CA setup.

• Strimzi does not validate/enforce SANs when using external CA, so mismatch is not the cause.

This appears to be a regression in how ssl.keystore.certificate.chain is populated from Secrets when generateCertificateAuthority: false.

Steps to reproduce

1. Deploy a Kafka cluster on Strimzi 0.47.0 using an externally managed cluster CA:

• Set spec.clusterCa.generateCertificateAuthority: false

• Provide custom CA certificate(s) and private key via Secrets (e.g., my-cluster-cluster-ca-cert containing ca.crt (full chain or root+intermediate) and ca.p12 / key in PKCS#8).

• Example minimal listener config in Kafka CR

listeners:
  - name: plain
    port: 9092
    type: internal
    tls: false
  - name: tls
    port: 9093
    type: internal
    tls: true
    authentication:
      type: tls   # or scram-sha-512, etc.

• Deploy with typical 3-broker setup (replicas: 3, storage, etc.).

• Confirm internal clients connect successfully to my-cluster-kafka-bootstrap.my-ns.svc.cluster.local:9093 (using truststore with root/intermediate CA).

2. Upgrade the Strimzi operator deployment to 0.49.1 (or 0.49.0):

• Update the operator image / Helm chart / YAML to version 0.49.1.

• Do not change the Kafka CR, Secrets, certificates, or any TLS-related fields.

• Wait for operator reconciliation (brokers roll out new pods if needed).

3. Test connectivity from an internal client (e.g., pod with kcat, Java producer, or Kafka console producer):

• Use bootstrap: my-cluster-kafka-bootstrap.my-ns.svc.cluster.local:9093

• Security protocol: SSL (or SASL_SSL if auth enabled)

• Truststore: contains only root/intermediate CA (no leaf cert)

• Observe SSL handshake failure:

org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: ... PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

4. (Optional verification) Exec into a broker pod and check presented chain

5. Roll back operator to 0.47.0 → connectivity restores immediately (full chain presented again).

Expected behavior

Brokers present the full certificate chain (as in 0.47.0) when using externally provided certificates, so clients can build a trust path to their trusted root CA.

Strimzi version

0.49.1

Kubernetes version

1.27+

Installation method

Helm charts

Infrastructure

Bare-Metal, Azure

Configuration files and logs

strimzi-kafka.zip

Additional context

https://github.com/orgs/strimzi/discussions/12340

• Operator logs: normal reconciliation, no certificate errors.

• Broker logs: may show client connection attempts failing at TLS layer.

• Certificate setup: multi-level CA (e.g., Root → Intermediate → kafka-cluster-ca), common in enterprise PKI.

• Workarounds (from discussion):Rollback to 0.47.0

  • Use custom per-listener certificates with full chain embedded (spec.listeners[].brokerCertChainAndKey)

[ppatierno]

Triaged on 5.2.2026: agreed that the previous behavior should be reverted back by having the chain as it was when using P12 instead of PEMs. @tinaselenge is already working on it.

[katheris]

@Taffarel @tinaselenge has created a PR to fix this if you want to take a look #12390

[scholzj]

@tinaselenge @katheris I reopened the issues because I think there is a gap. While the fix from #12390 uses the full chain for the new cluster, it does not update the existing secrets after upgrade. So existing users would need to delete the secrets or wait for certificate renewal to actually see the change in their clusters.

[scholzj]

@tinaselenge @katheris Ok, I tried to fix this in #12403. Please have a look if it makes sense to you or if you have better ideas. Or if you think we should just let it be as is and the users who need it should delete their certificate Secrets manually to get it recreated 🤷.