[Enhancement]: Client trust bundle for cluster CA

[sebastiangaiser]

Related problem

Strimzi, specifically the user operator, provides client credentials as Kubernetes secret per KafkaUser resource. When using mTLS, additionally, clients need the <cluster_name>-cluster-ca-cert for trust. This works great until it comes to a CA rotation especially when using own CA certificates and private keys. As already highlighted in the documentation, all clients must add both cluster CAs to their truststores while replacing the CA. This results in all clients would need to (manually) update their config twice which also needs to be coordinated.

Suggested solution

Build a trust bundle containing all relevant cluster CA certificates and add it to the KafkaUser Kubernetes secret. By doing this, (manual) operations for each client could be improved and minimized as the KafkaUser Kubernetes secret contains all relevant certificates.

Alternatives

1. Using trust-manager to build the trust-bundle with includeAllKeys option.
2. Build the trust bundle myself using a custom controller.

Additional context

This feature might be in relation to strimzi/kafka-access-operator#124 but for users not using KAO, having this natively would be very beneficial.

[scholzj]

I personally would be open to this. But it probably should have a proposal about the details of how to do it etc.