Use PEM files directly in operands instead of PKCS12

[tinaselenge]

We use PEM files to connect to Kafka from the most operators. The Strimzi generated certificates are also in PEM format. However, other operands such as KafkaConnect, MirrorMaker and KafkaBridge still use PKCS12 certificates to connect to Kafka. Also Kafka nodes use PKCS12 certificates to connect to each other. These operands have shell scripts to prepare PKCS12 certificates based on the volume mounted PEM files from secret sources.

We should use PEM files across all the operands for consistency and remove usage of manually generated PKCS12 certificates.
This also might give us an opportunity to reduce usage of shell script for manual certificate preparation.

[tinaselenge]

Hi @scholzj @katheris @ppatierno @fvaleri

While working on refactoring KafkaConnect and Kafka to use PEM files instead of PKCS12, I have identified a few points that need to be discussed first. Before moving forward with the PRs for them, I would like to hear your thoughts on them.

One of the points is related to the multiple secret sources for trusted certficates. I know we discussed this on the PR #11198 but I seem to be going back and forth on this, sorry I am bringing it up again.

Point 1

• We allow users to specify multiple secret sources for truststore in KafkaConnect. Currently, we generate a PKCS12 file combining all the volume mounted PEM files from secret sources.
  The PR Use PEM certificates loaded from secrets for KafkaConnect #11198 resolved this by creating an internal secret, copying data of multiple secret sources into it. But then how about when OAuth is used which we also allow multiple secret sources for trusted certificates? (The issue applies to MirrorMaker and KafkaBridge as well).
• For Kafka, we have predetermined names for certificate secrets, so that would be used for the secret config provider to access them but again how about when OAuth, KeyCloak, or OPA authorization is used? We seem to allow multiple secret sources for trusted certificates for them too.

So I'm not sure creating an internal secret for all of these would work. There are 2 options in how we might want to solve this:

1. Use the shell script to combine certificates from mounted secrets into a single .pem file (instead of generating PKCS12). If we go with this option, we don't get rid of the shell script for manually preparing tls, however I don't think that's the primary goal of this issue. Although it would be nice to rely less on scripts.

2. Make Kubernetes Secret Config Provider allow fetching multiple secrets and return combined value (similar to how it combines values of multiple keys from a secret). I'm not sure, how complicated this would be.

I'm now leaning towards option 1, because it seems to be less complicated to deal with in code, as most we need to do is to adjust ssl configurations (e.g. switch ssl.trusstore.type and ssl.keystore.type configurations to PEM).

Point 2

(If we don't go with solution 1)
In PR #11198, I removed mounting of cert secrets because these secrets are accessed via the secret Config Provider directly. But for Kafka, we have to leave these mounted because KafkaAgent configures the HTTPS server with them (it currently uses PKCS12 certificates so will have to change to use the PEM certs instead).

Also I’m not sure how changing volume mounts would have an impact on how pods are rolled when secret for certificates is updated (either secret is changed or is updated with new key and data). Especially when certificates are managed by users, they don’t have Strimzi’s ability to roll pods when needed. However, the volume mount is part of the pod definition so pods would be rolled automatically when user make changes to the secrets. If we remove the volume mounts, users would need to manually roll the pods.

So in general, maybe it’s best to leave these secrets mounted, regardless of whether we decide to use the secret config provider or not.

[fvaleri]

Point 1:

But then how about when OAuth is used which we also allow multiple secret sources for trusted certificates?

Maybe I'm missing something. Why can't we use the same solution? I'm not familiar with OAuth configuration, but the user should set the custom secret names somewhere, so the operator can collect them into a single "bundle" secret and use the glob pattern support of the Kubernetes config provider (.*).

---

Point 2:

Also I’m not sure how changing volume mounts would have an impact on how pods are rolled when secret for certificates is updated (either secret is changed or is updated with new key and data).

I think we would track all user-provided secrets that are configured in Strimzi, so we would need to add some logic to detect any update. Once an update is detected, the operator can update the Strimzi-manged "bundle" secret and roll the pods using it.

[scholzj]

Point 1

• We allow users to specify multiple secret sources for truststore in KafkaConnect. Currently, we generate a PKCS12 file combining all the volume mounted PEM files from secret sources.
  The PR Use PEM certificates loaded from secrets for KafkaConnect #11198 resolved this by creating an internal secret, copying data of multiple secret sources into it. But then how about when OAuth is used which we also allow multiple secret sources for trusted certificates? (The issue applies to MirrorMaker and KafkaBridge as well).
• For Kafka, we have predetermined names for certificate secrets, so that would be used for the secret config provider to access them but again how about when OAuth, KeyCloak, or OPA authorization is used? We seem to allow multiple secret sources for trusted certificates for them too.

So I'm not sure creating an internal secret for all of these would work. There are 2 options in how we might want to solve this:

1. Use the shell script to combine certificates from mounted secrets into a single .pem file (instead of generating PKCS12). If we go with this option, we don't get rid of the shell script for manually preparing tls, however I don't think that's the primary goal of this issue. Although it would be nice to rely less on scripts.
2. Make Kubernetes Secret Config Provider allow fetching multiple secrets and return combined value (similar to how it combines values of multiple keys from a secret). I'm not sure, how complicated this would be.

I'm now leaning towards option 1, because it seems to be less complicated to deal with in code, as most we need to do is to adjust ssl configurations (e.g. switch ssl.trusstore.type and ssl.keystore.type configurations to PEM).

I guess the best would be to explain why do you think this would not work for the other things. You could simply have multiple internal Secrets, or maybe even better, re-use the same Secret and use specific keys to distinguish the certificates (e.g. kafka-trusted-*.crt would be used for Kafka clients, oauth-*.crt for OAuth, etc.)

If you would not want to do this, maybe you can consider an alternative 1b -> have the certificates collected by the operator, merged into a single file and then passed along as a single key in the Secret to avoid the shell scripts.

Point 2

(If we don't go with solution 1) In PR #11198, I removed mounting of cert secrets because these secrets are accessed via the secret Config Provider directly. But for Kafka, we have to leave these mounted because KafkaAgent configures the HTTPS server with them (it currently uses PKCS12 certificates so will have to change to use the PEM certs instead).

This sucks as we need to keep the logic in the scripts I guess? Also, we would not be able to reload the certificates dynamically in the future this way, because even if we do it for the brokers, we would need to roll the pods for the agent. So maybe we should think about possible solutions for that as well 🤔.

Also I’m not sure how changing volume mounts would have an impact on how pods are rolled when secret for certificates is updated (either secret is changed or is updated with new key and data). Especially when certificates are managed by users, they don’t have Strimzi’s ability to roll pods when needed. However, the volume mount is part of the pod definition so pods would be rolled automatically when user make changes to the secrets. If we remove the volume mounts, users would need to manually roll the pods.

Actually, the volume mount itself would roll the pod only if it uses new / different secret. But not when it just updates the secret already used. We already do checksums of the used certificates and use them as Pod annotation to trigger the rolling updates. You can use the same here as well. And if we today don't do it for some, we should introduce it. (I think for Keycloak and OPA, we might rely only on the environment variable with secret names and keys that are used - that is not 100% solution)

[tinaselenge]

Thanks @scholzj @fvaleri!

So I think creating an internal secret for OAuth would work too, sorry I was trying to question if we want to manage multiple internal secrets. However, from what you both are saying, this is not a concern. I also like the idea of just bundling all the secrets into a single internal secret distinguishing them with specific key pattern.

We already do checksums of the used certificates and use them as Pod annotation to trigger the rolling updates. You can use the same here as well. And if we today don't do it for some, we should introduce it.

Agreed. We do the checksum for TLS trusted certificates, client cert, oauth cert, password secrets in KafkaConnect operator but for Kafka, I'm not sure about OAuth, KeyCloak and OPA, I will look into them.

So am I right thinking that the auth hash is sufficient to track the certificate changes in the secrets? For example, if any certificates changed in the existing secret, the internal secret will be reconciled with the change and the auth hash will change to roll pods. If any new secrets are added, as @fvaleri said, the operator will bundle the certificates from the new secrets to the internal secret. And then a new auth hash is calculated based on the new certificates to roll pods.

This sucks as we need to keep the logic in the scripts I guess? Also, we would not be able to reload the certificates dynamically in the future this way, because even if we do it for the brokers, we would need to roll the pods for the agent. So maybe we should think about possible solutions for that as well 🤔.

Yeah I will have to put more thoughts into this.

[ppatierno]

So am I right thinking that the auth hash is sufficient to track the certificate changes in the secrets? For example, if any certificates changed in the existing secret, the internal secret will be reconciled with the change and the auth hash will change to roll pods. If any new secrets are added, as @fvaleri said, the operator will bundle the certificates from the new secrets to the internal secret. And then a new auth hash is calculated based on the new certificates to roll pods.

Yeah I think so. You should be covered there.

Yeah I will have to put more thoughts into this.

I guess we have some thoughts during lunch at KubeCon :-). Can you report the findings here?