Use PEM certificates loaded from secrets for KafkaConnect

[tinaselenge]

Type of change

Select the type of your PR

• Refactoring

Description

• TLS certificates are loaded from secrets directly using KubernetesSecretConfigProvider, therefore also removes the population of the certificate related environment variables and the script for preparing TLS.
• Use key pattern (*.crt) of KubernetesSecretConfigProvider to load multiple certificates from a single secret for ssl.truststore.certificates configuration. OAuth truststore is however configured differently, because multiline line certificates in Jaas config is not parsed correctly. Instead it will continue to use ssl.truststore.location which maps to the given secret's volume mount path for the certificate in PEM format.
• Add RBAC right for reading secrets for KafkaConnect service account.

Resolves part of #11294

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[tinaselenge]

@scholzj @ppatierno @katheris can one of you please kick off the regression tests? I ran some of the relevant ST locally which seemed to pass, but there are many I haven't run so would like to try running the full suite. Thanks!

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tinaselenge]

thanks @scholzj for kicking off the tests. Looks like the failing tests are not related to this PR (time out failures in CruiseControlST which doesn't deploy any connect resource).

I will mark this PR ready for review now :) .

[scholzj]

I left some nits, but otherwise it mostly looks good.

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[im-konge]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tinaselenge]

I created an issue for this PR to discuss some of the points that would also apply when making the other operands use PEM files as well. I thought it would be easier to discuss in an issue than the PR.

[ppatierno]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ppatierno]

LGTM. Thanks Tina!

sorry If I approved but didn't notice your comment on #11294 so I guess we need more discussion.

[tinaselenge]

@ppatierno @katheris @fvaleri I have made a change to create an internal secret for trusted certificates for OAuth server when OAuth is used. Can you please review the new change when you have time? Thank you.

Can someone also please kick off the regression tests? Thanks!

[ppatierno]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[tinaselenge]

I'm having an issue with oauth truststore configuration. It seems like Jaas config for the oauth, doesn't like new lines of PEM format certificates. It wants everything inline I think:

Caused by: java.lang.IllegalArgumentException: Value not specified for key 'MIIDZTCCAk2gAwIBAgIUYB3lcFeRQUakgAHV6ioDogGuHmAwDQYJKoZIhvcNAQEL' in JAAS config

I also hit different issue with kafka-client 4.0 jar:

2025-04-08 23:14:10 DEBUG [main] CompositeStrategy:56 - Unable to load modern Subject methods, relying only on legacy methods
java.lang.NoSuchMethodException: javax.security.auth.Subject.current()
	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2675) ~[?:?]
	at org.apache.kafka.common.internals.ModernStrategy.<init>(ModernStrategy.java:43) ~[kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.internals.CompositeStrategy.<init>(CompositeStrategy.java:51) [kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.internals.CompositeStrategy.<clinit>(CompositeStrategy.java:39) [kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.common.internals.SecurityManagerCompatibility.get(SecurityManagerCompatibility.java:38) [kafka-clients-4.0.0.jar:?]
	at org.apache.kafka.connect.runtime.isolation.ClassLoaderFactory.newDelegatingClassLoader(ClassLoaderFactory.java:30) [connect-runtime-4.0.0.jar:?]
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74) [connect-runtime-4.0.0.jar:?]
	at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:65) [connect-runtime-4.0.0.jar:?]
	at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121) [connect-runtime-4.0.0.jar:?]
	at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:95) [connect-runtime-4.0.0.jar:?]
	at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:112) [connect-runtime-4.0.0.jar:?]

Looks it might be Java version incompatibility.

So I'm looking into how to solve this issue. Maybe oauth.ssl.truststore.location in Oauth Jaas config is less complex way to configure it, but we need to combine the trusted certificates into a single file and volume mount it. I don't know if this is too complex, but we could copy oauth trusted cert secrets into an internal secret under a single key and volume mount this internal secret.