[Enhancement]: support for deploying connect plugins without rebuilding custom image

[manishdusad]

Related problem

Currently only way one can deploy connect plugins is by rebuilding custom image either from base image or using connect CR. This image needs to be pushed to repository which needs to be maintained and rehydrate time to time to keep it upto date. This creates a TLM nightmare.

Suggested solution

It would be good to automated this without rebuilding custome image. Similar to the way confluent has done https://www.confluent.io/blog/declarative-connectors-with-confluent-for-kubernetes/ see “Automate connector plugins installation with CFK” section

Alternatives

Alternative this can be supported by defining the required plugin list and download it at runtime during connect startup.

Additional context

No response

[scholzj]

What is TLM nightmare?

---

In general, I do not think we plan to do anything like this:

• Sharing binaries through shared volumes is pretty nasty thing and goes against the container and security best practices.
• What you suggest likely breaks tzhe Confluent Hub licensing as well as the ideas of an open community
• We already provide other mechanisms for adding plugins to the Connect image

In the future, as the OCI Image Volume feature matures in Kubernetes we will likely add support for adding Connect plugins through OCI artifacts that way. But right now, this is still an early version in Kubernetes.

[manishdusad]

TLM- technology life cycle management. We need to keep updating and rehydrating the image.

I understand where you coming from but It doesn’t have to shared volume it can be container specific Node ephemeral volume emptyDir which has the same lifetime as a pod and jars can be loaded during init process.

Rebuilding the image every time to copy a jar/file is not ideal in my view. We have to do the same for tiered storage for packaging aiven jar.

Would appreciate if we can come up with some better optimized solution for this common problem in Strimzi. More than happy to contribute.

Thanks for all your support and guidance.

[scholzj]

TLM- technology life cycle management. We need to keep updating and rehydrating the image.

Still not sure what exactly is your problem.

I understand where you coming from but It doesn’t have to shared volume it can be container specific Node ephemeral volume emptyDir which has the same lifetime as a pod and jars can be loaded during init process.

Yes, of course it can be an empty dir. But that does not change anything on the points I made.

Rebuilding the image every time to copy a jar/file is not ideal in my view. We have to do the same for tiered storage for packaging aiven jar.

Would appreciate if we can come up with some better optimized solution for this common problem in Strimzi. More than happy to contribute.

As I said, I would expect the image volumes to be the solution for this.

[ppatierno]

I agree with Jakub's first point but ...

What you suggest likely breaks tzhe Confluent Hub licensing as well as the ideas of an open community

What do you mean by it Jakub? If it's about downloading some "different licensed" jar from Confluent Hub into the pod at runtime, isn't it the same if such a jar is used to build a custom Kafka Connect image. It's just a matter of when it happens (runtime vs buildtime). Or maybe you meant something different?

[scholzj]

What do you mean by it Jakub? If it's about downloading some "different licensed" jar from Confluent Hub into the pod at runtime, isn't it the same if such a jar is used to build a custom Kafka Connect image. It's just a matter of when it happens (runtime vs buildtime). Or maybe you meant something different?

As for the licensing ... Confluent Hub contains connectors with different licenses - from Apache 2.0 licensed connectors such as Debezium up to Confluent's own connectors under their own license and likely anything in between. In addition to that, the downloading of the binaries from there follows Confluent's own terms of use and license agreement (https://www.confluent.io/confluent-hub-terms-of-use/ and https://www.confluent.io/legal/confluent-license-agreement/).

Among other things it contains statements such as that one shall not use automated systems or software to extract data or Components from the Confluent Hub ... any direct integration into Strimzi would surely seem like an automated system to get the data. And there are some other statements we might be breaking

So license-wise, it is pretty complicated at best.

---

Please also keep in mind that for licensing purposes, there is also a difference between what:

• We can do as the Strimzi project and offer as a Strimzi feature to everyone
• What all of our users can do without any restrictions
• What some of our users can do without any restrictions

So for example, a particular Strimzi user might be allowed by the Confluent terms of use and licenses allowed to use Confluent Hub or some COnfluent licensed Kafka components because they will use it only internally. But at the same time, another user might not be allowed to use the same piece of software because they use it to provide some service to other companies.

So - even if we leave the statements like the one I copied above which likely forbid Strimzi from using the Confluent Hub aside - the question is whether we want Strimzi to have features that can be legally used only by some of its users. I think we do not want such features. And at the end that was also one of the reasons why we never touched the Confluent Schema Registry or why we wrote our own HTTP Bridge.

[ppatierno]

Totally agree Jakub. Of course, everything you wrote is valid if you use locationType: confluentHub as per the linked article. It seems they also provide a more general locationType: url where you can specify anything hosting a Connector plug in (and it's not Confluent Hub). Using this mechanism would not break Confluent related licensing but of course it would come to the other security related issues.

[scholzj]

It seems they also provide a more general locationType: url where you can specify anything hosting a Connector plug in (and it's not Confluent Hub). Using this mechanism would not break Confluent related licensing but of course it would come to the other security related issues.

Well, if you have the URL, you can use our own URL-type artifact.

[ppatierno]

Yes but in our case it's anyway for baking the downloaded jar into the image.