KafkaConnect - Introduce annotation to ignore KafkaConnector state

[swapsCAPS]

Hi all!

Would it be possible to start, stop and pause Connectors by using the Connect API while still using strimzi.io/use-connector-resources: true?
The operator will currently set whatever state is configured on the KafkaConnector, so if a user uses the Connect API, or a tool like AKHQ to change the Connector state it will quickly be reset to its original state in the spec.

Could a new annotation be introduced like strimzi.io/sync-connector-resources-state: false (which would default to true) to tell the operator to ignore whatever state the KafkaConnector has and just sync the rest of the spec. I could also imagine some sort of enum like strimzi.io/use-connector-resources: true | "full" | "spec_only" | "etc."

I can see how this would be potentially confusing, where an operator is managing some part of a resource, but isn't managing another... However, it would make it easier to pause and resume Connectors for folks who don't have access to the k8s cluster.

Would love to hear your thoughts on this

[scholzj]

I do not think there are plans for anything like that.

[swapsCAPS]

Regardless of if there are plans at the moment. Would it be a desirable feature and good practice architecturally? Wouldn't mind investing some time on this, but of course would like to know if something like this would even be accepted.
Was not sure if I should use this Discussions - Ideas section, or immediately create an [Enhancement] issue in strimzi-kafka-operator.

[scholzj]

I guess it depends on what kind of answer you want. In the discussions, you will get informal answers from different people. My answer wasn't exactly direct, sorry for that. But it can also be written as I personally do not think this is an interesting idea.

If you open an enhancement, it will go through a more formal triage. It might be rejected, accepted, or request a proposal. For a feature like this, I would not expect it to be accepted right away, as this would be a significant change. So it would be either rejected or request a proposal. The proposal is a more formal process than the triage. So assuming it is not rejected, you will need to write a proposal to describe the benefits, use cases, but also the technical details of how will it be implemented. And the proposal will be voten upon to decide if it will be accepted or not.

[scholzj]

(It is, of course, up to you on how you want to proceed. You can also simply open the proposal right away. But the triage might give you an indication of the chances of having the proposal accepted.)

[scholzj]

Also to clarify why I do not think this is a great idea -> it adds a lot of complexity and confusion to how the connector operator works. The complexity also means a lot of effort in the long term. Also with tools such as AKHQ, any change you do will be reset by Strimzi. Whether it pausing a connector, changing its settings, recnfiguring a Kafka broker. So this would create a very different pattern of how Strimzi works.

[swapsCAPS]

Thanks @scholzj. I think we share some of the same worries. From a high level, having two systems managing the same resource seems like an error prone process. And would increase the amount of future edge cases to handle. That's also exactly why I was looking for some feedback on the idea. I am not an expert on k8s operators and was unsure if this is common practice.
Thanks for the clarification on the Strimzi proposal process and for the informal triage process 🙏🏻 😁

Cheers!

[reidmeyer]

I share @swapsCAPS desire for this feature.

I think it is fair to want to have the operator not manage the "state" of the connector. That should be something that the connect cluster itself does.

I think having an option to say: ignore connector state syncing, would be nice.

The current problem:
We manage connectors through infra as code in github. Argocd syncs them to the k8s cluster, and the operator deploys them. We have a situation where we want to shut down our connectors at night, since the databases also go down at night. Typically, when using confluent kafka connect platform, we had a cronjob that would send api calls to the connect cluster to pause the connectors. With Strimzi, right now, the only option I can think of is doing some type of workflow that automatically commits to the main branch every day, the pause annotation. You can imagine this might not be my favorite solution.

Do you have any other thoughts on how we could solve this problem, @scholzj? If not, I will open a proposal if you think that's best.

PS: I enjoyed your talk on kraft migration at Axual in Utrecht a few years ago.

[scholzj]

I can understand that the scenario you are describing is a bit tricky.

---

That said, I think it is still pretty questionable thing to do ...

• I do not think the implementation is completely simple as the operator needs to understand the state of the connector
• I do not really like the idea of yet another completely different mode for running Connectors:
  • We have the one where you can use the REST API to manage connectors
  • We have the KafkaConnector model where you use the declarative mode
  • This would add a third one which is somewhere in between
• The approach does not really scale. We cannot have a special mode for everyone who wants to configure some part in a different way.

My advice if you want to continue with this: I think the best start would be to open an enhancement issue with a rough description and wait for the triage (the triage usually happens during the community calls - https://github.com/strimzi/strimzi-kafka-operator/blob/main/README.md#strimzi-community-meetings - so you can join it and be part of the discussion if you want). While it is unlikely that this could be approved without a proposal, the triage will (unlike this discussion) get the attention of more maintainers and will tell you what are the chances the proposal would have. If it does not pass the triage, it is unlikely the proposal would have a chance. So it might save you time from writing a proposal that nobody is really interested in accepting.

---

Also, have you considered some alternatives? E.g. something like simply scaling the Connect cluster to 0 instead of stopping the connectors etc.?

[reidmeyer]

Thank you for taking the time to respond.

I will most likely open the enhancement issue and wait for the triage. I'm curious what others think and if there are any other unique ideas.

It is a good idea to think about scaling down the connect cluster, but we have a multi tenant system, so it does not work for that. Only certain connectors need to go down at night.

If I wanted to use the rest api model instead of the declarative model, I would have to create workflows to make sure that the connectors are added to the connect cluster when they are committed to the repo, effectively recreating the argocd->declarative flow. This is not impossible, but you can imagine not desired.

Indeed it is a bit of an in between, but I think it's also reasonable that the actual state of the connector, i.e. whether it is running or stopped, is something different than the configuration of the connector, and is something we might not want to manage through declaration.