[Bug]: Strimzi repeatedly reconciles services to remove externalIPs

[klasvo]

Bug Description

We have our Kafka resource configured with a listener of type loadbalancer deployed to an OpenShift cluster. As described in the OpenShift documentation, the resulting Service resource has set spec.type: LoadBalancer and thus gets an external IP injected into the manifest (spec.externalIPs: ["1.2.3.4"]).

When Strimzi does its reconcile run, it detects that the spec of the service has been modified / is different as Strimzi thinks it should be, as spec.externalIPs was added. It then removes that part; however, OpenShift immediately readds it again, so that Strimzi finds the diff again in the next run.

The actual issue happens with ExternalDNS as it is triggered when a service is changed. I.e. one run of ExternalDNS is triggered each time Strimzi does a reconcile run although nothing has changed overall.

Strimzi should ignore spec.externalIPs when it is not present in its configuration. Of course, Strimzi should reconcile spec.externalIPs when externalIPs is set in the listener configuration of type nodeport.

Steps to reproduce

Deploy a loadbalancer listener in an OpenShift / OKD cluster with working externalIP configuration and watch Strimzi reconcile the services.

Expected behavior

Strimzi ignores changes to spec.externalIPs when it is not set in its own configuration

Strimzi version

0.45.0

Kubernetes version

OpenShift 4.15

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

No response

[im-konge]

Strimzi is operator and it manages its resources. That means that if you want to do some change to the Services, PVCs etc. you need to do it using the custom resources. I don't think this is a bug TBH, it's normal and expected that Strimzi will change the Service to the configuration corresponding to the particular configuration of the CRs.

[im-konge]

Please have a look on https://strimzi.io/docs/operators/latest/configuring.html#type-GenericKafkaListenerConfiguration-schema-reference in case that there is something that would fix your issue (specifying the externalIPs etc.)
You should check Strimzi documentation for configuring the CRs and not OpenShift one (as you should not edit the Services, but use the Kafka CR as I mentioned before).

[klasvo]

The modification to spec.externalIPs is not done manually, but by OpenShift and is not configurable. So there is an incompatibility between Strimzi and OpenShift.

We already checked the Strimzi docs for alternatives and found externalIPs, which is only usable with listeners of type nodeport. However, this does not work correctly, I am preparing a bug report for that issue, as well.

I would classify this issue as a bug as Strimzi does support OpenShift in general, but does not work well in this particular case.

[scholzj]

@klasvo I think the main thing to figure out is who is assigning the External IP to the service. I don't think that is OpenShift, but some other tooling you use. But as explained by Lukas, the service is owned by Strimzi, so if this is needed, Strimzi would likely need to be extended to allow setting the external IP even for load balancers and then you would have to set the external IP in the Kafka CR.

[klasvo]

@scholzj I linked the OpenShift documentation in my original post where it is described in the 4th paragraph:

Automatic assignment of an external IP
OpenShift Container Platform automatically assigns an IP address from the autoAssignCIDRs CIDR block to the spec.externalIPs[] array when you create a Service object with spec.type=LoadBalancer set. For this configuration, OpenShift Container Platform implements a cloud version of the load balancer service type and assigns IP addresses to the services. Automatic assignment is disabled by default and must be configured by a cluster administrator as described in the "Configuration for ExternalIP" section.

Alas, setting spec.externalIP for services of type LoadBalancer does not work as one would expect: you would end up with two IP addresses, the one you specified and an auto assigned one.
This is why we have to use services of type ClusterIP (NodePort would work also) when we want to set a static IP for e.g. our ingress controllers.

[scholzj]

Automatic assignment of an external IP

OpenShift Container Platform automatically assigns an IP address from the autoAssignCIDRs CIDR block to the spec.externalIPs[] array when you create a Service object with spec.type=LoadBalancer set. For this configuration, OpenShift Container Platform implements a cloud version of the load balancer service type and assigns IP addresses to the services. Automatic assignment is disabled by default and must be configured by a cluster administrator as described in the "Configuration for ExternalIP" section.

I do not think this really reproduces for me. I do not have any issues as you describe when using load balancers on OCP.

This is why we have to use services of type ClusterIP (NodePort would work also) when we want to set a static IP for e.g. our ingress controllers.

I do not understand what this means. If you want to use Ingress, you should never use loadbalancer or node port services in Strimzi. But if you are on OCP, you should probably just use Routes which are supported by Strimzi out of the box.

[klasvo]

The ingress controllers were just an example regarding OCP and static IPs and have nothing to do with Strimzi.
In our OCP cluster we don't use Routes at all, but Ingress controllers.

Our external IPs are backed by keepalived-operator, maybe this is a difference in our setups.

[scholzj]

The ingress controllers were just an example regarding OCP and static IPs and have nothing to do with Strimzi.
In our OCP cluster we don't use Routes at all, but Ingress controllers.

Well, you can use Ingress with Strimzi as long as you use TLS passthrough. But keep in mind that the type: ingress Strimzi listeners are designed and tested with the Kubernetes Nginx Ingress and might not work with other Ingress controllers.

Our external IPs are backed by keepalived-operator, maybe this is a difference in our setups.

Ok, I guess that could be an explanation. I'm afraid I have no idea what keepalived-operator is and why would someone use it.

In any case please keep in mind that you should not expect the Server Side Apply support to be added anytime soon. The original authors abandoned it and nobody is looking into that right now as far as I know.