Kafka-Operator does not create the cluster using EKS

[sebasdelagua]

Bug Description

Hello!

We have a problem, using this yaml to create my cluster on kafka on Amazon EKS, Kubernetes v1.24.11-eks-a59e1f0:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: cluster-vf
  namespace: kafka
spec:
  kafka:
    version: 3.4.0
    replicas: 1
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.4"
    storage:
      type: ephemeral
  zookeeper:
    replicas: 1
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}

Steps to reproduce

1. kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka
2. kubectl apply -f single.yaml
3. Nothing happens

Expected behavior

When we apply the file the expected behaivour it's to create a cluster cluster-vf

Strimzi version

0.34

Kubernetes version

v1.24.11-eks-a59e1f0

Installation method

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

Infrastructure

Amazon EKS

Configuration files and logs

kubectl -n kafka logs -f strimzi-cluster-operator-58967f75c7-jnq87

2023-05-15 15:12:21 INFO  Main:341 - Health and metrics server is ready on port 8080)
2023-05-15 15:12:21 INFO  Main:250 - Waiting to become a leader
2023-05-15 15:12:22 INFO  LeaderElectionManager:119 - The new leader is strimzi-cluster-operator-58967f75c7-w2n67
2023-05-15 15:12:22 INFO  LeaderElectionManager:119 - The new leader is strimzi-cluster-operator-58967f75c7-jnq87
2023-05-15 15:12:22 INFO  LeaderElectionManager:100 - Started being a leader
2023-05-15 15:12:22 INFO  Main:238 - I'm the new leader
2023-05-15 15:12:22 WARN  PlatformFeaturesAvailability:167 - API Group route.openshift.io is not supported
2023-05-15 15:12:22 WARN  PlatformFeaturesAvailability:167 - API Group build.openshift.io is not supported
2023-05-15 15:12:22 WARN  PlatformFeaturesAvailability:167 - API Group image.openshift.io is not supported
2023-05-15 15:12:22 INFO  Main:127 - Environment facts gathered: PlatformFeaturesAvailability(KubernetesVersion=1.24,OpenShiftRoutes=false,OpenShiftBuilds=false,OpenShiftImageStreams=false)
2023-05-15 15:12:22 INFO  PodSecurityProviderFactory:43 - Found PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2023-05-15 15:12:22 INFO  PodSecurityProviderFactory:62 - Initializing PodSecurityProvider io.strimzi.plugin.security.profiles.impl.BaselinePodSecurityProvider
2023-05-15 15:12:22 INFO  ClusterOperator:94 - Creating ClusterOperator for namespace kafka
2023-05-15 15:12:22 INFO  ClusterOperator:109 - Starting ClusterOperator for namespace kafka
2023-05-15 15:12:22 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkas' with unstable version 'v1beta2'
2023-05-15 15:12:23 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnects' with unstable version 'v1beta2'
2023-05-15 15:12:23 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormaker2s' with unstable version 'v1beta2'
2023-05-15 15:12:24 WARN  VersionUsageUtils:60 - The client is using resource type 'strimzipodsets' with unstable version 'v1beta2'
2023-05-15 15:12:24 INFO  StrimziPodSetController:548 - Starting the StrimziPodSet controller
2023-05-15 15:12:24 INFO  StrimziPodSetController:517 - Starting StrimziPodSet controller for namespace kafka
2023-05-15 15:12:24 INFO  StrimziPodSetController:520 - Waiting for informers to sync
2023-05-15 15:12:24 INFO  StrimziPodSetController:525 - Informers are in-sync
2023-05-15 15:12:24 INFO  ClusterOperator:124 - Opened watch for Kafka operator
2023-05-15 15:12:24 INFO  ClusterOperator:124 - Opened watch for KafkaMirrorMaker operator
2023-05-15 15:12:24 INFO  ClusterOperator:124 - Opened watch for KafkaConnect operator
2023-05-15 15:12:24 INFO  ClusterOperator:124 - Opened watch for KafkaBridge operator
2023-05-15 15:12:24 INFO  ClusterOperator:124 - Opened watch for KafkaMirrorMaker2 operator
2023-05-15 15:12:24 INFO  ClusterOperator:136 - Setting up periodic reconciliation for namespace kafka
2023-05-15 15:12:24 INFO  Main:208 - Cluster Operator verticle started in namespace kafka without label selector
2023-05-15 15:14:24 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-05-15 15:14:24 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkamirrormakers' with unstable version 'v1beta2'
2023-05-15 15:14:25 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkabridges' with unstable version 'v1beta2'
2023-05-15 15:14:25 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkarebalances' with unstable version 'v1beta2'
2023-05-15 15:14:25 WARN  VersionUsageUtils:60 - The client is using resource type 'kafkaconnectors' with unstable version 'v1beta2'
2023-05-15 15:16:24 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-05-15 15:18:04 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Kafka cluster-vf in namespace kafka was ADDED
2023-05-15 15:18:05 INFO  AbstractOperator:239 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Kafka cluster-vf will be checked for creation or modification
2023-05-15 15:18:05 INFO  OperatorWatcher:38 - Reconciliation #2(watch) Kafka(kafka/cluster-vf): Kafka cluster-vf in namespace kafka was MODIFIED
2023-05-15 15:18:05 INFO  CrdOperator:133 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Status of Kafka cluster-vf in namespace kafka has been updated
2023-05-15 15:18:05 INFO  Ca:1135 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Generating CA with subject=Subject(organizationName='io.strimzi', commonName='cluster-ca v0', dnsNames=[], ipAddresses=[])
2023-05-15 15:18:06 INFO  Ca:1135 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Generating CA with subject=Subject(organizationName='io.strimzi', commonName='clients-ca v0', dnsNames=[], ipAddresses=[])
2023-05-15 15:18:08 INFO  Ca:399 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Generating certificate Subject(organizationName='io.strimzi', commonName='cluster-operator', dnsNames=[], ipAddresses=[]), signed by CA cluster-ca
2023-05-15 15:18:08 WARN  AbstractModel:200 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): A ZooKeeper cluster with a single replica and ephemeral storage will be in a defective state after any restart or rolling update. It is recommended that a minimum of three replicas are used.
2023-05-15 15:18:08 INFO  Ca:399 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Generating certificate Subject(organizationName='io.strimzi', commonName='cluster-vf-zookeeper', dnsNames=[cluster-vf-zookeeper-0.cluster-vf-zookeeper-nodes.kafka.svc.cluster.local, *.cluster-vf-zookeeper-client.kafka.svc, *.cluster-vf-zookeeper-nodes.kafka.svc, *.cluster-vf-zookeeper-nodes.kafka.svc.cluster.local, cluster-vf-zookeeper-client.kafka.svc, cluster-vf-zookeeper-0.cluster-vf-zookeeper-nodes.kafka.svc, *.cluster-vf-zookeeper-client.kafka.svc.cluster.local, cluster-vf-zookeeper-client.kafka.svc.cluster.local, cluster-vf-zookeeper-client.kafka, cluster-vf-zookeeper-client], ipAddresses=[]), signed by CA cluster-ca
2023-05-15 15:18:09 INFO  StrimziPodSetController:343 - Reconciliation #3(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): StrimziPodSet will be reconciled
2023-05-15 15:18:19 ERROR StrimziPodSetController:374 - Reconciliation #3(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): StrimziPodSet cluster-vf-zookeeper in namespace kafka reconciliation failed
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1107) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.maybeCreateOrPatchPod(StrimziPodSetController.java:435) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.reconcile(StrimziPodSetController.java:363) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.run(StrimziPodSetController.java:531) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.IOException: request timed out
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:533) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:347) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:704) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        ... 6 more
Caused by: java.net.http.HttpTimeoutException: request timed out
        at jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:63) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899) ~[java.net.http:?]
2023-05-15 15:18:19 INFO  StrimziPodSetController:379 - Reconciliation #3(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): reconciled
2023-05-15 15:18:19 INFO  StrimziPodSetController:343 - Reconciliation #4(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): StrimziPodSet will be reconciled
2023-05-15 15:18:24 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-05-15 15:18:29 ERROR StrimziPodSetController:374 - Reconciliation #4(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): StrimziPodSet cluster-vf-zookeeper in namespace kafka reconciliation failed
io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:129) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:122) ~[io.fabric8.kubernetes-client-api-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:44) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:1107) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.create(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.maybeCreateOrPatchPod(StrimziPodSetController.java:435) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.reconcile(StrimziPodSetController.java:363) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at io.strimzi.operator.cluster.operator.assembly.StrimziPodSetController.run(StrimziPodSetController.java:531) ~[io.strimzi.cluster-operator-0.34.0.jar:0.34.0]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.io.IOException: request timed out
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:533) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:570) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:554) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleCreate(OperationSupport.java:347) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:704) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleCreate(BaseOperation.java:93) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        at io.fabric8.kubernetes.client.dsl.internal.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:42) ~[io.fabric8.kubernetes-client-6.3.1.jar:?]
        ... 6 more
Caused by: java.net.http.HttpTimeoutException: request timed out
        at jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:63) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1270) ~[java.net.http:?]
        at jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:899) ~[java.net.http:?]
2023-05-15 15:18:29 INFO  StrimziPodSetController:379 - Reconciliation #4(watch) StrimziPodSet(kafka/cluster-vf-zookeeper): reconciled
2023-05-15 15:19:04 INFO  AbstractOperator:380 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Reconciliation is in progress
2023-05-15 15:20:04 INFO  AbstractOperator:380 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Reconciliation is in progress
2023-05-15 15:20:24 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka
2023-05-15 15:21:04 INFO  AbstractOperator:380 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Reconciliation is in progress
2023-05-15 15:22:04 INFO  AbstractOperator:380 - Reconciliation #1(watch) Kafka(kafka/cluster-vf): Reconciliation is in progress
2023-05-15 15:22:24 INFO  ClusterOperator:139 - Triggering periodic reconciliation for namespace kafka

kubectl -n kafka get all

NAME                                            READY   STATUS    RESTARTS   AGE
pod/strimzi-cluster-operator-58967f75c7-jnq87   1/1     Running   0          15m

NAME                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/cluster-vf-zookeeper-client   ClusterIP   10.100.127.89   <none>        2181/TCP                     10m
service/cluster-vf-zookeeper-nodes    ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   10m

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/strimzi-cluster-operator   1/1     1            1           16m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/strimzi-cluster-operator-58967f75c7   1         1         1       16m

Additional context

No response

[scholzj]

The error suggests the calls to your Kubernetes APi server are timing out. This could be issue of the API server. But more likely you have some unresponsive admission controller or something like that which blocks the Pod creation by running into some error or timing out.

[sebasdelagua]

We downgraded the EKS cluster to 1.23 and works. The problem it's in kubernetes 1.24 in EKS with the service account with no token associated.

There is a feature introduced in version 1.24:

Up to Kubernetes 1.23, creating a service account in a cluster results in Kubernetes automatically creating a Secret with a token for that service account. This token never expires, which can be useful but is also a security issue. Starting with Kubernetes 1.24, these Secrets will no longer be created automatically.

It can be disabled with LegacyServiceAccountTokenNoAutoGeneration, for example it will work in Digital Ocean because this feature is disabled for 1.24 K8S clusters

[scholzj]

The operator is completely fine talking with the Kubernetes API for all the other requests it sent. It's just the Pod creation which does not work. So I doubt this is a service account token issue -> without the token, the operator will not be able to start because right at the start it created the watches, informers ... and from the log, it looks like it created some secrets, configmaps, updated status of the Kafka CR and the StrimziPodSet. So why would any issue with the service account caused only the pod creation to timeout?

[sebasdelagua]

Thanks @scholzj for the response. In EKS v1.23 we faced the same issue, the problem was that the strimzi operator conflicts with this component:

---
# Source: vault/templates/injector-mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: vault-agent-injector-cfg
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
webhooks:
  - name: vault.hashicorp.com
    failurePolicy: Ignore
    matchPolicy: Exact
    sideEffects: None
    timeoutSeconds: 30
    admissionReviewVersions: ["v1", "v1beta1"]
    clientConfig:
      service:
        name: vault-agent-injector-svc
        namespace: vault
        path: "/mutate"
      caBundle: ""
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    objectSelector:
      matchExpressions:
      - key: app.kubernetes.io/name
        operator: NotIn
        values:
        - vault-agent-injector

This component it's from vault agent injector. We disabled this component and the operator works perfectly.

Thanks for you support!!

[markoftw]

We had the same issue, which was resolved after removing the vault agent injector too. Thanks for the hint!

EKS 1.25

[brsolomon-deloitte]

Hi @sebasdelagua - how did you determine that the vault-agent-injector-cfg was the source of the problem here?

We're seeing a similar strimzi error, but do not have HashiCorp Vault installed/in use at all, so are having trouble identifying what is the cause of pods not being created when strimzi requests them from kube-apiserver.

[sebasdelagua]

Sorry for the delay, we deploy the strimzi operator in a fresh environment. It works! Then we continue to add new components until we found the problem was with vault. We insert each component of vault separately until we found the problem was in the injector.

[hamedbrd]

Thank you guys. We ran into the same issue with k8s v1.31 and Strimzi v0.45 and it turned out it was another recent CRD we installed.
But not sure this is an issue in Strimzi side or others CRDs.

[sosi-andrea]

Hello @hamedbrd, could you please let us know which CRD you found the issue in? Thank you!

[sosi-andrea]

We have same issue on EKS v1.34.1-eks-3025e55 using Strimzi v.0.49.1.
We tested in a clean environment (minikube) with and without Vault installed and it worked. Should be related to EKS or another CRDs.

[sosi-andrea]

There were no external CRDs that cause the issue.
I managed to resolve it by adding to the Strimzi operator the env var KUBERNETES_REQUEST_TIMEOUT=60000 (default is 10000)

[sosi-andrea]

We figured out that the core problem was network filtering (Security Group) being too restrictive, causing a communication breakdown between the EKS and worker nodes.