karpenter pods CrashLoopBackOff - Readiness probe failed read: connection reset by peer / Liveness probe failed connect: connection refused

[iamsaurabhgupt]

Description

Observed Behavior:

kubectl get pods -n kube-system

NAME READY STATUS RESTARTS AGE
aws-node-dc9hb 2/2 Running 0 109m
aws-node-pzbww 2/2 Running 0 109m
coredns-789f8477df-8r5zd 1/1 Running 0 114m
coredns-789f8477df-tc5pt 1/1 Running 0 114m
eks-pod-identity-agent-gqwrz 1/1 Running 0 109m
eks-pod-identity-agent-sbng9 1/1 Running 0 109m
karpenter-df9d8f6dd-xbz9d 0/1 Running 0 118s
karpenter-df9d8f6dd-znnjw 0/1 Pending 0 118s
kube-proxy-l8bcp 1/1 Running 0 109m
kube-proxy-mnw6n 1/1 Running 0 109m

kubectl describe pod karpenter-df9d8f6dd-xbz9d -n kube-system
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
kube-api-access-n9sbj:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 3m15s default-scheduler Successfully assigned kube-system/karpenter-df9d8f6dd-xbz9d to ip-10-110-164-199.ec2.internal
Normal Pulled 75s (x2 over 3m15s) kubelet Container image "public.ecr.aws/karpenter/controller:1.0.5@sha256:f2df98735b232b143d37f0c6819a6cae2be4740e3c8b38297bceb365cf3f668b" already present on machine
Normal Created 75s (x2 over 3m15s) kubelet Created container controller
Normal Killing 75s kubelet Container controller failed liveness probe, will be restarted
Warning Unhealthy 75s kubelet Readiness probe failed: Get "http://10.xxx.1x5.153:8081/readyz": read tcp 10.xxx.1x4.1x9:33238->10.xxx.1x5.1x3:8081: read: connection reset by peer
Warning Unhealthy 75s (x2 over 75s) kubelet Readiness probe failed: Get "http://10.xxx.1x5.153:8081/readyz": dial tcp 10.xxx.1x5.153:8081: connect: connection refused
Normal Started 74s (x2 over 3m14s) kubelet Started container controller
Warning Unhealthy 5s (x5 over 2m35s) kubelet Readiness probe failed: Get "http://10.xxx.1x5.153:8081/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 5s (x4 over 2m15s) kubelet Liveness probe failed: Get "http://10.xxx.1x5.153:8081/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Expected Behavior:
karpenter pod must get to Running stage

Reproduction Steps (Please include YAML):
EKS cluster version 1.31 created using
followed both https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/
and https://karpenter.sh/docs/getting-started/migrating-from-cas/ but nothing worked

eksctl create cluster -f - <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: xxxxx
region: us-east-1
version: "1.31"
tags:
karpenter.sh/discovery: xxxxx

privateCluster:
enabled: true
skipEndpointCreation: true

iam:
withOIDC: true
podIdentityAssociations:

• namespace: "kube-system"
  serviceAccountName: karpenter
  roleName: xxxx-karpenter
  permissionPolicyARNs:
  • arn:aws:iam::xxxxxxxx:policy/KarpenterControllerPolicy-xxxx

iamIdentityMappings:

• arn: "arn:aws:iam::xxxxxxx:role/KarpenterNodeRole-xxxx"
  username: system:node:{{EC2PrivateDNSName}}
  groups:
  • system:bootstrappers
  • system:nodes

managedNodeGroups:

• instanceType: m5d.large
  amiFamily: AmazonLinux2
  name: xxxxx-ng
  desiredCapacity: 2
  minSize: 1
  maxSize: 10
  privateNetworking: true

addons:

• name: eks-pod-identity-agent
• name: coredns
• name: vpc-cni
• name: kube-proxy
  EOF

KARPENTER_VERSION = 1.0.5 (tried 1.0.6 as well but didn't work)
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace
--set "settings.clusterName=${CLUSTER_NAME}"
--set "settings.interruptionQueue=${CLUSTER_NAME}"
--set "settings.isolatedVPC=true"
--set controller.resources.requests.cpu=1
--set controller.resources.requests.memory=1Gi
--set controller.resources.limits.cpu=1
--set controller.resources.limits.memory=1Gi
--wait

tried dnsPolicy=Default but didn't work
kubectl logs karpenter-df9d8f6dd-xbz9d -n kube-system
{"level":"DEBUG","time":"2024-10-21T00:04:42.255Z","logger":"controller","caller":"operator/operator.go:149","message":"discovered karpenter version","commit":"652e6aa","version":"1.0.5"}

kubectl get events -A --field-selector source=karpenter --sort-by='.lastTimestamp' -n 100
No resources found

tried DISABLE_WEBHOOK=true but didn't work as well

Versions:

• Chart Version: 1.0.5 and 1.0.6 both don't work
• Kubernetes Version (kubectl version): 1.31

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[njtran]

There should be more logs than "discovered karpenter version", can you post those?

[pkit]

@njtran there should be, but there isn't
It literally means that liveness probe fails out of the blue for no reason. And then perfectly working pod is restarted.
It looks like it's a problem from #6637 or #2186 but it's not clear why it fails still.
In my case the shit started when I upgraded from 0.36.2 to 0.37.5
And yes, I use fargate.

[iamsaurabhgupt]

how we fixed it:

1. increase liveness probe to 1200s to avoid premature crashes without logs (it should actually be default on start)
2. when we got actual logs, we saw the actual issue was authorization error.
3. on digging more, we found some VPN/security group was rejecting the access. once allowed, it got fixed.

Recommendation to maintainers:

• increase liveness/readiness initialDelaySeconds on helm chart

          livenessProbe:
            initialDelaySeconds: 600
            timeoutSeconds: 300
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            initialDelaySeconds: 540
            timeoutSeconds: 300
            httpGet:
              path: /readyz
              port: http

[pkit]

LOL. So need to re-deploy it manually just to find out what the error is?
Nice! Thanks!

[qspors]

how we fixed it:

• increase liveness/readiness initialDelaySeconds on helm chart

          livenessProbe:
            initialDelaySeconds: 600
            timeoutSeconds: 300
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            initialDelaySeconds: 540
            timeoutSeconds: 300
            httpGet:
              path: /readyz
              port: http

Have same issues.
regarding this changes ? I have check values.yaml for last release. does't see this settings inside values.yaml
this settings is not templated for values.

Did you change it directly thru deployment settings or you added changes to chart and install it.
Cuz I do deployment for karpenter thru terraform with fargate profile and able to add changes only values existing in values.yaml

[brandonphan]

how we fixed it:

• increase liveness/readiness initialDelaySeconds on helm chart

          livenessProbe:
            initialDelaySeconds: 600
            timeoutSeconds: 300
            httpGet:
              path: /healthz
              port: http
          readinessProbe:
            initialDelaySeconds: 540
            timeoutSeconds: 300
            httpGet:
              path: /readyz
              port: http

Have same issues. regarding this changes ? I have check values.yaml for last release. does't see this settings inside values.yaml this settings is not templated for values.

Did you change it directly thru deployment settings or you added changes to chart and install it. Cuz I do deployment for karpenter thru terraform with fargate profile and able to add changes only values existing in values.yaml

@qspors Ended up using the solution for debugging and edited the deployment directly since I couldn't find any configuration in the helm chart.

[pkit]

Okay. In my case it was a simple OOM.
I.e. the default fargate_profile of 0.25cpu/0.5Gi RAM is not working for me starting from 0.37.5 (worked fine for <=0.34)
It was almost impossible to catch, only by chance I saw the error.
The solution was to set RAM to 1Gi in the helm chart values (terraform with aws-ia/eks-blueprints-addons/aws):

  karpenter = {
    chart_version = "0.37.5"
    values = [
      <<-EOT
        controller:
          resources:
            limits:
              cpu: 1000m
              memory: 1Gi
            requests:
              cpu: 1000m
              memory: 1Gi
      EOT
    ]
  }

[yaroslav-nakonechnikov]

i've also caught that on clean install 1.1.1.

logs before crash:

{"level":"ERROR","time":"2025-01-23T14:01:35.137Z","logger":"controller.controller-runtime.source.EventHandler","message":"failed to get informer from cache","commit":"3298d91","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"INFO","time":"2025-01-23T14:01:35.137Z","logger":"controller","message":"Starting workers","commit":"3298d91","controller":"operatorpkg.node.events","controllerGroup":"","controllerKind":"Event","worker count":10}
{"level":"INFO","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"Shutdown signal received, waiting for all workers to finish","commit":"3298d91","controller":"operatorpkg.node.events","controllerGroup":"","controllerKind":"Event"}
{"level":"INFO","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim"}
{"level":"INFO","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"nodeclass.hash","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass"}
{"level":"ERROR","time":"2025-01-23T14:01:35.137Z","logger":"controller.controller-runtime.source.EventHandler","message":"failed to get informer from cache","commit":"3298d91","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.137Z","logger":"controller","message":"Could not wait for Cache to sync","commit":"3298d91","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","error":"failed to wait for nodeclaim.disruption caches to sync: failed to get informer from cache: Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"error received after stop sequence was engaged","commit":"3298d91","error":"failed to wait for nodeclaim.disruption caches to sync: failed to get informer from cache: Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"INFO","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"operatorpkg.ec2nodeclass.status","controllerGroup":"karpenter.k8s.aws","controllerKind":"EC2NodeClass"}
{"level":"INFO","time":"2025-01-23T14:01:35.138Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"operatorpkg.node.events","controllerGroup":"","controllerKind":"Event"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"disruption","namespace":"","name":"","reconcileID":"6c5ad283-f55b-4e82-8f70-ecf2a434faa9","error":"disrupting via reason=\"drifted\", determining candidates, tracking PodDisruptionBudgets, Timeout: failed waiting for *v1.PodDisruptionBudget Informer to sync"}
{"level":"INFO","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"disruption"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-common"},"namespace":"","name":"prj-splunk-common","reconcileID":"238c7bba-f6a8-4a1b-a332-bff72eeaf663","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer5"},"namespace":"","name":"prj-splunk-indexer5","reconcileID":"b582eee9-ec68-4309-9ae4-d02578e11e3e","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer6"},"namespace":"","name":"prj-splunk-indexer6","reconcileID":"477605a8-7a25-413d-899e-1a55288c8ae5","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-licensemanager"},"namespace":"","name":"prj-splunk-licensemanager","reconcileID":"573ddb94-915f-44e4-adf8-c23d3f3b0eb7","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-ds"},"namespace":"","name":"prj-splunk-ds","reconcileID":"747c46d5-8f8a-4dea-8307-11cc95048e11","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer1"},"namespace":"","name":"prj-splunk-indexer1","reconcileID":"b5bd1caa-a125-4079-af11-15c9611b060a","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-shc"},"namespace":"","name":"prj-splunk-shc","reconcileID":"7196c933-6691-41e8-a36e-bd54ecc798e9","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-she"},"namespace":"","name":"prj-splunk-she","reconcileID":"695dfd41-81c0-4e0e-8325-d55d1e469401","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer2"},"namespace":"","name":"prj-splunk-indexer2","reconcileID":"57d7caa9-dad5-4397-9d48-1f0602010a3d","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer3"},"namespace":"","name":"prj-splunk-indexer3","reconcileID":"3bc8179b-f4fd-46d2-8830-b1a49a1e93de","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-indexer4"},"namespace":"","name":"prj-splunk-indexer4","reconcileID":"45d7c88a-1f19-4cb1-ac69-ba6cbbb3b779","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-splunk-clustermanager"},"namespace":"","name":"prj-splunk-clustermanager","reconcileID":"3828fb70-106f-4d75-b4f5-9b3159cc3b06","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"ERROR","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"Reconciler error","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool","NodePool":{"name":"prj-ingress"},"namespace":"","name":"prj-ingress","reconcileID":"cd71bad7-a36e-4859-b4fd-481724193d6d","error":"Timeout: failed waiting for *v1.EC2NodeClass Informer to sync"}
{"level":"INFO","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"nodepool.readiness","controllerGroup":"karpenter.sh","controllerKind":"NodePool"}
{"level":"INFO","time":"2025-01-23T14:01:35.139Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"state.pod","controllerGroup":"","controllerKind":"Pod"}
{"level":"INFO","time":"2025-01-23T14:01:35.162Z","logger":"controller","message":"All workers finished","commit":"3298d91","controller":"provisioner"}
{"level":"INFO","time":"2025-01-23T14:01:35.162Z","logger":"controller","message":"Stopping and waiting for caches","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.162Z","logger":"controller","message":"k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: watch of *v1.PodDisruptionBudget ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: watch of *v1.StorageClass ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: watch of *v1.CSINode ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: watch of *v1.PersistentVolume ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"Stopping and waiting for webhooks","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"Stopping and waiting for HTTP servers","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller","message":"shutting down server","commit":"3298d91","name":"health probe","addr":"[::]:8081"}
{"level":"INFO","time":"2025-01-23T14:01:35.163Z","logger":"controller.controller-runtime.metrics","message":"Shutting down metrics server with timeout of 1 minute","commit":"3298d91"}
{"level":"INFO","time":"2025-01-23T14:01:41.771Z","logger":"controller.controller-runtime.healthz","message":"healthz check failed","commit":"3298d91","statuses":[{},{},{}]}
{"level":"INFO","time":"2025-01-23T14:01:42.014Z","logger":"controller","message":"Wait completed, proceeding to shutdown the manager","commit":"3298d91"}
panic: failed to wait for state.daemonset caches to sync: timed out waiting for cache to be synced for Kind *v1.DaemonSet

goroutine 222 [running]:
github.com/samber/lo.must({0x38c81e0, 0xc001376e40}, {0x0, 0x0, 0x0})
        github.com/samber/lo@v1.47.0/errors.go:53 +0x1df
github.com/samber/lo.Must0(...)
        github.com/samber/lo@v1.47.0/errors.go:72
sigs.k8s.io/karpenter/pkg/operator.(*Operator).Start.func1()
        sigs.k8s.io/karpenter@v1.1.1/pkg/operator/operator.go:220 +0x75
created by sigs.k8s.io/karpenter/pkg/operator.(*Operator).Start in goroutine 1
        sigs.k8s.io/karpenter@v1.1.1/pkg/operator/operator.go:218 +0xa5

ps. 1.0.8 works as expected

[jonathan-innis]

@yaroslav-nakonechnikov Did any of the suggestions above help you? We typically see these issues when there are auth problems so I'm curious if moving to v1 somehow affected permissions assigned to the controller

[jonathan-innis]

Actually, I think I know what the issue is here. 1.1.1 has an issue because we tried to LISTEN/WATCH events on that version. 1.1.2 and 1.2.1 should fix this issue

[yaroslav-nakonechnikov]

@jonathan-innis no, for me it is not a big problem, as i downgraded installation and it works atm on 1.0