Controller "error retrieving resource lock karpenter/karpenter-leader-election"

[Fran-Rg]

Description

Observed Behavior:
Controller keeps restarting with a panic crash. The panic is related to the management of the karpenter-leader-election lease:

{"level":"INFO","time":"2026-01-14T10:31:20.376Z","logger":"controller","message":"attempting to acquire leader lease karpenter/karpenter-leader-election...","commit":"f15a737"}
{"level":"INFO","time":"2026-01-14T10:31:35.638Z","logger":"controller","message":"successfully acquired lease karpenter/karpenter-leader-election","commit":"f15a737"}
...
later
...
{"level":"ERROR","time":"2026-01-14T10:32:28.778Z","logger":"controller","message":"Failed to update lock optimistically: Put \"https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/karpenter/leases/karpenter-leader-election?timeout=5s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers), falling back to slow path","commit":"f15a737"}
{"level":"ERROR","time":"2026-01-14T10:32:37.534Z","logger":"controller","message":"error retrieving resource lock karpenter/karpenter-leader-election: context deadline exceeded","commit":"f15a737"}
{"level":"INFO","time":"2026-01-14T10:32:37.535Z","logger":"controller","message":"failed to renew lease karpenter/karpenter-leader-election: context deadline exceeded","commit":"f15a737"}
{"level":"ERROR","time":"2026-01-14T10:33:07.036Z","logger":"controller","message":"error retrieving resource lock karpenter/karpenter-leader-election: Get \"https://172.20.0.1:443/apis/coordination.k8s.io/v1/namespaces/karpenter/leases/karpenter-leader-election?timeout=5s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)","commit":"f15a737"}
{"level":"INFO","time":"2026-01-14T10:33:01.720Z","logger":"controller.controller-runtime.healthz","message":"healthz check failed","commit":"f15a737","statuses":[{},{},{}]}
{"level":"INFO","time":"2026-01-14T10:33:20.988Z","logger":"controller","message":"Stopping and waiting for non leader election runnables","commit":"f15a737"}
panic: leader election lost

goroutine 239 [running]:
github.com/samber/lo.must({0x1ae3e00, 0xc0134ae9e0}, {0x0, 0x0, 0x0})
	github.com/samber/lo@v1.51.0/errors.go:55 +0x1df
github.com/samber/lo.Must0(...)
	github.com/samber/lo@v1.51.0/errors.go:74
sigs.k8s.io/karpenter/pkg/operator.(*Operator).Start.func1()
	sigs.k8s.io/karpenter@v1.8.1/pkg/operator/operator.go:246 +0x75
created by sigs.k8s.io/karpenter/pkg/operator.(*Operator).Start in goroutine 1
	sigs.k8s.io/karpenter@v1.8.1/pkg/operator/operator.go:244 +0xa5

Expected Behavior:
Lock management doesn't cause panic

Reproduction Steps (Please include YAML):
Unsure, installed karpenter-crd and karpenter both on 1.8.4. Some of my managers are fine but 1 isn't.

I checked:

> kubectl -n karpenter get lease
NAME                        HOLDER                                                            AGE
karpenter-leader-election   karpenter-666f7d78f6-2bn26_0002b9e8-4ba3-4105-a553-fe691db3451f   17h

Then deleted the lease and restarted karpenter, a new lease got created:

> kubectl -n karpenter get lease
NAME                        HOLDER                                                            AGE
karpenter-leader-election   karpenter-666f7d78f6-2bn26_f70fc46d-2cba-4acf-af3b-5d2bb5cb3061   78s

Then same crash happened.

Versions:

• Chart Version: 1.8.4
• Kubernetes Version (kubectl version): v1.34.1-eks-3025e55

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Fran-Rg]

Looks related to #4168
Controller pod is running on CapacityProvisioned=0.25vCPU 0.5GB

Is it possible to increase timeout?

[Fran-Rg]

Going to CapacityProvisioned=1vCPU 2GB and the problem looks resolved. I would have expected this to be a light footprint agent.

[jmdeal]

There are a number of reasons this can occur, it looks like in your case it was CPU starvation but it can also be API throttling.

I would have expected this to be a light footprint agent

Not really, Karpenter has to maintain a lot of internal state to perform it's scheduling simulations. It's both CPU and memory intensive. The actual requirements will scale with your cluster (e.g. number of nodes, pods, complexity of scheduling constraints, etc) but we recommend 1 vCPU and 1 Gi of memory as a starting point in our getting started guide.

[Fran-Rg]

But what causes the panic? I don't understand why getting the lease lock always succeed (first part of the logs) but updating it timeouts and cause the panic (second part)?

[github-actions]

This issue has been inactive for 7 days and is marked as "triage/solved". StaleBot will close this stale issue after 7 more days of inactivity.