API Server Memory Spikes & Master Node OOM After Upgrading Kepler (Big Cluster)

[laurall974]

Kepler Version

0.10.0 or later (Current/Supported)

Bug Description

After upgrading Kepler from v0.8.0 to a recent release v0.11.3, our production Kubernetes cluster experienced severe API server memory spikes that resulted in master node OOM kills. This happened even though Kepler was deployed only on worker nodes, and process-level metrics were disabled.
Master nodes began OOM-killing critical control-plane components, including:

• kube-apiserver
• kube-controller-manager
• CNI controllers
• Storage controllers

Once we rolled Kepler back and/or removed it, the issue disappeared.

Steps to Reproduce

Deploy a large cluster (~50+ nodes, ~2000 pods).
Deploy the latest Kepler agent on all worker nodes.
Observe master node memory usage, especially kube-apiserver.

Expected Behavior

Kepler should not trigger extreme kube-apiserver load or cause control plane OOM.
API server interactions should be rate-limited, cached, or distributed to avoid synchronized spikes.

Environment

Nodes: 54 worker nodes
Masters: 3 control-plane nodes (≈30GB RAM each)
Pods: ~2260 running pods
Kepler: Upgraded from v0.8.0 to the new release v0.11.3
Kepler placement: Only deployed on worker nodes (bare metal)

Logs and Error Messages

No Kepler logs because Kepler was removed after the incident. 
API server logs were also rolled out.

Additional Context

Prometheus also experienced a spike to ~40GB during compaction, due to Kepler process metrics creating ~1.5M time series.
We resolved this by disabling process metrics, and Prometheus stabilized.
However, even after disabling process metrics, the API server continued to crash until Kepler was fully removed.

[vimalk78]

i tried reproducing this issue in openshift.
Cluster: 5 workers and 3 master nodes
Total Pods = 1000+
Total Kepler = 50 ( using a deployment, not a daemonset)

But i didn't see any spike in spi-server memory usage. there is a small spike in "Pod WATCH/LIST Request Rate" when 50 keplers start.

[vimalk78]

Can you please tell the memory size configured in the master node?

[laurall974]

Can you please tell the memory size configured in the master node?

@vimalk78 ≈30GB RAM each

[laurall974]

Upon inspecting the Kepler v0.8.0 implementation, it appears that pod discovery was performed through the local kubelet endpoint instead of querying the kube-apiserver directly. This design reduces load on the control plane and may explain why earlier versions did not trigger the API server memory spikes we are seeing with v0.11.3.

[vimalk78]

old kepler in archived branch also used api server to get pod information
https://github.com/sustainable-computing-io/kepler/blob/archived/pkg/manager/manager.go#L53-L58

[vimalk78]

i tested again on a larger 15 ( 3 master + 12 worker) node openshift cluster. There are 2500 pods created for test, and 50 kepler pods created via a deployment.

and there no change in api server memory when keplers query api-server.

@laurall974 can you please give details of the kubernetes distribution, version etc.

[laurall974]

@laurall974 can you please give details of the kubernetes distribution, version etc.

@vimalk78 Kubernetes Version: v1.30.5
Kubernetes cluster managed by OpenStack Magnum.
There is bare-metal nodes, but the master is VM.
The CNI is calico.

[laurall974]

old kepler in archived branch also used api server to get pod information https://github.com/sustainable-computing-io/kepler/blob/archived/pkg/manager/manager.go#L53-L58

@vimalk78 I am probably missing something but I thought that by default it was set to false:

kepler/pkg/config/config.go

Line 183 in e6aceef

EnableAPIServer: getBoolConfig("ENABLE_API_SERVER", false),

[vimalk78]

Thanks for sharing the details @laurall974

Since the issue happened during upgrade, can you share the upgrade steps?
Cc: @vprashar2929

[laurall974]

Since the issue happened during upgrade, can you share the upgrade steps? Cc: @vprashar2929

@vimalk78 We updated the Kepler Helm chart version and Helm values in our Argo CD deployment.
After syncing Argo, the upgrade rolled out successfully and all bare-metal nodes were running the new version.
Post-upgrade, we encountered issues with Prometheus related to process metrics. Disabling process metrics collection resolved the problem.

And then we had the problem with master node being OOM.

[amine-lah]

Hello,

I see there is a PR open related to this issue: #2369, are there any timeline for this to be released ?

I'd be happy to help in review or testing if needed

[vimalk78]

yes, that PR addresses the issue raised in this bug. but so far we have not been able to reproduce this bug.
but i agree that if calls to api server can be avoided, we should avoid it.