Kubectl top pods command is giving error "error: Metrics not available for pod". even kubectl top nodes is working

[omduttpathak]

I am using kubernetes version
Client Version: v1.22.0
Server Version: v1.24.1

I have installed metrics server by using below command and URL.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.7.2/components.yaml

I can get the output successfully using "kubectl top nodes:" command. but Unable to get the metrics on "kubectl top pods" command. getting below error.

error: Metrics not available for pod kube-system/cadvisor-6rp4r, age: 2709h18m37.384021169s

I tried with other namespace also but same error "error: Metrics not available for pod xxx"

[dashpole]

/assign @yangjunmyfm192085
/triage accepted

[yangjunmyfm192085]

Hi, @omduttpathak You can refer to the following known issues to see if they can help you.
https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md#kubelet-doesnt-report-pod-metrics

[jgoodall]

@omduttpathak -- i am having the exact same problem with the same version of metrics-server. Were you able to solve this?

[omduttpathak]

Yes @jgoodall , its working after updating cri-dockerd version to latest one.

[BartoszZawadzki]

I've got the same issue:

Kubernetes:
  Client Version: v1.29.9
  Server Version: v1.29.9

metrics-server v0.71 and v0.72

I'm using kops to build and manage cluster on AWS.

I've checked the kubelet and the metrics are available there, and I've got the following error in metrics-server logs:

E0108 07:44:02.364508       1 scraper.go:149] "Failed to scrape node" err="Get \"https://ip-172-20-43-46.eu-west-1.compute.internal:10250/metrics/resource\": dial tcp 172.20.43.46:10250: i/o timeout" node="i-086d9acf9d7015c22"
E0108 07:44:17.379642       1 scraper.go:147] "Failed to scrape node, timeout to access kubelet" err="Get \"https://ip-172-20-43-46.eu-west-1.compute.internal:10250/metrics/resource\": context deadline exceeded" node="i-086d9acf9d7015c22" timeout="10s"
E0108 07:44:32.340847       1 scraper.go:147] "Failed to scrape node, timeout to access kubelet" err="Get \"https://ip-172-20-43-46.eu-west-1.compute.internal:10250/metrics/resource\": context deadline exceeded" node="i-086d9acf9d7015c22" timeout="10s"

Metrics from all nodes are available.

I've also updated containerd and runc to the latest version, but the issue persists. Base image for all my nodes is flatcar-linux (https://www.flatcar.org/) so I'm using cgroups v2

UPDATE: I've switched base image from flatcar-linux to Ubuntu 24.04.1 LTS - didn't help.

UPDATE2: We were finally able to solve this, and the reason behind the missing metrics was a feature gate, that we've enabled PodAndContainerStatsFromCRI. After disabling the flag, things got back to 'normal'.

More info here: kubernetes/kubernetes#111276