Fix pod metrics in Lens for Prometheus as they display incorrect value which 2 x times bigger then actual value

[dragoangel]

What would you like to be added:
Could you add an option that will allow passing custom query parameters for metrics requests?

Why is this needed:
This is required for configuring aspects of monitoring, for example, pass timeout parameter to Prometheus. Also, not having the option to set query parameters when Lens is pointed to solutions like Thanos and Prometheus HA leading to displaying metrics wrongly. Lens will show duplicated data from HA, f.e.: pod CPU and RAM usage will be multiplied by the count of replicas in HA. To display data correctly and not fail on partial_response: dedup=1&partial_response=1 would help, but Lens does not accept query parameters in PROMETHEUS SERVICE ADDRESS unfortunately and does not have a separate field to add them.

Environment you are Lens application on:

• Kubernetes distribution: Rancher
• Metrics Prometheus type: Prometheus Operator
• Metrics Prometheus service: Thanos Query Frontend that query Prometheus in HA
• Desktop OS: Any

[dragoangel]

Hm, after testing actually I found that reason of multiplied CPU and ram usage on pod compared of sum of containers not related to thanos usage and query params, because from what I tested:

1. thanos deduplicate data by default
2. on another setup without sharding and thanos issue reproduce in same way.

So this a bug. Will try debug tomorrow per query what is wrong

lens/packages/core/src/main/prometheus/operator-provider.injectable.ts.ts

Line 6 in 2b721ab

import type { PrometheusProvider } from "./provider";

[dragoangel]

I found issue at

lens/packages/core/src/main/prometheus/operator-provider.injectable.ts.ts

Lines 71 to 95 in 2b721ab

	case "pods":
	switch (queryName) {
	case "cpuUsage":
	return `sum(rate(container_cpu_usage_seconds_total{pod=~"${opts.pods}", namespace="${opts.namespace}"}[${rateAccuracy}])) by (${opts.selector})`;
	case "cpuRequests":
	return `sum(kube_pod_container_resource_requests{pod=~"${opts.pods}", resource="cpu", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "cpuLimits":
	return `sum(kube_pod_container_resource_limits{pod=~"${opts.pods}", resource="cpu", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "memoryUsage":
	return `sum(container_memory_working_set_bytes{pod=~"${opts.pods}", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "memoryRequests":
	return `sum(kube_pod_container_resource_requests{pod=~"${opts.pods}", resource="memory", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "memoryLimits":
	return `sum(kube_pod_container_resource_limits{pod=~"${opts.pods}", resource="memory", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "fsUsage":
	return `sum(container_fs_usage_bytes{pod=~"${opts.pods}", namespace="${opts.namespace}"}) by (${opts.selector})`;
	case "fsWrites":
	return `sum(rate(container_fs_writes_bytes_total{pod=~"${opts.pods}", namespace="${opts.namespace}"}[${rateAccuracy}])) by (${opts.selector})`;
	case "fsReads":
	return `sum(rate(container_fs_reads_bytes_total{pod=~"${opts.pods}", namespace="${opts.namespace}"}[${rateAccuracy}])) by (${opts.selector})`;
	case "networkReceive":
	return `sum(rate(container_network_receive_bytes_total{pod=~"${opts.pods}", namespace="${opts.namespace}"}[${rateAccuracy}])) by (${opts.selector})`;
	case "networkTransmit":
	return `sum(rate(container_network_transmit_bytes_total{pod=~"${opts.pods}", namespace="${opts.namespace}"}[${rateAccuracy}])) by (${opts.selector})`;
	}

Need to add container!=""

@Nokel81 sorry for bothering you, but can you please check this? Thank you in advance.

[Nokel81]

I think we need to add a toggle for that, because we have added and then removed such a filter several times.

[dragoangel]

@Nokel81 you mean that previously there was such filter like sum(kube_pod_container_resource_limits{container!="", pod=~"${opts.pods}", resource="cpu", namespace="${opts.namespace}"}) by (${opts.selector}) but it was removed due to some issue?

P.s. in general adding the option to add custom query params would be nice feature, even that not as critical as I think about it initially when was creating this issue 😊

[oleksandr-selezniov]

Hi. I'm observing doubled metric plots for pods too. And I found out that container!="" won't help for me, as my cause of duplication is two datasets with different service in Prometheus output. Those services are "kubelet" and "prometeus-kube-prometheus-kubelet"
And both datasets fit the condition container!=""

[dragoangel]

Those services are "kubelet" and "prometeus-kube-prometheus-kubelet"

I think this issue with your jobs in prometheus that you collect same metrics twice then. You need properly setup your prometheus stack.

[vitaliyf]

Same as #7679

[Tantino]

Same as #7679

[jkroepke]

Container metrics are fine for me while pod metrics are 2x. It may depends on the installed CRI. On a AKS installed, container!="" would help.

Running from AKS with kube-prometheus-stack.

[dragoangel]

Container metrics are fine for me while pod metrics are 2x. It may depends on the installed CRI. On a AKS installed, container!="" would help.

Running from AKS with kube-prometheus-stack.

Yes, this exactly what I mentioned

[jkroepke]

Looking forward to #7777

[jkroepke]

I can confirm that if I run minikube with docker driver, the metric container_memory_working_set_bytes does not have an container label.

It depends on the CRI runtime, if the metric container_memory_working_set_bytes does have the container metrics or not.

@Nokel81 That could be reason why user has some trouble in history.

If container_memory_working_set_bytes has a container label, then container!="". If not, then container!="" should be not present.

#7777 would be the best solution for all.