kube-state-metrics Metrics Handler Failure with "context canceled" Error #2579
Description
Description:
The metrics handler in kube-state-metrics fails to start, resulting in a "context canceled" error. Despite this error, the service does not crash and continues to serve the metrics collected at the time of the failure. The issue occurs during the auto-detection of sharding settings.
Environment:
Deployment method: Helm chart
Chart version: 5.19.0
Chart reference: prometheus-community/kube-state-metric
Steps to Reproduce:
Deploy kube-state-metrics using the specified Helm chart version 5.19.0.
Monitor the logs of the kube-state-metrics pod.
Observe the "context canceled" error in the logs.
Expected Behavior:
The metrics handler should start successfully and continue collecting and serving metrics without interruption.
Actual Behavior:
The metrics handler fails to start, and the following error is logged:
E1212 20:07:19.015489 1 discovery.go:253] "failed to run metrics handler" err="context canceled"
However, the service does not crash and continues to serve the metrics collected at the time of the failure.
Logs:
I1212 20:07:16.096580 1 config.go:84] "Using custom resource plural" resource="autoscaling.k8s.io_v1_VerticalPodAutoscaler" plural="verticalpodautoscalers"
I1212 20:07:16.096635 1 config.go:209] "reloaded factory" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:16.296547 1 builder.go:208] "Updating store" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:16.296612 1 discovery.go:274] "discovery finished, cache updated"
I1212 20:07:16.298253 1 metrics_handler.go:106] "Autosharding enabled with pod" pod="kube-prometheus/kube-state-metrics-0"
I1212 20:07:16.298283 1 metrics_handler.go:107] "Auto detecting sharding settings"
I1212 20:07:19.014737 1 config.go:84] "Using custom resource plural" resource="autoscaling.k8s.io_v1_VerticalPodAutoscaler" plural="verticalpodautoscalers"
I1212 20:07:19.014860 1 config.go:209] "reloaded factory" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:19.015001 1 builder.go:208] "Updating store" GVR="autoscaling.k8s.io/v1, Resource=verticalpodautoscalers"
I1212 20:07:19.015052 1 discovery.go:274] "discovery finished, cache updated"
I1212 20:07:19.015088 1 metrics_handler.go:106] "Autosharding enabled with pod" pod="kube-prometheus/kube-state-metrics-0"
I1212 20:07:19.015118 1 metrics_handler.go:107] "Auto detecting sharding settings"
E1212 20:07:19.015489 1 discovery.go:253] "failed to run metrics handler" err="context canceled"
Impact:
The failure of the metrics handler prevents the collection and serving of new metrics. However, the service continues to serve the metrics collected before the failure occurred. This can lead to stale or outdated metrics being served, which may impact monitoring and alerting systems relying on kube-state-metrics.
Versions:
kube-state-metrics version: 2.12.0
Kubernetes version: v1.29.5
Cloud provider or hardware configuration: vCloud