-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Description
Description
Metrics exposed by karpenter has huge cardinality.
Main contributor when karpenter runs in AWS EKS cluster are metrics related to AWS instance pricing
karpenter_cloudprovider_instance_type_offering_available
karpenter_cloudprovider_instance_type_offering_price_estimate
cardinality of labels here is:
instance_type 891
capacity_type 2
This together makes 1782
When karpenter is installed with ServiceMonitor enabled on default kube-prometheus stack it adds more labels like: node, zone and then cardinality explodes. In out case we end up with ~100K for a single metric.
While I understand that this data is needed for karpenter to make decision about which instance to use, I want to propose rethinking if exposing these metrics by default is needed.
In majority of EKS clusters I've seen karpenter is number 1 contributor to "observability costs" (metrics cardinality is often primary cost driver in pricing models of observability vendors)
How important is this feature to you?
We can override this behavour via extra configuration to ServiceMonitor via relabeling) already, but
The request is to either add such configuration as default or (better) do not expose these metrics by default and only if requested (via new flag like -expose-cloud-pricing-metrics which would be not set in default helm chart values)
This would help reducing observability costs for anyone using that helm chart unless they know what they are doing.