Skip to content

Do not expose AWS pricing metrics by default #8798

@pjastrzabek

Description

@pjastrzabek

Description

Metrics exposed by karpenter has huge cardinality.
Main contributor when karpenter runs in AWS EKS cluster are metrics related to AWS instance pricing

karpenter_cloudprovider_instance_type_offering_available 
karpenter_cloudprovider_instance_type_offering_price_estimate

cardinality of labels here is:
instance_type 891
capacity_type 2

This together makes 1782

When karpenter is installed with ServiceMonitor enabled on default kube-prometheus stack it adds more labels like: node, zone and then cardinality explodes. In out case we end up with ~100K for a single metric.

While I understand that this data is needed for karpenter to make decision about which instance to use, I want to propose rethinking if exposing these metrics by default is needed.

In majority of EKS clusters I've seen karpenter is number 1 contributor to "observability costs" (metrics cardinality is often primary cost driver in pricing models of observability vendors)

How important is this feature to you?
We can override this behavour via extra configuration to ServiceMonitor via relabeling) already, but

The request is to either add such configuration as default or (better) do not expose these metrics by default and only if requested (via new flag like -expose-cloud-pricing-metrics which would be not set in default helm chart values)

This would help reducing observability costs for anyone using that helm chart unless they know what they are doing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureNew feature or requestneeds-triageIssues that need to be triaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions