feat: Add new hpa metrics to prevent prometheus timeseries duplication #2614

CountryTk · 2025-02-19T13:44:30Z

What this PR does / why we need it:

Added 4 new hpa metrics to prevent duplicated timeseries events like described in this issue: #2403

Added new metrics are:

kube_horizontalpodautoscaler_spec_target_container_metric
kube_horizontalpodautoscaler_spec_target_object_metric
kube_horizontalpodautoscaler_status_target_container_metric
kube_horizontalpodautoscaler_status_target_object_metric

How does this change affect the cardinality of KSM: (increases, decreases or does not change cardinality)
Cardinality is increased because of new metrics

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes # #2403

FYI: I've also tested this change in our prelive cluster and for us it fixed the issue

…MetricSource and add tests

linux-foundation-easycla · 2025-02-19T13:44:36Z

The committers listed above are authorized under a signed CLA.

✅ login: CountryTk / name: Tõnis Kõiv (e724198, 6b33b6f, 8269109, 5a2cf75, 1fe588b, 3584159, 2466943)

k8s-ci-robot · 2025-02-19T13:44:38Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: CountryTk
Once this PR has been reviewed and has the lgtm label, please assign rexagod for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-02-19T13:44:40Z

Welcome @CountryTk!

It looks like this is your first PR to kubernetes/kube-state-metrics 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kube-state-metrics has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

richabanker · 2025-02-20T17:52:03Z

/triage accepted
@CatherineF-dev @rexagod could you take a look here if possible, thanks!

CountryTk · 2025-02-25T13:13:26Z

Hey, could this please be reviewed @CatherineF-dev @rexagod

CatherineF-dev · 2025-03-04T12:48:18Z

Ok! In reviewing

CatherineF-dev · 2025-03-04T12:50:16Z

internal/store/horizontalpodautoscaler.go

-	targetMetricLabels = []string{"metric_name", "metric_target_type"}
+	targetMetricLabels    = []string{"metric_name", "metric_target_type"}
+	containerMetricLabels = []string{"metric_name", "metric_target_type", "container"}
+	objectMetricLabels    = []string{"metric_name", "metric_target_type", "full_target_name"}


nit: full_target_name -> target_name, container -> container_name

CatherineF-dev · 2025-03-04T12:55:23Z

internal/store/horizontalpodautoscaler.go

+	)
+}
+
+func createHPASpecTargetMetric() generator.FamilyGenerator {


nit: Unify createHPASpecTargetMetric and createHPASpecTargetObjectMetric. These two functions have some similar codes.

For example

createHPASpecTargetMetric(CollectContainerResourceMetricSourceType=false)

createHPASpecTargetMetric(CollectContainerResourceMetricSourceType=true)

CatherineF-dev · 2025-03-04T12:55:45Z

Overall LGTM. Two small comments.

…rgetObjectMetric

CountryTk · 2025-03-04T13:23:18Z

Overall LGTM. Two small comments.

Thanks for the review, I've implemented your suggestions @CatherineF-dev

CatherineF-dev · 2025-03-04T13:47:52Z

internal/store/horizontalpodautoscaler.go

+		createHPASpecTargetContainerMetric(),
+		createHPAStatusTargetContainerMetric(),
+		createHPAStatusTargetObjectMetric(),
+		createHPASpecTargetMetric(true),


Nit:
createHPASpecTargetObjectMetric()
createHPASpecTargetMetric()

func createHPASpecTargetObjectMetric() {
createHPASpecTarget([]{autoscaling.ObjectMetricSourceType})
}

func createHPASpecTargetMetric() {
createHPASpecTarget([]{autoscaling.PodsMetricSourceType, autoscaling.ResourceMetricSourceType, ...})
}

func createHPASpecTarget(autoscalingTypes) {
if m.Type not in autoscalingTypes {
return
}
}

Made some adjustments according to how I understood this comment, not 100% sure if I did it as you wish though.

Hey, @CatherineF-dev could you take a look if the changes requested are what you desired? Thank you

Overall LGTM. Left a small comment on some metrics are removed.

rexagod · 2025-03-16T20:12:28Z

Thank you for the patch.

I believe this identifies a resource-agnostic pitfall where we loop over certain nested fields without including a primary key in the overall generated metrics' label-sets.

I'll take a closer look tomorrow but so far this lgtm.

rexagod

Question, wouldn't the earlier kube_horizontalpodautoscaler_spec_target_metric still be there after this is in, and cause the error to still show up?

CatherineF-dev · 2025-03-18T00:35:49Z

Could you help enable allowed changes from maintainers?

I want to make a small change.

Or could you apply these small changes? 96d948a

CountryTk · 2025-03-18T07:56:47Z

Could you help enable allowed changes from maintainers?

I want to make a small change.

Or could you apply these small changes? 96d948a

For me it shows maintainer edit access is enabled.

Anyway, I've added your suggested changes in the latest commit.

Question, wouldn't the earlier kube_horizontalpodautoscaler_spec_target_metric still be there after this is in, and cause the error to still show up?

Nope because kube_horizontalpodautoscaler_spec_target_metric is only going to have PodsMetricSourceType, ResourceMetricSourceType and ExternalMetricSourceType now, which didn't cause those duplicated errors.

ContainerResourceMetricSourceType and ObjectMetricSourceType have separate functions with added labels to prevent duplicated data

CatherineF-dev · 2025-03-18T13:50:01Z

internal/store/horizontalpodautoscaler_test.go

 				kube_horizontalpodautoscaler_spec_target_metric{horizontalpodautoscaler="hpa1",metric_name="events",metric_target_type="average",namespace="ns1"} 30
-				kube_horizontalpodautoscaler_spec_target_metric{horizontalpodautoscaler="hpa1",metric_name="hits",metric_target_type="average",namespace="ns1"} 12


These metrics are deleted. Could you double check?

metric name "hits" and some others are now under their respective metric
hits is now under kube_horizontalpodautoscaler_spec_target_object_metric instead of kube_horizontalpodautoscaler_spec_target_metric

This seems a breaking change. Could we keep kube_horizontalpodautoscaler_spec_target_metric not changed?

Yes it is a breaking change but I was under the impression that it's fine since these metrics are experimental.
#2403 (comment)

I like the second option since the metric is still experimental so we can make changes to it. It will surely break some users but we didn't provide guarantees for this metrics. So as long as we make a smooth transition it should be fine.
The benefit of this approach is that the metrics will be specialized to the metric type they are targeting, so the labels will make sense for it and we won't run in scenarios where we have labels that are not used by some types.

kube_horizontalpodautoscaler_spec_target_object_metric{target_name="",horizontalpodautoscaler="hpa1",metric_name="hits",metric_target_type="average",namespace="ns1"} 12

@dgrisonnet I'm not sure if this is what you meant, can you PTAL?

feat: Create new metrics for ContainerResourceMetricSource and Object…

e724198

…MetricSource and add tests

k8s-ci-robot requested review from CatherineF-dev and logicalhan February 19, 2025 13:44

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 19, 2025

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 19, 2025

CountryTk added 2 commits February 20, 2025 11:25

Run gofmt -s

6b33b6f

Update documentation

3584159

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 20, 2025

Try fix documentation linter

8269109

CatherineF-dev reviewed Mar 4, 2025

View reviewed changes

Rename labels and unify createHPASpecTargetMetric and createHPASpecTa…

2466943

…rgetObjectMetric

CountryTk requested a review from CatherineF-dev March 4, 2025 13:23

CatherineF-dev reviewed Mar 4, 2025

View reviewed changes

Refactor functions further to create one more generic function

1fe588b

CatherineF-dev mentioned this pull request Mar 6, 2025

fix(horizontalpodautoscaler): add selector labels where applicable for hpa spec|status target metrics #2621

Closed

CountryTk requested a review from CatherineF-dev March 7, 2025 08:26

rexagod reviewed Mar 17, 2025

View reviewed changes

Apply suggested changes

5a2cf75

CatherineF-dev reviewed Mar 18, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add new hpa metrics to prevent prometheus timeseries duplication #2614

feat: Add new hpa metrics to prevent prometheus timeseries duplication #2614

CountryTk commented Feb 19, 2025 •

edited

Loading

linux-foundation-easycla bot commented Feb 19, 2025 •

edited

Loading

k8s-ci-robot commented Feb 19, 2025

k8s-ci-robot commented Feb 19, 2025

richabanker commented Feb 20, 2025

CountryTk commented Feb 25, 2025

CatherineF-dev commented Mar 4, 2025

CatherineF-dev Mar 4, 2025

CatherineF-dev Mar 4, 2025

CatherineF-dev commented Mar 4, 2025

CountryTk commented Mar 4, 2025 •

edited

Loading

CatherineF-dev Mar 4, 2025

CountryTk Mar 5, 2025

CountryTk Mar 12, 2025

CatherineF-dev Mar 18, 2025

rexagod commented Mar 16, 2025 •

edited

Loading

rexagod left a comment •

edited

Loading

CatherineF-dev commented Mar 18, 2025 •

edited

Loading

CountryTk commented Mar 18, 2025 •

edited

Loading

CatherineF-dev Mar 18, 2025

CountryTk Mar 18, 2025 •

edited

Loading

CatherineF-dev Mar 18, 2025

CountryTk Mar 18, 2025

rexagod Mar 22, 2025

feat: Add new hpa metrics to prevent prometheus timeseries duplication #2614

Are you sure you want to change the base?

feat: Add new hpa metrics to prevent prometheus timeseries duplication #2614

Conversation

CountryTk commented Feb 19, 2025 • edited Loading

linux-foundation-easycla bot commented Feb 19, 2025 • edited Loading

k8s-ci-robot commented Feb 19, 2025

k8s-ci-robot commented Feb 19, 2025

richabanker commented Feb 20, 2025

CountryTk commented Feb 25, 2025

CatherineF-dev commented Mar 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CatherineF-dev commented Mar 4, 2025

CountryTk commented Mar 4, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rexagod commented Mar 16, 2025 • edited Loading

rexagod left a comment • edited Loading

Choose a reason for hiding this comment

CatherineF-dev commented Mar 18, 2025 • edited Loading

CountryTk commented Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

CountryTk Mar 18, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CountryTk commented Feb 19, 2025 •

edited

Loading

linux-foundation-easycla bot commented Feb 19, 2025 •

edited

Loading

CountryTk commented Mar 4, 2025 •

edited

Loading

rexagod commented Mar 16, 2025 •

edited

Loading

rexagod left a comment •

edited

Loading

CatherineF-dev commented Mar 18, 2025 •

edited

Loading

CountryTk commented Mar 18, 2025 •

edited

Loading

CountryTk Mar 18, 2025 •

edited

Loading