Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add karpenter_pods_drained_total metric to track pod draining by reason #2044

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

omerap12
Copy link
Member

@omerap12 omerap12 commented Mar 2, 2025

Fixes #2021

Description
Implements a new Prometheus metric to count pods drained during node termination, labeled by the reason for draining. This provides visibility into the number of pods affected by different termination scenarios.

How was this change tested?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@k8s-ci-robot k8s-ci-robot requested a review from jackfrancis March 2, 2025 17:54
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 2, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: omerap12
Once this PR has been reviewed and has the lgtm label, please assign ellistarn for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested a review from tallaxes March 2, 2025 17:54
@k8s-ci-robot k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Mar 2, 2025
@omerap12 omerap12 force-pushed the karpenter_pods_drained_total branch from c5cfc20 to e1588e7 Compare March 2, 2025 19:44
@omerap12 omerap12 changed the title Add karpenter_pods_drained_total metric to track pod draining by reason feat: add karpenter_pods_drained_total metric to track pod draining by reason Mar 2, 2025
@coveralls
Copy link

Pull Request Test Coverage Report for Build 13618713849

Details

  • 1 of 1 (100.0%) changed or added relevant line in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.02%) to 81.679%

Totals Coverage Status
Change from base Build 13595577059: 0.02%
Covered Lines: 9478
Relevant Lines: 11604

💛 - Coveralls

@engedaam
Copy link
Contributor

engedaam commented Mar 3, 2025

/assign @engedaam

crmetrics.Registry,
prometheus.CounterOpts{
Namespace: metrics.Namespace,
Subsystem: metrics.NodeSubsystem,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs to be in the metrics.PodsSubsystem -- as it stands right now this is called karpenter_nodes_pods_drained_total which doesn't sound right

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

@jonathan-innis
Copy link
Member

I also think we need a rebase with one of the latest changes that was made to this section of code

@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 4, 2025
@jonathan-innis
Copy link
Member

It might also be nice to add a test to make sure that we are properly emitting the metric -- I think you should be able to find similar metric tests for node eviction requests total

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add karpenter_pods_drained_total during Node drain
5 participants