WIP: add metrics for workloads count

[kannon92]

What type of PR is this?

/kind feature

What this PR does / why we need it:

We will provide an operator to manage the configuration of Kueue and we expect to have various customers with varrying frameworks supported for Kueue.

Over time we know we would like to see what kind of Kinds are managed by Kueue and how many.

We want to propose a counter per in-tree framework that counts the number of workloads kueue managed and whether or not the workload is actually enabled.

Which issue(s) this PR fixes:

Fixes: #4336

Special notes for your reviewer:

This is open for comments.

Does this PR introduce a user-facing change?

Add metrics for workloads to display if workloads are supported in Kueue and if so how many workloads kueue is managing.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kannon92
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	4639bf5
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/67b8f85b3a33fd0008e31768

[mimowo]

Please don't multiply the metrics by the framework name. Framework name should be a label in the metrics.

[tenzen-y]

Basically, increasing kind of metrics affect scheduling performance since the metrics counter need to take a lock every time.

So, I would recommend adding the kind of workload type like batch/v1 Job to the existing Workload counting metrics as @mimowo mentioned.

[kannon92]

Do you all think I should have two metrics.

one for supported frameworks and the other for external?

I wasn’t sure if exposing the GVK for external frameworks would be valid.

[tenzen-y]

I imagined adding framework-type label to the existing Worklaod-related metric like

kueue/pkg/metrics/metrics.go

Line 124 in 4639bf5

QuotaReservedWorkloadsTotal = prometheus.NewCounterVec(

@mimowo If you have any other view, let us know.

[mimowo]

I like the idea of extending the existing workload metrics more than entirely new ones. If this is risky we may use a feature gate for a release or two

[mimowo]

you can aggregate the metrics at the level of grafana, by summing over the CQ names. Then you will display results by frameworkName

[kannon92]

Are you okay with including external frameworks in that same label?

[mimowo]

I'm ok with that, every workload needs to belong to some framework, either built-in or external, so otherwise we would need to say "". I guess user needs to have some other way (if she / he wants) to determine which are external and which built-in.

[KPostOffice]

Basically, increasing kind of metrics affect scheduling performance since the metrics counter need to take a lock every time

Prometheus metric counter is thread safe no?

[tenzen-y]

Prometheus metric counter is thread safe no?

Yes, that is thread safe. I meant metrics counter take a lock internally. We do not need to take a lock in our side.

[mimowo]

/hold
To avoid merging accidentally since the PR seems still far from being ready. It is ok to prototype, but for merging we need first agreement on the metrics and their labels,

/retitle WIP: add metrics for workloads count

[mimowo]

Leaving the first proper pass to @gabesaba, just raising #4350 (comment) as a top-level remark.

[k8s-ci-robot]

@kannon92: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kueue-test-unit-main	4639bf5	link	true	/test pull-kueue-test-unit-main

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[kannon92]

I'm trying to find the place to put the count of workloads. Anyone have an idea?

[gabesaba]

/assign