[WIP] Fix metrics test: nvidia_dra_requests_total may not exist before first use#1028
Closed
dims wants to merge 1 commit intokubernetes-sigs:mainfrom
Closed
[WIP] Fix metrics test: nvidia_dra_requests_total may not exist before first use#1028dims wants to merge 1 commit intokubernetes-sigs:mainfrom
dims wants to merge 1 commit intokubernetes-sigs:mainfrom
Conversation
…t use The nvidia_dra_requests_total counter is only registered after the first DRA allocation request. When the metrics test runs right after chart install (before any GPU pods), the counter does not appear in the metrics output. Make the assertion conditional so it does not fail when the counter has not been observed yet. The nvidia_dra_prepared_devices gauge is always present and remains the primary assertion.
github-project-automation
bot
moved this to Backlog
in Planning Board: k8s-dra-driver-gpu
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dims The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
approved
Member
Author
|
/assign @shivamerla @shivamerla should we fix the root cause? this is masking a symptom |
shivamerla
reviewed
Apr 12, 2026
| run kubectl exec -n dra-driver-nvidia-gpu "${plugin_pod}" -c gpus -- \ | ||
| sh -c 'curl -sf http://localhost:8080/metrics 2>/dev/null || wget -qO- http://localhost:8080/metrics' | ||
| assert_output --partial "nvidia_dra_prepared_devices" | ||
| assert_output --partial "nvidia_dra_requests_total" |
Contributor
Member
Author
There was a problem hiding this comment.
@shivamerla i'd rather land this one which is complete fix #1029
rather than the mitigation in this PR.
Member
Author
|
/hold going to show a better way hopefully! |
k8s-ci-robot
added
the
do-not-merge/hold
dims
changed the title
k8s-ci-robot
added
the
do-not-merge/work-in-progress
github-project-automation
bot
moved this from Backlog
to Closed
in Planning Board: k8s-dra-driver-gpu
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
nvidia_dra_requests_totalPrometheus counter is only registered after the first DRA allocation request. When the metrics smoke test runs immediately after chart install — before any GPU pod has been scheduled — the counter does not appear in the/metricsoutput. This causes theassert_output --partial "nvidia_dra_requests_total"assertion to fail.Make the assertion conditional: still validate the counter if it's present, but don't fail when it hasn't been observed yet. The
nvidia_dra_prepared_devicesgauge is always present and remains the primary assertion.This is the root cause of the Lambda CI failure in #1025:
Prow log