Skip to content

Commit be2e76c

Browse files
committed
fix(metrics): ensure consistent label cardinality for eval CounterVec
The metrics code registers a CounterVec with label names from the first LMEvalJob it sees. If a subsequent job has a different set of modelArgs (e.g. one has base_url and another doesn't), the label count mismatches and Prometheus panics with "inconsistent label cardinality". Fix by always initializing both model_name and base_url labels to empty strings before populating from modelArgs, ensuring every job produces the same 6-label set. This is a pre-existing bug on main that was not surfaced because there was only one LMES test case. The new CA bundle tests in this PR create jobs with varying modelArgs, exposing the panic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED
1 parent 5c843c1 commit be2e76c

1 file changed

Lines changed: 4 additions & 8 deletions

File tree

controllers/lmes/lmevaljob_controller.go

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -480,20 +480,16 @@ func createJobCreationMetrics(log logr.Logger, job *lmesv1alpha1.LMEvalJob) {
480480
labels["model_type"] = job.Spec.Model
481481
labels["task"] = task
482482

483-
// grab model name
484-
hasUrl := false
485-
hasName := false
483+
// grab model name and base_url; always set both labels so the
484+
// CounterVec has a consistent label cardinality across jobs.
485+
labels["model_name"] = ""
486+
labels["base_url"] = ""
486487
for _, arg := range job.Spec.ModelArgs {
487488
if arg.Name == "model" {
488489
labels["model_name"] = arg.Value
489-
hasUrl = true
490490
}
491491
if arg.Name == "base_url" {
492492
labels["base_url"] = arg.Value
493-
hasName = true
494-
}
495-
if hasUrl && hasName {
496-
break
497493
}
498494
}
499495

0 commit comments

Comments
 (0)