add labels to prowjobs to make better selections

[dhiller]

Is your feature request related to a problem? Please describe:
As a Grafana admin I'd like to be able to select specific jobs by k8s version, SIG etc. to improve drilling-down the data that is present, i.e. currently I have to select specific jobs per category by using regular expressions. See the e2e jobs overview panel for an example

Describe the solution you'd like:
We could add labels to the jobs, so that they support selecting categories by those labels.

I've asked Claude on that matter, response was this:

---

How Prow job labels become Prometheus labels

The prow-exporter (running in your cluster) automatically exposes all
Kubernetes labels on ProwJob CRD objects as the prow_job_labels Prometheus
metric. No changes to config.yaml are needed.

Mechanism

1. Add labels to the job spec — the labels: section in the presubmit YAML
   attaches labels to the ProwJob CRD when a job is triggered:

  labels:
    my-custom-label: "value"
    ci.kubevirt.io/prometheus: ""

2. Prow-exporter converts them automatically — the exporter's Collect() in
   cmd/exporter/collector.go lists all ProwJobs and emits a prow_job_labels gauge
   metric with every label (except denylisted ones) as a Prometheus label
   dimension, prefixed with label_:

  prow_job_labels{job_name="...", label_my_custom_label="value",
  label_ci_kubevirt_io_prometheus="", ...} 1

3. Label key sanitization — any character invalid in Prometheus label names
   (/, ., -, etc.) is replaced with _. So ci.kubevirt.io/prometheus becomes
   label_ci_kubevirt_io_prometheus.
4. Annotations are exposed similarly as prow_job_annotations with annotation_
   prefix.

Denylisted labels (excluded from metrics)

These internal Prow labels are filtered out and will not appear in
prow_job_labels:

• prow.k8s.io/type
• prow.k8s.io/id
• prow.k8s.io/build-id
• prow.k8s.io/job
• prow.k8s.io/refs.org
• prow.k8s.io/refs.repo
• prow.k8s.io/refs.pull
• created-by-prow

Summary

Just add whatever labels you want to the labels: block in the job spec —
they'll automatically appear in the prow_job_labels metric. The pattern
ci.kubevirt.io/prometheus: "" already used on the kwok performance jobs (line
1565 in kubevirt-presubmits.yaml) is an example of this in practice.

How to select jobs by using those labels

The prow_job_labels metric always has value 1 and acts as a label inventory.
To filter other Prow metrics by job label, you join using job_name.

Basic label selection

Select all job instances with a specific label:

  prow_job_labels{label_ci_kubevirt_io_prometheus=""}

For a label like my-custom-label: "value", the dashes become underscores:

  prow_job_labels{label_my_custom_label="value"}

Filtering other metrics by job label

To filter runtime/state metrics, join prow_job_labels with the target metric
on job_name:

  prow_job_runtime_seconds_sum
    * on(job_name) group_left()
    prow_job_labels{label_ci_kubevirt_io_prometheus=""}

The group_left() carries over labels from prow_job_runtime_seconds_sum (the
"many" side). You can also pull additional label values from prow_job_labels
into the result:

  prow_job_runtime_seconds_sum
    * on(job_name) group_left(label_ci_kubevirt_io_prometheus)
    prow_job_labels{label_ci_kubevirt_io_prometheus=""}

Important caveat: staleness

The exporter only keeps the latest run of each named job (see getLatest() in
collector.go). So prow_job_labels reflects the labels of whichever run of that
job name was most recent. If you change a job's labels between runs, only the
new labels will show.

Checking what labels are available

To see all label dimensions exposed for a given job:

  prow_job_labels{job_name="pull-kubevirt-e2e-k8s-1.34-sig-performance-kwok"}

---

Describe alternatives you've considered:
A clear and concise description of any alternative solutions or features you've considered.

Additional context:
Add any other context or screenshots about the feature request here.

/area monitoring

[dhiller]

/triage accepted
/priority backlog