Skip to content

Commit f75bb0d

Browse files
authored
Fix LF rollover metric to count both lf. and lf- runner labels (#8250)
**Impact:** HUD `/metrics` "Percentage of jobs rolled over to Linux Foundation" panel only **Risk:** low ## What Updates the `lf_rollover_percentage` ClickHouse query to classify Linux Foundation jobs by both the `lf.` (dot) and `lf-` (dash) label prefixes, excludes the new `c-` dash canary form, and fixes the `Linux Fundation` → `Linux Foundation` typo on the series label. ## Why LF runners started migrating from `lf.` to `lf-` labels around June 7, 2026. The classifier only matched the dot prefix, so roughly 79% of LF jobs (the `lf-…` ones) were silently counted as Meta. From mid-June the panel showed LF rollout collapsing to ~1–2% when it had actually grown — it was just renamed. With the fix the metric reads ~18–29% in the dash era. # Notes - The output contract (`bucket`, `fleet`, `percentage` + the three params) is unchanged, so the panel, query loader, and CI are unaffected. - The LF-vs-Meta `FULL OUTER JOIN` was removed and replaced with a direct per-bucket conditional ratio. Verified to produce numerically identical output (diff = 0 across every bucket) while dropping a silent dependency on the server's `join_use_nulls` setting; the LF-prefix rule is now consolidated into a single `is_lf` column. - Kept single-series (LF% only) intentionally — Meta% = 100 − LF%, so one line encodes both. This matches the earlier deliberate removal of the two-series variant in commit e2efc38. - Known limitation (not changed here): the "Meta" side of the denominator is mostly un-prefixed EC2, not strictly `mt-`, so the metric is "LF share of all comparable runs" rather than a strict `lf-` vs `mt-` head-to-head. A true fleet-vs-fleet comparison would be a separate product change. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
1 parent 8062973 commit f75bb0d

1 file changed

Lines changed: 9 additions & 45 deletions

File tree

  • torchci/clickhouse_queries/lf_rollover_percentage

torchci/clickhouse_queries/lf_rollover_percentage/query.sql

Lines changed: 9 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
WITH
22
normalized_jobs AS (
33
SELECT
4-
l AS label,
54
extract(j.name, '[^,]*') AS job_name, -- Remove shard number and label from job names
6-
j.workflow_name,
5+
match(l, '^lf[.-]') AS is_lf,
76
DATE_TRUNC({granularity: String}, j.created_at) AS bucket
87
FROM
98
-- Deliberatly not adding FINAL to this workflow_job.
@@ -26,67 +25,32 @@ WITH
2625
AND l NOT LIKE 'lf.c.%'
2726
AND l NOT LIKE '%.canary'
2827
AND l NOT LIKE 'c.%'
28+
AND l NOT LIKE 'c-%'
2929
),
3030
lf_jobs AS (
3131
SELECT
32-
DISTINCT j.job_name
32+
DISTINCT job_name
3333
FROM
34-
normalized_jobs AS j
34+
normalized_jobs
3535
WHERE
36-
j.label LIKE 'lf.%'
36+
is_lf
3737
),
3838
comparable_jobs AS (
3939
SELECT
4040
j.bucket,
41-
j.label,
42-
j.job_name,
43-
j.workflow_name
41+
j.is_lf
4442
FROM
4543
normalized_jobs AS j
4644
INNER JOIN
4745
lf_jobs AS lfj ON j.job_name = lfj.job_name
4846
),
49-
success_stats AS (
47+
comparison_stats AS (
5048
SELECT
51-
count(*) AS group_size,
5249
bucket,
53-
replaceOne(label, 'lf.', '') AS label_ref,
54-
if(substring(label, 1, 3) = 'lf.', True, False) AS lf_fleet
50+
CAST(countIf(is_lf) AS Float32) / count(*) * 100 AS percentage,
51+
'Linux Foundation' AS fleet
5552
FROM
5653
comparable_jobs
57-
GROUP BY
58-
bucket, label_ref, lf_fleet
59-
),
60-
lf_success_stats AS (
61-
SELECT
62-
*
63-
FROM
64-
success_stats
65-
WHERE
66-
lf_fleet = True
67-
),
68-
meta_success_stats AS (
69-
SELECT
70-
*
71-
FROM
72-
success_stats
73-
WHERE
74-
lf_fleet = False
75-
),
76-
comparison_stats AS (
77-
SELECT
78-
-- *
79-
greatest(lf.bucket, m.bucket) AS bucket,
80-
CAST(SUM(lf.group_size) AS Float32) / SUM(lf.group_size + m.group_size) * 100 AS percentage,
81-
-- IF(lf.lf_fleet, 'Linux Foundation', 'Meta') AS fleet
82-
'Linux Fundation' AS fleet
83-
FROM
84-
lf_success_stats AS lf
85-
FULL OUTER JOIN
86-
meta_success_stats AS m
87-
ON
88-
lf.label_ref = m.label_ref
89-
AND lf.bucket = m.bucket
9054
GROUP BY
9155
bucket
9256
)

0 commit comments

Comments
 (0)