Skip to content

Commit 4baaadd

Browse files
subinz1subinz1
authored
[CRCR] Treat expected non-success outcomes (xfail/xcancel/xtimeout) as successes in pass rate (#8376)
## Summary Fixes the CRCR pass rate calculation to correctly handle **expected non-success outcomes** in health-probe repos like `pytorch/crcr-test`. The `crcr-test` repo uses jobs with `x`-prefixed names (`xfail`, `xcancel`, `xtimeout`) that intentionally end in failure, cancellation, or timeout to test the relay's handling of these outcomes (see [pytorch/crcr-test#19](pytorch/crcr-test#19)). These expected results were being counted as failures in the ClickHouse pass rate queries, causing `pytorch/crcr-test` to show a degraded health status even though the relay is functioning correctly. ### Changes Three ClickHouse queries updated with the same logic: 1. **`crcr_summary`** — CRCR summary page (`/crcr`) 2. **`crcr_success_rate`** — CRCR metrics page (`/crcr/metrics`) 3. **`crcr_backend_summary`** — per-repo dashboard (`/crcr/[org]/[repo]`) For each query: - **Successes**: now includes jobs where `job_name LIKE '%xfail%'`, `'%xcancel%'`, or `'%xtimeout%'` - **Failures**: excludes `xfail` jobs - **Timed out**: excludes `xtimeout` jobs This uses the `job_name` convention established in [pytorch/crcr-test#19](pytorch/crcr-test#19) — jobs prefixed with `x` are intentional/expected outcomes, not bugs. Fixes #8306 ## Test plan - [ ] Verify CRCR summary page shows improved pass rate for `pytorch/crcr-test` - [ ] Verify metrics page charts reflect corrected rates - [ ] Verify per-repo dashboard for `pytorch/crcr-test` shows accurate stats
1 parent 30f7d96 commit 4baaadd

3 files changed

Lines changed: 81 additions & 9 deletions

File tree

‎torchci/clickhouse_queries/crcr_backend_summary/query.sql‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,31 @@
11
SELECT
2-
countIf(conclusion = 'success') AS successes,
3-
countIf(conclusion = 'failure') AS failures,
4-
countIf(conclusion = 'timed_out') AS timed_out,
2+
countIf(
3+
conclusion = 'success'
4+
OR (
5+
{repo: String} = 'pytorch/crcr-test'
6+
AND (
7+
(job_name LIKE '%xfail%' AND conclusion = 'failure')
8+
OR (job_name LIKE '%xcancel%' AND conclusion = 'cancelled')
9+
OR (job_name LIKE '%xtimeout%' AND conclusion = 'timed_out')
10+
)
11+
)
12+
) AS successes,
13+
countIf(
14+
conclusion = 'failure'
15+
AND NOT (
16+
{repo: String} = 'pytorch/crcr-test'
17+
AND job_name LIKE '%xfail%'
18+
AND conclusion = 'failure'
19+
)
20+
) AS failures,
21+
countIf(
22+
conclusion = 'timed_out'
23+
AND NOT (
24+
{repo: String} = 'pytorch/crcr-test'
25+
AND job_name LIKE '%xtimeout%'
26+
AND conclusion = 'timed_out'
27+
)
28+
) AS timed_out,
529
count() AS total_jobs,
630
if(total_jobs > 0, successes / total_jobs, 0) AS pass_rate,
731
uniqExact(pr_number) AS total_prs,

‎torchci/clickhouse_queries/crcr_success_rate/query.sql‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,33 @@
11
SELECT
22
toDate(started_at) AS day,
33
downstream_repo AS repo,
4-
countIf(conclusion = 'success') AS successes,
5-
countIf(conclusion = 'failure') AS failures,
6-
countIf(conclusion = 'timed_out') AS timed_out,
4+
countIf(
5+
conclusion = 'success'
6+
OR (
7+
downstream_repo = 'pytorch/crcr-test'
8+
AND (
9+
(job_name LIKE '%xfail%' AND conclusion = 'failure')
10+
OR (job_name LIKE '%xcancel%' AND conclusion = 'cancelled')
11+
OR (job_name LIKE '%xtimeout%' AND conclusion = 'timed_out')
12+
)
13+
)
14+
) AS successes,
15+
countIf(
16+
conclusion = 'failure'
17+
AND NOT (
18+
downstream_repo = 'pytorch/crcr-test'
19+
AND job_name LIKE '%xfail%'
20+
AND conclusion = 'failure'
21+
)
22+
) AS failures,
23+
countIf(
24+
conclusion = 'timed_out'
25+
AND NOT (
26+
downstream_repo = 'pytorch/crcr-test'
27+
AND job_name LIKE '%xtimeout%'
28+
AND conclusion = 'timed_out'
29+
)
30+
) AS timed_out,
731
count() AS total,
832
if(total > 0, successes / total, 0) AS pass_rate
933
FROM

‎torchci/clickhouse_queries/crcr_summary/query.sql‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,33 @@
11
SELECT
22
downstream_repo AS repo,
33
anyLast(downstream_repo_level) AS downstream_repo_level,
4-
countIf(conclusion = 'success') AS successes,
5-
countIf(conclusion = 'failure') AS failures,
6-
countIf(conclusion = 'timed_out') AS timed_out,
4+
countIf(
5+
conclusion = 'success'
6+
OR (
7+
downstream_repo = 'pytorch/crcr-test'
8+
AND (
9+
(job_name LIKE '%xfail%' AND conclusion = 'failure')
10+
OR (job_name LIKE '%xcancel%' AND conclusion = 'cancelled')
11+
OR (job_name LIKE '%xtimeout%' AND conclusion = 'timed_out')
12+
)
13+
)
14+
) AS successes,
15+
countIf(
16+
conclusion = 'failure'
17+
AND NOT (
18+
downstream_repo = 'pytorch/crcr-test'
19+
AND job_name LIKE '%xfail%'
20+
AND conclusion = 'failure'
21+
)
22+
) AS failures,
23+
countIf(
24+
conclusion = 'timed_out'
25+
AND NOT (
26+
downstream_repo = 'pytorch/crcr-test'
27+
AND job_name LIKE '%xtimeout%'
28+
AND conclusion = 'timed_out'
29+
)
30+
) AS timed_out,
731
count() AS total,
832
if(total > 0, successes / total, 0) AS pass_rate,
933
avg(duration_seconds) AS avg_duration_s,

0 commit comments

Comments
 (0)