Skip to content

Commit e022736

Browse files
authored
[CRCR] Replace Flaky Jobs card with Timeout Rate and align Pass Rate colors with L3 criteria (#8421)
## Summary Two changes to the CRCR per-repo dashboard page (`/crcr/{org}/{repo}`), aligning stat card thresholds with the L3 promotion criteria proposed in [pytorch/rfcs#102](pytorch/rfcs#102). ### 1. Replace Flaky Jobs card → Timeout Rate **ClickHouse query** (`crcr_backend_summary/query.sql`): - Removed the complex flaky-job subquery (correlated subquery scanning the table twice) - Added `timeout_rate = timed_out / total_jobs` **Frontend card**: - Shows percentage (e.g., `0.0%`, `1.2%`) - Sub-text: `X timed out / Y jobs` - Color: green (0%), orange (>0% but <10%), red (≥10% — exceeds L3 threshold) ### 2. Align Pass Rate color thresholds Updated to match L3 criteria (`job pass rate > 90%`): - **Green**: 100% - **Orange**: 90–99.9% (meets L3) - **Red**: < 90% (below L3) Previously: ≥95% green, ≥80% orange, <80% red. ## Context The L3 criteria in [pytorch/rfcs#102](pytorch/rfcs#102) defines: - **Timeout rate < 10%** as an infrastructure reliability signal - **Job pass rate > 90%** as a test quality signal These cards make L3 readiness directly visible on the per-repo dashboard. ## Test plan - [ ] Verify `crcr_backend_summary` query returns `timeout_rate` correctly - [ ] Verify Timeout Rate card renders with proper color coding - [ ] Verify Pass Rate card shows red when < 90%, orange when 90–99.9%, green at 100%
1 parent 51bd043 commit e022736

2 files changed

Lines changed: 18 additions & 25 deletions

File tree

torchci/clickhouse_queries/crcr_backend_summary/query.sql

Lines changed: 5 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,24 +7,11 @@ SELECT
77
uniqExact(pr_number) AS total_prs,
88
avg(queue_time) AS avg_queue_time_s,
99
avg(execution_time) AS avg_exec_time_s,
10-
-- Flaky: jobs where the same job_name has both success and failure
11-
-- across different run_attempts for the same PR
12-
uniqExactIf(
13-
job_name,
14-
job_name IN (
15-
SELECT job_name
16-
FROM default.crcr_workflow_job FINAL
17-
WHERE
18-
downstream_repo = {repo: String}
19-
AND started_at > now() - INTERVAL {days: UInt64} DAY
20-
AND status = 'completed'
21-
AND pr_number > 0
22-
GROUP BY pr_number, job_name
23-
HAVING
24-
countIf(conclusion = 'success') > 0
25-
AND countIf(conclusion = 'failure') > 0
26-
)
27-
) AS flaky_jobs
10+
if(
11+
total_jobs > 0,
12+
timed_out / total_jobs,
13+
0
14+
) AS timeout_rate
2815
FROM
2916
default.crcr_workflow_job FINAL
3017
WHERE

torchci/pages/crcr/[org]/[repo].tsx

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ interface SummaryStats {
5858
total_prs: number;
5959
avg_queue_time_s: number | null;
6060
avg_exec_time_s: number | null;
61-
flaky_jobs: number;
61+
timeout_rate: number;
6262
}
6363

6464
// ---- Summary Stat Cards ----
@@ -102,9 +102,9 @@ function StatCard({
102102

103103
function SummaryCards({ stats }: { stats: SummaryStats }) {
104104
const passColor =
105-
stats.pass_rate >= 0.95
105+
stats.pass_rate >= 1.0
106106
? "#2e7d32"
107-
: stats.pass_rate >= 0.8
107+
: stats.pass_rate >= 0.9
108108
? "#ed6c02"
109109
: "#d32f2f";
110110

@@ -149,10 +149,16 @@ function SummaryCards({ stats }: { stats: SummaryStats }) {
149149
sub="start to completion"
150150
/>
151151
<StatCard
152-
label="Flaky Jobs"
153-
value={stats.flaky_jobs}
154-
sub="same job: pass + fail across attempts"
155-
color={stats.flaky_jobs > 0 ? "#ed6c02" : undefined}
152+
label="Timeout Rate"
153+
value={`${(stats.timeout_rate * 100).toFixed(1)}%`}
154+
sub={`${stats.timed_out} timed out / ${stats.total_jobs} jobs`}
155+
color={
156+
stats.timeout_rate >= 0.1
157+
? "#d32f2f"
158+
: stats.timeout_rate > 0
159+
? "#ed6c02"
160+
: undefined
161+
}
156162
/>
157163
</Box>
158164
</Stack>

0 commit comments

Comments
 (0)