Skip to content

[HUD] Show unstable-marked jobs on the reliability page #8419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
georgehong wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[HUD] Show unstable-marked jobs on the reliability page #8419

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
584732c
[HUD] Show unstable-marked jobs on the reliability page
georgehong Aug 3, 2026
e779b46
[HUD] Clarify Unstable jobs panel subtitle
georgehong Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions torchci/clickhouse_queries/master_commit_red_jobs/params.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,31 @@
"params": {
"startTime": "DateTime64(3)",
"stopTime": "DateTime64(3)",
"workflowNames": "Array(String)"
"workflowNames": "Array(String)",
"unstableOnly": "UInt8"
},
"defaults": {
"workflowNames": [],
"unstableOnly": 0
},
"tests": [
{
"startTime": "2025-03-18T21:09:47.987",
"stopTime": "2025-03-25T21:09:47.987",
"workflowNames": ["pull"]
"workflowNames": ["pull"],
"unstableOnly": 0
},
{
"startTime": "2025-03-18T21:09:47.987",
"stopTime": "2025-03-25T21:09:47.987",
"workflowNames": ["lint", "pull", "trunk"],
"unstableOnly": 0
},
{
"startTime": "2025-03-18T21:09:47.987",
"stopTime": "2025-03-25T21:09:47.987",
"workflowNames": ["lint", "pull", "trunk"]
"workflowNames": [],
"unstableOnly": 1
}
]
}
}
24 changes: 22 additions & 2 deletions torchci/clickhouse_queries/master_commit_red_jobs/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
-- This query is used to show failures on https://hud.pytorch.org/reliability/pytorch/pytorch
-- {unstableOnly}=0 backs the workflow-scoped panels; {unstableOnly}=1 backs the
-- "Unstable jobs" panel (see the WHERE branch below).
WITH all_jobs AS (
SELECT
p.head_commit. 'timestamp' AS time,
Expand Down Expand Up @@ -41,9 +43,27 @@ WITH all_jobs AS (
AND j.name != 'generate-test-matrix'
AND j.name NOT LIKE '%rerun_disabled_tests%'
AND j.name NOT LIKE '%filter%'
AND j.name NOT LIKE '%unstable%'
AND j.name LIKE '%/%'
AND has({workflowNames: Array(String) }, lower(j.workflow_name))
-- Normal mode ({unstableOnly}=0): exclude unstable jobs, keep the requested

@izaitsevfb izaitsevfb Aug 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test-infra/torchci/clickhouse_queries/master_commit_red_jobs/query.sql

Lines 10 to 19 in e779b46

CONCAT(
j.workflow_name,
' / ',
arrayElement(splitByString(' / ', j.name), 1),
' / ',
arrayElement(
splitByString(', ', arrayElement(splitByString(' / ', j.name), 2)),
1
)
) AS name,

test-infra/torchci/clickhouse_queries/master_commit_red_percent_groups/query.sql

Lines 26 to 28 in e779b46

AND j.name NOT LIKE '%unstable%'
AND j.name LIKE '%/%'
AND has({workflowNames: Array(String) }, lower(w.name))

a minor bug, please take a look

🟡 The newly shown rows link back to a chart that excludes them, so clicking one plots a different population or nothing. (ai-generated section)

The job label this query builds keeps only the text up to the first comma of a job's config list. The runtime unstable flag is appended after that point — in all 52 distinct marked job names seen on main over the last 14 days — so an unstable row's label is identical to the label the same job config produces when it is not marked.

That label is rendered as a link to ?jobName=<label>, which ticks the matching checkbox in the chart at the top of the page. The chart is fed by master_commit_red_percent_groups, which still drops every %unstable% job unconditionally. So for the three labels that currently exist both marked and unmarked — trunk / linux-jammy-rocm-py3.10-mi350 / test (default) and its (distributed) and (inductor) siblings — the link plots the stable series beside an unstable failure rate; for every other marked-only label there is no matching series at all. Those same three labels also render in this panel and in "Viable/strict blocking jobs" at the same time, computed over disjoint sets of runs, with nothing on the row itself to tell them apart.

There is no cheap fix here, which is why it is worth deciding rather than patching: the chart is shared by every panel, so widening its query would change what the other panels' rows select too, and suffixing the label would separate the two panels while stopping the link matching any checkbox. Selecting the unstable population explicitly, or dropping the link on this panel, are the two shapes that leave the rest of the page alone.

Reviewed by codex gpt-5.6-sol at xhigh effort, against e779b46.

-- workflows. Unstable mode ({unstableOnly}=1): the union of the "unstable"
-- workflow and jobs whose name carries the runtime unstable marker;
-- workflowNames is unused there. Both modes drop workflow_run-chained jobs
-- via the shared line below.
AND (
(
{unstableOnly: UInt8} = 0
AND j.name NOT LIKE '%unstable%'
AND has({workflowNames: Array(String) }, lower(j.workflow_name))
)
OR
(
{unstableOnly: UInt8} = 1
AND (
lower(j.workflow_name) = 'unstable'
OR j.name LIKE '%unstable%'
)
)
)
AND j.workflow_event != 'workflow_run' -- Filter out worflow_run-triggered jobs, which have nothing to do with the SHA
AND p.ref = 'refs/heads/main'
AND p.repository. 'owner'.'name' = 'pytorch'
Expand Down
13 changes: 9 additions & 4 deletions torchci/pages/reliability/[repoOwner]/[repoName]/[[...page]].tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -573,13 +573,18 @@ export default function Page() {
</Grid>

<Grid size={{ xs: 6 }} height={ROW_HEIGHT}>
{/*
Unstable jobs to watch: the union of unstable.yml and jobs whose name
carries the runtime unstable marker.
*/}
<GroupReliabilityPanel
title={"Unstable jobs"}
subtitle={
UNSTABLE_WORKFLOWS.map((w) => `${w}.yml`).join(", ") +
" + jobs tagged unstable at runtime"
}
queryName={queryName}
queryParams={{
workflowNames: UNSTABLE_WORKFLOWS,
...queryParams,
}}
queryParams={{ unstableOnly: 1, ...queryParams }}
metricName={metricName}
metricHeaderName={metricHeaderName}
filter={filter}
Expand Down
Loading