Skip to content

[HUD] Show unstable-marked jobs on the reliability page - #8419

Open
georgehong wants to merge 2 commits into
mainfrom
reliability-unstable-jobs-panel
Open

[HUD] Show unstable-marked jobs on the reliability page#8419
georgehong wants to merge 2 commits into
mainfrom
reliability-unstable-jobs-panel

Conversation

@georgehong

@georgehong georgehong commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Previously, Unstable jobs panel feels a bit narrow since it was only including jobs part of the unstable workflow. Expand the query to also include jobs marked as unstable (e.g. with the Issue workflow). This provides a flow for users to monitor cordoned jobs and improvements in reliability.

@vercel

vercel Bot commented Aug 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
torchci Ready Ready Preview Aug 3, 2026 9:40pm

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 3, 2026
The 'Unstable jobs' panel only matched the dedicated 'unstable' workflow (green
scaffolding), so jobs marked unstable via 'UNSTABLE ...' issues -- which keep
running on main with an ', unstable' config marker -- were never shown. Add an
unstableOnly toggle to master_commit_red_jobs: 0 keeps the existing workflow-scoped
behavior (verified byte-identical); 1 flips the filter to show the union of the
dedicated 'unstable' workflow and any job marked unstable on main (workflow_run-
chained jobs excluded, as in normal mode). Point the panel at it with unstableOnly=1.
Reuses the shared query instead of a fork.
@georgehong

Copy link
Copy Markdown
Contributor Author

Name the two groups the panel unions and say the marker is applied at
runtime, so the subtitle doesn't read as the filed UNSTABLE issue list.
@georgehong

georgehong commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Note: there's a moderate drain effect where non-unstable history may show the same job being present in both the stable and unstable panel(s). I think it's useful to surface cordoned jobs so they may have recovery monitored in the span of a few days to week(s), but this may be subject to discussion to ensure there's no user confusion.

The alternative may require some form of matching between unstable jobs, issue open/close timing history, and revising the job history, and we would need to update the existing panels.

@georgehong
georgehong marked this pull request as ready for review August 3, 2026 23:18

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@izaitsevfb izaitsevfb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, but please consider fixing a bug below

AND j.name NOT LIKE '%unstable%'
AND j.name LIKE '%/%'
AND has({workflowNames: Array(String) }, lower(j.workflow_name))
-- Normal mode ({unstableOnly}=0): exclude unstable jobs, keep the requested

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CONCAT(
j.workflow_name,
' / ',
arrayElement(splitByString(' / ', j.name), 1),
' / ',
arrayElement(
splitByString(', ', arrayElement(splitByString(' / ', j.name), 2)),
1
)
) AS name,

AND j.name NOT LIKE '%unstable%'
AND j.name LIKE '%/%'
AND has({workflowNames: Array(String) }, lower(w.name))

a minor bug, please take a look


🟡 The newly shown rows link back to a chart that excludes them, so clicking one plots a different population or nothing. (ai-generated section)

The job label this query builds keeps only the text up to the first comma of a job's config list. The runtime unstable flag is appended after that point — in all 52 distinct marked job names seen on main over the last 14 days — so an unstable row's label is identical to the label the same job config produces when it is not marked.

That label is rendered as a link to ?jobName=<label>, which ticks the matching checkbox in the chart at the top of the page. The chart is fed by master_commit_red_percent_groups, which still drops every %unstable% job unconditionally. So for the three labels that currently exist both marked and unmarked — trunk / linux-jammy-rocm-py3.10-mi350 / test (default) and its (distributed) and (inductor) siblings — the link plots the stable series beside an unstable failure rate; for every other marked-only label there is no matching series at all. Those same three labels also render in this panel and in "Viable/strict blocking jobs" at the same time, computed over disjoint sets of runs, with nothing on the row itself to tell them apart.

There is no cheap fix here, which is why it is worth deciding rather than patching: the chart is shared by every panel, so widening its query would change what the other panels' rows select too, and suffixing the label would separate the two panels while stopping the link matching any checkbox. Selecting the unstable population explicitly, or dropping the link on this panel, are the two shapes that leave the rest of the page alone.

Reviewed by codex gpt-5.6-sol at xhigh effort, against e779b46.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants