[HUD] Show unstable-marked jobs on the reliability page - #8419
[HUD] Show unstable-marked jobs on the reliability page#8419georgehong wants to merge 2 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The 'Unstable jobs' panel only matched the dedicated 'unstable' workflow (green scaffolding), so jobs marked unstable via 'UNSTABLE ...' issues -- which keep running on main with an ', unstable' config marker -- were never shown. Add an unstableOnly toggle to master_commit_red_jobs: 0 keeps the existing workflow-scoped behavior (verified byte-identical); 1 flips the filter to show the union of the dedicated 'unstable' workflow and any job marked unstable on main (workflow_run- chained jobs excluded, as in normal mode). Point the panel at it with unstableOnly=1. Reuses the shared query instead of a fork.
69180c9 to
584732c
Compare
|
Reliability page available at: https://torchci-git-reliability-unstable-jobs-panel-fbopensource.vercel.app/reliability |
Name the two groups the panel unions and say the marker is applied at runtime, so the subtitle doesn't read as the filed UNSTABLE issue list.
|
Note: there's a moderate drain effect where non-unstable history may show the same job being present in both the stable and unstable panel(s). I think it's useful to surface cordoned jobs so they may have recovery monitored in the span of a few days to week(s), but this may be subject to discussion to ensure there's no user confusion. The alternative may require some form of matching between unstable jobs, issue open/close timing history, and revising the job history, and we would need to update the existing panels. |
izaitsevfb
left a comment
There was a problem hiding this comment.
lgtm, but please consider fixing a bug below
| AND j.name NOT LIKE '%unstable%' | ||
| AND j.name LIKE '%/%' | ||
| AND has({workflowNames: Array(String) }, lower(j.workflow_name)) | ||
| -- Normal mode ({unstableOnly}=0): exclude unstable jobs, keep the requested |
There was a problem hiding this comment.
a minor bug, please take a look
🟡 The newly shown rows link back to a chart that excludes them, so clicking one plots a different population or nothing. (ai-generated section)
The job label this query builds keeps only the text up to the first comma of a job's config list. The runtime unstable flag is appended after that point — in all 52 distinct marked job names seen on main over the last 14 days — so an unstable row's label is identical to the label the same job config produces when it is not marked.
That label is rendered as a link to ?jobName=<label>, which ticks the matching checkbox in the chart at the top of the page. The chart is fed by master_commit_red_percent_groups, which still drops every %unstable% job unconditionally. So for the three labels that currently exist both marked and unmarked — trunk / linux-jammy-rocm-py3.10-mi350 / test (default) and its (distributed) and (inductor) siblings — the link plots the stable series beside an unstable failure rate; for every other marked-only label there is no matching series at all. Those same three labels also render in this panel and in "Viable/strict blocking jobs" at the same time, computed over disjoint sets of runs, with nothing on the row itself to tell them apart.
There is no cheap fix here, which is why it is worth deciding rather than patching: the chart is shared by every panel, so widening its query would change what the other panels' rows select too, and suffixing the label would separate the two panels while stopping the link matching any checkbox. Selecting the unstable population explicitly, or dropping the link on this panel, are the two shapes that leave the rest of the page alone.
Reviewed by codex gpt-5.6-sol at xhigh effort, against e779b46.
Previously, Unstable jobs panel feels a bit narrow since it was only including jobs part of the unstable workflow. Expand the query to also include jobs marked as unstable (e.g. with the Issue workflow). This provides a flow for users to monitor cordoned jobs and improvements in reliability.