[CI] Parity auto-trigger: collect scheduled architectures including Preview - #3555
Open
ethanwee1 wants to merge 25 commits into
Open
[CI] Parity auto-trigger: collect scheduled architectures including Preview#3555ethanwee1 wants to merge 25 commits into
ethanwee1 wants to merge 25 commits into
Conversation
The "Job ID" links in the parity summary were built by pairing every shard
dir's "_<job_id>" with a single "_wf_run_id" file. But the default,
distributed and inductor configs resolve to DIFFERENT upstream runs (e.g.
mi350 default+inductor come from a trunk push while distributed comes from
periodic / a trunk fallback), and they all unzip into the same rocm_xml
folder. "_wf_run_id" is overwritten per config (last write wins), so shards
from the other runs got a URL like runs/<trunk_run>/job/<distributed_job_id>
- a run the job never belonged to, which 404s ("busted" in the summary).
Concrete case: run 30444136839 (mi350) linked
runs/30416143612/job/90430654935, but job 90430654935 (mi350 distributed
shard 1) actually lives in run 30404499951.
download_testlogs now records a job_id -> run_id map in "_wf_run_ids.json"
next to the shard dirs (accumulated across configs), and
summarize_xml_testreports.py looks each shard's job id up in that map,
falling back to the legacy single "_wf_run_id" only when absent. Job URLs
are now built with the run that actually produced each shard.
Read CUDA and ROCm shard totals from workflow YAML at the tested commit, self-heal renamed job prefixes, and follow reusable-workflow artifacts to their hosting run.
Keep topology discovery aligned with the scheduled rocm-preview lane and remove the retired nightly key.
Keep job-link generation aligned with the scheduled rocm-preview lane and remove the retired nightly key.
Prevent same-SHA jobs from another ROCm workflow from self-healing to the wrong prefix by traversing nested reusable runs and filtering candidates by architecture.
Use the live rocm-preview mi350 topology and remove the retired nightly CLI/config key on the current develop history.
Auto-parity was trunk-scoped (mi350 only). mi300/mi200/navi31 run in their own scheduled upstream workflows on their own SHAs at a different cadence, so the trunk-push scan never reached them. - Discover candidate SHAs from both trunk.yml pushes and the scheduled per-arch workflows (fetch_scheduled_commits / fetch_candidate_commits), deduped newest-first; a 40-hex guard drops stray non-row output. - Hold back SHAs newer than the newest scheduled run so a lagging mi300/mi200/navi batch can still join that SHA's report, yielding ONE combined parity report per SHA (parity.yml's matrix already emits a per-arch artifact plus a merged summary) once every arch that ran has finished. When no scheduled runs exist, nothing is held (mi350/trunk behaves exactly as before). - Expand ARCHS_IN to mi350 mi300 mi200 navi31. - Per-config CUDA-baseline gating: a SHA whose CUDA jobs for a test config did not run (e.g. a failed trunk run that never launched CUDA default) has no baseline for that config, so exclude just that config from the dispatch (via parity.yml's exclude_* inputs) instead of emitting a bogus all-MISSED column, and drop its ROCm shards from the completion gate. No parity_job_config.json change: mi300 has no fallback; its default/ distributed/inductor come from rocm-mi300 / periodic-rocm-mi300 / inductor-rocm-mi300, which run together on one scheduled SHA.
Treat the daily rocm-preview lane as a scheduled architecture source and remove the retired nightly exclusion.
… job failed When an upstream config's jobs are missing (e.g. ROCm distributed test job skipped during GPU capacity crunches), download_testlogs exits non-zero and, with `set -o pipefail`, aborted the whole generate-parity job before the CSV was built or uploaded -- so no report at all for that arch. Capture download_testlogs' real exit code instead of aborting, let the report generate and upload from whatever downloaded (default/inductor), then fail the job at the end via a dedicated "Flag incomplete download" step. Net effect for e.g. mi300 with distributed off: the per-arch report + artifact are still produced (and summarize still aggregates via `if: !cancelled()`), but the arch job is clearly marked failed so the incomplete download is visible.
… the run mi300's default/distributed/inductor come from three separately-scheduled workflows (rocm-mi300 / periodic-rocm-mi300 / inductor-rocm-mi300) that do not always land on the same SHA. When one config's workflow run did not run for a SHA, download_testlogs raised "workflow not found ... increasing max_pages" and aborted the whole download -- dropping the configs that DID run and producing no report. (The message was also misleading: download_workflow_run already filters by head_sha, so it's not a pagination problem -- the run genuinely doesn't exist for that SHA.) The parity-auto gate can't prevent this per-arch: it dispatches arches in one combined run, so its dispatch-level exclude flags can't drop a config for just one arch. Handle it in download_testlogs (which runs per-arch): when a config's ROCm workflow (default/distributed/inductor) or the CUDA inductor baseline is absent for the SHA, log it, record it in error_msgs, and skip just that config. The report is still built from the available configs, and the job is still flagged failed at exit so the gap stays visible.
…dic rerun_disabled trunk run) The ROCm inductor block resolved its own trunk run via download_workflow_run, which could return the periodic rerun_disabled_tests/mem_leak_check trunk run (only variant inductor jobs) instead of the canonical push run. Reuse trunk_full_wf like default/distributed already do.
The log-based flaky detector inferred flakiness from the last individual "... PASSED" line seen before a "Test succeeded in new process" marker. That heuristic attaches the flaky flag to whatever test happened to pass most recently, which is often an unrelated test that passed normally much earlier in the shard -- not the test that actually crashed/failed and then passed on rerun. Concrete example (run 29255582284, mi350, inductor/test_compiled_autograd): test_wrap_recreates_contexts_for_repeated_runs passed normally at ~16% and was wrongly reported as flaky, while the test that truly failed-then-passed on rerun was FuncTorchHigherOrderOpTestsWithCompiledAutograd::test_jacfwd. run_test.py already prints an authoritative per-shard summary naming the exact tests: The following tests failed and then succeeded when run in a new process ['test/inductor/test_compiled_autograd.py::...::test_jacfwd'] Parse that line directly instead of guessing. Verified against the real ROCm and CUDA logs from the run above: test_wrap is no longer flagged and test_jacfwd is correctly reported on both platforms.
This was referenced Aug 11, 2026
Match the unique rocm-preview lane without pinning its OS, Python, or runner suffix so routine upstream renames do not disable auto-parity.
Resolve the stacked Preview regex change while preserving canonical per-shard job URLs. # Conflicts: # .automation_scripts/pytorch-unit-test-scripts/parity_job_config.json
Use the restored eight-shard default matrix so parity downloads every live MI300 shard.
Use the restored eight-shard default matrix so parity downloads every live MI300 shard.
Resolve the Preview matcher in favor of the resilient workflow token and refresh fallback shard counts to current upstream topology.
Preserve dynamic topology, artifact-host resolution, and shard discovery while making absent ROCm or CUDA configurations non-fatal to report generation.
Prevent partial architecture runs from dispatching parity before all required, non-excluded test configurations have produced check-runs.
ethanwee1
changed the base branch from
ethanwee/parity-preview-replacement
to
ethanwee/parity-flaky-develop
August 25, 2026 19:57
jithunnair-amd
pushed a commit
that referenced
this pull request
Aug 25, 2026
## Summary - replace the retired `nightly` parity key with `preview` - match the live daily `rocm-preview` `linux-noble-rocm-preview-py3.12-mi350` jobs - use the current 8/3/2 default/distributed/inductor fallback matrix This clean replacement is based on current `develop`; it supersedes #3406, whose pre-reset history cannot merge without a force-push or unrelated-history merge. ## Test plan - [x] Parse parity config JSON - [x] Compile `download_testlogs` - [x] Parse `parity.yml` as YAML - [x] Verify all 13 test jobs against scheduled [pytorch/pytorch run 31344407133](https://github.com/pytorch/pytorch/actions/runs/31344407133) - [x] Run end-to-end Preview parity dispatch ## Dependencies Land this before the Preview auto-trigger update in #3555 and before retargeting the operational parity PRs. Made with [Cursor](https://cursor.com) ## Alignment landing order This is the root dependency. After it lands, retarget the stacked PRs to `develop` in this order: 1. #3535 dynamic workflow/prefix/shard discovery 2. #3536 canonical per-shard job URLs 3. #3523 consolidated classifier rules (independent base; may land earlier) 4. #3559 variant-run filtering 5. #3558 graceful missing-config handling 6. #3557 partial-report preservation 7. #3560 authoritative flaky attribution 8. #3556 latest-SHA resolution 9. #3555 scheduled multi-architecture automation last Validation: full Preview [run 31506711303](https://github.com/ethanwee1/pytorch/actions/runs/31506711303) and combined Preview+MI350 [run 31507307756](https://github.com/ethanwee1/pytorch/actions/runs/31507307756) both succeeded on matching SHA `cf18dda7`. ## August 25 refresh - Merged current ROCm `develop` and reran JSON, Python, and workflow YAML validation. - The previously documented Preview and combined Preview+MI350 runs satisfy the end-to-end checklist item.
This was referenced Aug 25, 2026
ethanwee1
changed the base branch from
ethanwee/parity-flaky-develop
to
develop
August 26, 2026 16:10
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Clean replacement for #3397 on the current
develophistory. This PR is stacked on #3554; retarget it todevelopafter #3554 lands.Test plan
parity-auto.ymlas YAMLpreviewis in scope andnightlyis absentrocm-previewdiscoveryDependencies
Made with Cursor
Alignment landing order
Automation lands last, after #3554, #3535, #3536, and the operational resilience PRs. Its PR dry-run 31506030239 loaded
preview, discoveredrocm-preview, and produced the correct workflow/job regex maps. End-to-end combined validation: 31507307756.August 25 refresh
Stacked last on the complete operational parity stack ([CI] Parity: use authoritative flaky-test summaries #3560).
Added per-architecture
test_configsand now require every non-excluded config before dispatch.Local regression for
6a0eb752...reports MI200 missing distributed/inductor; JSON and actionlint validation pass.Dry-run and real combined dispatch evidence will be recorded below after completion.
Parity stack merge order
All open parity PRs are targeted to
develop. Merge in this order:#3259 is a separate follow-up: reconstruct FrameworkWeb ingestion after the operational stack lands.