Poll for new RC tags in the runner-group reconcile - #8510
Open
atalman wants to merge 1 commit into
Open
Conversation
v2.14.0-rc2 was tagged at 12:36 UTC today and all 7 mt-rel-* jobs of its docker-release run have been queued ever since, because the runner-group allow-list still pins v2.14.0-rc1 and GitHub matches selected_workflows on the exact ref. pytorch#8502 taught the reconcile to allow-list RC tags, but nothing re-runs it when a tag is cut: RC tags are created in pytorch/pytorch, which cannot trigger a workflow here, and the only automatic trigger was a push touching generate_binary_build_matrix.py. The last apply was 2026-08-11T23:34Z, before rc2 existed. So the allow-list is only ever as fresh as the last unrelated matrix change or the last time somebody remembered to dispatch by hand. Add an hourly schedule, and let it apply -- a polling run that only ever dry-ran would discover the new RC and do nothing about it. Two related holes closed while here: - release_manage_runner_groups.py and this workflow were absent from the push paths, so a change to the reconcile logic never deployed itself. That is why pytorch#8502 needed a manual dispatch a minute after it merged. - concurrency was cancel-in-progress, which was harmless when runs were rare but now lets the hourly run kill a release-time manual dispatch mid-apply and leave the allow-list half-written. Runs are short API calls, so queue them instead.
|
@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel. A member of the Team first needs to authorize it. |
meta-cla
Bot
added
the
CLA Signed
huydhn
reviewed
Aug 12, 2026
| description: "Apply changes (otherwise dry-run)" | ||
| type: boolean | ||
| default: false | ||
| schedule: |
Contributor
There was a problem hiding this comment.
I thought that we didn't want a schedule run and would rely only on workflow dispatch? Having it too is ok though as it does no harm IMO
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this pull request
Aug 12, 2026
Release binaries build from **tags**, and the self-hosted runner groups they need are allow-listed per **exact ref** by a reconcile job in [pytorch/test-infra](https://github.com/pytorch/test-infra/actions/workflows/release-manage-runner-groups.yml). That job cannot observe a tag push in this repo, so today the allow-list only refreshes when an unrelated change lands in test-infra, or when somebody dispatches it by hand. When it is stale, a freshly cut RC is not authorized for the release runner groups and its build jobs **queue indefinitely, with no error**. That is what happened to `v2.14.0-rc2`. It was tagged at 12:36 UTC; all 7 `mt-rel-*` jobs of its [docker-release run](https://github.com/pytorch/pytorch/actions/runs/31597398139) were still queued ~2h later. The allow-list still pinned `v2.14.0-rc1`, because the last reconcile had run the previous evening — before rc2 existed. A test-infra dry run confirmed the exact gap: ``` Target refs: [... 'refs/tags/v2.14.0-rc2', ...] Discovered 9 release workflow(s) on pytorch/pytorch@v2.14.0-rc2 + .../docker-release.yml@refs/tags/v2.14.0-rc2 - .../docker-release.yml@refs/tags/v2.14.0-rc1 ``` ## What this does On a release tag push, dispatch the test-infra reconcile. It is dispatched on test-infra's `main` rather than at this tag — the reconcile discovers the new tag itself via `git/matching-refs`, and the workflow only exists on that default branch. Design points worth noting: - **`runs-on: ubuntu-latest`** — deliberately GitHub-hosted, so this is never gated by the very runner groups it exists to unblock. - **Tag filters match `create_release.yml` and `docker-release.yml`** (`v[0-9]+.[0-9]+.[0-9]+` and the `-rc[0-9]+` form), so it fires for exactly the tags that build release binaries, and not for `ciflow/*` tags. - **A failed dispatch fails the job**, surfacing in the tag's checks, rather than silently leaving the queue stuck — which is the failure mode this is meant to eliminate. - **`permissions: contents: read`**; the cross-repo call uses `GH_PYTORCHBOT_TOKEN`. ## Two things reviewers should check **1. Token scope.** This uses `secrets.GH_PYTORCHBOT_TOKEN`, which is already used cross-repo here — `nightly.yml` and `weekly.yml` pass it to the commit-hash-update action targeting other org repos. If it is a classic PAT with `repo` scope it can already dispatch test-infra workflows and no new secret is needed, but **I could not verify its scopes**. Someone who can should confirm `actions: write` on `pytorch/test-infra` before relying on this. If it turns out to be insufficient, the alternative is a narrowly scoped app token. **2. This does nothing for 2.14 until cherry-picked.** A tag build runs the workflow as of the tagged commit, so landing here only covers tags cut from `main`. For `v2.14.0-rc3` to be covered it must be on `release/2.14` before rc3 is cut. @atalman is handling that cherry-pick. ## Related pytorch/test-infra#8510 adds an hourly poll on the test-infra side. The two are complementary rather than redundant: this hook is the fast path (~1-2 min), and the poll is the backstop for tags cut from branches that do not carry this workflow, for manual runner-group edits drifting, and for reconcile-script changes. ## Test plan `lintrunner` ACTIONLINT clean (pinned actionlint via `.lintbin`). The dispatch path cannot be exercised before merge, since tag-triggered workflows do not run from a PR ref — it will first execute on the next release tag cut from a branch carrying it. Pull Request resolved: #193186 Approved by: https://github.com/huydhn
atalman
added a commit
to pytorch/pytorch
that referenced
this pull request
Aug 13, 2026
Refresh release runner groups when a release tag is pushed (#193186) Release binaries build from **tags**, and the self-hosted runner groups they need are allow-listed per **exact ref** by a reconcile job in [pytorch/test-infra](https://github.com/pytorch/test-infra/actions/workflows/release-manage-runner-groups.yml). That job cannot observe a tag push in this repo, so today the allow-list only refreshes when an unrelated change lands in test-infra, or when somebody dispatches it by hand. When it is stale, a freshly cut RC is not authorized for the release runner groups and its build jobs **queue indefinitely, with no error**. That is what happened to `v2.14.0-rc2`. It was tagged at 12:36 UTC; all 7 `mt-rel-*` jobs of its [docker-release run](https://github.com/pytorch/pytorch/actions/runs/31597398139) were still queued ~2h later. The allow-list still pinned `v2.14.0-rc1`, because the last reconcile had run the previous evening — before rc2 existed. A test-infra dry run confirmed the exact gap: ``` Target refs: [... 'refs/tags/v2.14.0-rc2', ...] Discovered 9 release workflow(s) on pytorch/pytorch@v2.14.0-rc2 + .../docker-release.yml@refs/tags/v2.14.0-rc2 - .../docker-release.yml@refs/tags/v2.14.0-rc1 ``` ## What this does On a release tag push, dispatch the test-infra reconcile. It is dispatched on test-infra's `main` rather than at this tag — the reconcile discovers the new tag itself via `git/matching-refs`, and the workflow only exists on that default branch. Design points worth noting: - **`runs-on: ubuntu-latest`** — deliberately GitHub-hosted, so this is never gated by the very runner groups it exists to unblock. - **Tag filters match `create_release.yml` and `docker-release.yml`** (`v[0-9]+.[0-9]+.[0-9]+` and the `-rc[0-9]+` form), so it fires for exactly the tags that build release binaries, and not for `ciflow/*` tags. - **A failed dispatch fails the job**, surfacing in the tag's checks, rather than silently leaving the queue stuck — which is the failure mode this is meant to eliminate. - **`permissions: contents: read`**; the cross-repo call uses `GH_PYTORCHBOT_TOKEN`. ## Two things reviewers should check **1. Token scope.** This uses `secrets.GH_PYTORCHBOT_TOKEN`, which is already used cross-repo here — `nightly.yml` and `weekly.yml` pass it to the commit-hash-update action targeting other org repos. If it is a classic PAT with `repo` scope it can already dispatch test-infra workflows and no new secret is needed, but **I could not verify its scopes**. Someone who can should confirm `actions: write` on `pytorch/test-infra` before relying on this. If it turns out to be insufficient, the alternative is a narrowly scoped app token. **2. This does nothing for 2.14 until cherry-picked.** A tag build runs the workflow as of the tagged commit, so landing here only covers tags cut from `main`. For `v2.14.0-rc3` to be covered it must be on `release/2.14` before rc3 is cut. @atalman is handling that cherry-pick. ## Related pytorch/test-infra#8510 adds an hourly poll on the test-infra side. The two are complementary rather than redundant: this hook is the fast path (~1-2 min), and the poll is the backstop for tags cut from branches that do not carry this workflow, for manual runner-group edits drifting, and for reconcile-script changes. ## Test plan `lintrunner` ACTIONLINT clean (pinned actionlint via `.lintbin`). The dispatch path cannot be exercised before merge, since tag-triggered workflows do not run from a PR ref — it will first execute on the next release tag cut from a branch carrying it. Pull Request resolved: #193186 Approved by: https://github.com/huydhn (cherry picked from commit 5c9459b) Co-authored-by: Andrey Talman <atalman@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v2.14.0-rc2was tagged at 12:36 UTC today, and all 7mt-rel-*jobs of its docker-release run have been queued ever since. The runner-group allow-list still pinsv2.14.0-rc1, and GitHub matchesselected_workflowson the exact ref, so nothing is authorized to pick them up.Why the allow-list is stale
#8502 taught the reconcile to allow-list RC tags — that part works. What is missing is anything that re-runs it when a tag is cut.
RC tags are created in pytorch/pytorch, which cannot trigger a workflow in test-infra. The only automatic trigger here was a push touching
generate_binary_build_matrix.py. So:v2.14.0-rc1taggedv2.14.0-rc2taggedrefs/tags/v2.14.0-rc2mainsince Aug 11 23:34The allow-list is only ever as fresh as the last unrelated matrix change, or the last time somebody remembered to dispatch by hand.
#8502 deliberately skipped a cron — "the desired state only changes at release milestones, so run on demand". That reasoning is right about frequency and wrong about observability: a new RC is a milestone, but it happens in another repo, so "on demand" silently means "a human notices the queue".
Changes
Hourly schedule, which applies. A polling run that only ever dry-ran would discover the new RC and then do nothing about it, so
SHOULD_APPLYnow includesschedule. Cron is at:23to dodge the top-of-hour scheduling backlog. This caps staleness at ~1h instead of unbounded.release_manage_runner_groups.pyand this workflow added to the push paths. They were absent, so a change to the reconcile logic never deployed itself — exactly why #8502 needed a manual dispatch a minute after merging. Second-order, but it is how the first hole stayed invisible.cancel-in-progress: false. Harmless when runs were rare; with an hourly trigger it would let the scheduled run kill a release-time manual dispatch mid-apply and leave the allow-list half-written. Runs are short API calls, so queueing costs little.Verified
actionlint(1.6.21, the pinned version) clean. The apply gate resolves as intended for every event:applyinputThe PR path stays discovery-only: the token is gated behind
environment: ${{ github.ref == 'refs/heads/main' && 'runner-group' || '' }}, so forks and PRs never see it.This does not unblock rc2 by itself
Landing this starts the polling, but the currently-queued jobs need the allow-list updated now. Someone with access should dispatch release-manage-runner-groups on
mainwithapply: true— the allow-list is evaluated when a job starts, so the queued jobs should pick up without a re-run. Worth doing a dry run first to confirmv2.14.0-rc2appears in the computed set.I inferred the stale-allow-list diagnosis from the run history plus #8502's newest-only pinning; I cannot read the runner-group config directly, since that needs
RUNNER_GROUP_TOKEN. If anapply: truerun does not release the queue, the cause is elsewhere and this PR is still worth having on its own merits.Frequency is a judgement call
Hourly trades ~24 cheap API-only runs/day against a worst case of ~1h of queued release builds. Happy to tighten to
*/30or*/15if a release-day hour of dead queue is too much.