Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds (#8602) - #8604
Open
atalman wants to merge 1 commit into
Open
Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds (#8602)#8604atalman wants to merge 1 commit into
atalman wants to merge 1 commit into
Conversation
…ilds (#8602) ## Summary Linux aarch64 nightly wheel builds have been failing since 2026-08-21 for the domain libraries (torchvision, torchaudio, ao, executorch). Nothing in any PyTorch repository changed. `build_wheels_linux.yml` bootstraps conda on aarch64 by downloading Miniforge from the `releases/latest` pointer. That pointer is mutable. It moved, and the newer installer puts the base environment on Python 3.14. The shared `setup-binary-builds` action then runs `conda install -y conda=25.3.0`, and conda 25.3.0 has no Python 3.14 build, so the solve is unsatisfiable: ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-25.3.0-py310h4c7bcd0_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed ... Pins seem to be involved in the conflict. Currently pinned specs: - python=3.14 ``` This step runs in the base environment before the target environment is created, so all target Python versions fail identically. In the last torchvision aarch64 nightly, 80 of 81 jobs failed, every one of them at this step. From the job logs: | when (UTC) | what the installer produced | `conda install -y conda=25.3.0` | | --- | --- | --- | | 2026-08-20 19:49 | conda 26.3.2, base Python 3.13 | succeeded, as a downgrade | | 2026-08-21 11:48 | conda 26.5.3, base Python 3.14 | fails | Miniforge 26.5.3-0 was published on 2026-08-15, but `releases/latest` still served 26.3.2 on 08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on its own, days after that release already existed, which is why this broke with no commit behind it. ## Fix Pin the installer to `26.3.2-3`, the release that was last green, and add `-f` to the curl. This restores exactly the base environment the last green run had. Verified by downloading the tagged aarch64 asset and comparing package build strings against the last green job log: | package | last green run | in the 26.3.2-3 asset | | --- | --- | --- | | conda | `26.3.2-py313hd81a959_1` | yes | | conda-libmamba-solver | `26.4.2-pyhd8ed1ab_0` | yes | | libmambapy | `2.5.0-py313h44afa9f_0` | yes | | mamba | `2.5.0-hd347465_0` | yes | | openssl | `3.6.2-h546c87b_0` | yes | | python | `3.13.13-h11c0449_100_cp313` | yes | The tagged release publishes the asset under the same unversioned filename, so only the path segment of the URL changes. The `-f` matters now that the URL carries a hand written tag. Without it, curl treats a 404 as a successful transfer, writes the error body into the installer file and exits 0. `chmod +x` then succeeds and the shell executes the error page, which reports a syntax error on line 1 and exit 127 instead of a download failure. ## Why not change the conda pin instead The guard in `setup-binary-builds` is `if [[ "${RUNNER_OS}" != "macOS" ]]`, which keys on the operating system rather than on whether the base environment actually needs the pin. aarch64 is now a counter-example, so that guard is worth revisiting. It should not be revisited in this change: - On aarch64 the pin is a downgrade, so skipping it hands the Python 3.15 solve to conda 26.5.3. Python 3.15 resolves from the `conda-forge/label/python_dev` and `conda-forge/label/python_rc` pre-release labels, and the only note in this repository about a newer base conda against those labels is the comment in the macOS branch saying it returns 400. 20 of torchvision's 81 aarch64 jobs are 3.15 or 3.15t, and all of them were green with the pin in place. - Keeping the pin only for 3.15 does not work either, because that reinstalls conda 25.3.0 on a Python 3.14 base, which is the failure above. - Bumping the pin for everyone was already proposed in #8284 and dropped before merge. The comment added there records that upgrading the Linux and Windows containers yields `NoChannelsConfiguredError`. A good follow-up is to measure whether conda 26.5.3 can resolve `python=3.15.0b4` from those two labels on linux-aarch64. If it can, the pin can be skipped on aarch64 and the installer can float again. If it cannot, the pin stays and the real reason for it should be written down, because the comment above it currently gives a reason that does not hold on aarch64. ## Test plan What was checked before opening this: - x86_64, Windows and macOS cannot be affected. The changed step is gated on `inputs.architecture == 'aarch64'`, and `setup-binary-builds` is not touched. - The `26.3.2-3` release publishes `Miniforge3-Linux-aarch64.sh` under the same filename this step already uses. - The package build strings in that asset match the last green run, as listed above. - curl behaviour with and without `-f` on a 404 was reproduced locally against a local server. What still needs CI, and what a reviewer should look for: - One aarch64 wheel build on this branch. In the log, the installer should report conda 26.3.2, and `conda install -y conda=25.3.0` should solve. A 3.15 job is the most valuable one to include. - This has not been run through an aarch64 nightly yet, so that confirmation is still open. Authored with assistance from Claude Code (AI assistant). (cherry picked from commit 2f89f62)
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
huydhn
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Linux aarch64 nightly wheel builds have been failing since 2026-08-21 for the domain
libraries (torchvision, torchaudio, ao, executorch). Nothing in any PyTorch repository
changed.
build_wheels_linux.ymlbootstraps conda on aarch64 by downloading Miniforge from thereleases/latestpointer. That pointer is mutable. It moved, and the newer installer putsthe base environment on Python 3.14. The shared
setup-binary-buildsaction then runsconda install -y conda=25.3.0, and conda 25.3.0 has no Python 3.14 build, so the solve isunsatisfiable:
This step runs in the base environment before the target environment is created, so all
target Python versions fail identically. In the last torchvision aarch64 nightly, 80 of 81
jobs failed, every one of them at this step.
From the job logs:
conda install -y conda=25.3.0Miniforge 26.5.3-0 was published on 2026-08-15, but
releases/lateststill served 26.3.2 on08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on its own, days after
that release already existed, which is why this broke with no commit behind it.
Fix
Pin the installer to
26.3.2-3, the release that was last green, and add-fto the curl.This restores exactly the base environment the last green run had. Verified by downloading
the tagged aarch64 asset and comparing package build strings against the last green job log:
| package | last green run | in the 26.3.2-3 asset | | --- | --- | --- |
| conda |
26.3.2-py313hd81a959_1| yes || conda-libmamba-solver |
26.4.2-pyhd8ed1ab_0| yes | | libmambapy |2.5.0-py313h44afa9f_0| yes || mamba |
2.5.0-hd347465_0| yes || openssl |
3.6.2-h546c87b_0| yes || python |
3.13.13-h11c0449_100_cp313| yes |The tagged release publishes the asset under the same unversioned filename, so only the path
segment of the URL changes.
The
-fmatters now that the URL carries a hand written tag. Without it, curl treats a 404 asa successful transfer, writes the error body into the installer file and exits 0.
chmod +xthen succeeds and the shell executes the error page, which reports a syntax error on line 1
and exit 127 instead of a download failure.
Why not change the conda pin instead
The guard in
setup-binary-buildsisif [[ "${RUNNER_OS}" != "macOS" ]], which keys on theoperating system rather than on whether the base environment actually needs the pin. aarch64
is now a counter-example, so that guard is worth revisiting. It should not be revisited in
this change:
26.5.3. Python 3.15 resolves from the
conda-forge/label/python_devandconda-forge/label/python_rcpre-release labels, and the only note in this repository abouta newer base conda against those labels is the comment in the macOS branch saying it returns
the pin in place.
a Python 3.14 base, which is the failure above.
comment added there records that upgrading the Linux and Windows containers yields
NoChannelsConfiguredError.A good follow-up is to measure whether conda 26.5.3 can resolve
python=3.15.0b4from thosetwo labels on linux-aarch64. If it can, the pin can be skipped on aarch64 and the installer
can float again. If it cannot, the pin stays and the real reason for it should be written down,
because the comment above it currently gives a reason that does not hold on aarch64.
Test plan
What was checked before opening this:
inputs.architecture == 'aarch64', andsetup-binary-buildsis not touched.26.3.2-3release publishesMiniforge3-Linux-aarch64.shunder the same filename thisstep already uses.
-fon a 404 was reproduced locally against a local server.What still needs CI, and what a reviewer should look for:
26.3.2, and
conda install -y conda=25.3.0should solve. A 3.15 job is the most valuableone to include.
Authored with assistance from Claude Code (AI assistant).
(cherry picked from commit 2f89f62)