Skip to content

Commit 2f89f62

Browse files
authored
Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds (#8602)
## Summary Linux aarch64 nightly wheel builds have been failing since 2026-08-21 for the domain libraries (torchvision, torchaudio, ao, executorch). Nothing in any PyTorch repository changed. `build_wheels_linux.yml` bootstraps conda on aarch64 by downloading Miniforge from the `releases/latest` pointer. That pointer is mutable. It moved, and the newer installer puts the base environment on Python 3.14. The shared `setup-binary-builds` action then runs `conda install -y conda=25.3.0`, and conda 25.3.0 has no Python 3.14 build, so the solve is unsatisfiable: ``` LibMambaUnsatisfiableError: Encountered problems while solving: - package conda-25.3.0-py310h4c7bcd0_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed ... Pins seem to be involved in the conflict. Currently pinned specs: - python=3.14 ``` This step runs in the base environment before the target environment is created, so all target Python versions fail identically. In the last torchvision aarch64 nightly, 80 of 81 jobs failed, every one of them at this step. From the job logs: | when (UTC) | what the installer produced | `conda install -y conda=25.3.0` | | --- | --- | --- | | 2026-08-20 19:49 | conda 26.3.2, base Python 3.13 | succeeded, as a downgrade | | 2026-08-21 11:48 | conda 26.5.3, base Python 3.14 | fails | Miniforge 26.5.3-0 was published on 2026-08-15, but `releases/latest` still served 26.3.2 on 08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on its own, days after that release already existed, which is why this broke with no commit behind it. ## Fix Pin the installer to `26.3.2-3`, the release that was last green, and add `-f` to the curl. This restores exactly the base environment the last green run had. Verified by downloading the tagged aarch64 asset and comparing package build strings against the last green job log: | package | last green run | in the 26.3.2-3 asset | | --- | --- | --- | | conda | `26.3.2-py313hd81a959_1` | yes | | conda-libmamba-solver | `26.4.2-pyhd8ed1ab_0` | yes | | libmambapy | `2.5.0-py313h44afa9f_0` | yes | | mamba | `2.5.0-hd347465_0` | yes | | openssl | `3.6.2-h546c87b_0` | yes | | python | `3.13.13-h11c0449_100_cp313` | yes | The tagged release publishes the asset under the same unversioned filename, so only the path segment of the URL changes. The `-f` matters now that the URL carries a hand written tag. Without it, curl treats a 404 as a successful transfer, writes the error body into the installer file and exits 0. `chmod +x` then succeeds and the shell executes the error page, which reports a syntax error on line 1 and exit 127 instead of a download failure. ## Why not change the conda pin instead The guard in `setup-binary-builds` is `if [[ "${RUNNER_OS}" != "macOS" ]]`, which keys on the operating system rather than on whether the base environment actually needs the pin. aarch64 is now a counter-example, so that guard is worth revisiting. It should not be revisited in this change: - On aarch64 the pin is a downgrade, so skipping it hands the Python 3.15 solve to conda 26.5.3. Python 3.15 resolves from the `conda-forge/label/python_dev` and `conda-forge/label/python_rc` pre-release labels, and the only note in this repository about a newer base conda against those labels is the comment in the macOS branch saying it returns 400. 20 of torchvision's 81 aarch64 jobs are 3.15 or 3.15t, and all of them were green with the pin in place. - Keeping the pin only for 3.15 does not work either, because that reinstalls conda 25.3.0 on a Python 3.14 base, which is the failure above. - Bumping the pin for everyone was already proposed in #8284 and dropped before merge. The comment added there records that upgrading the Linux and Windows containers yields `NoChannelsConfiguredError`. A good follow-up is to measure whether conda 26.5.3 can resolve `python=3.15.0b4` from those two labels on linux-aarch64. If it can, the pin can be skipped on aarch64 and the installer can float again. If it cannot, the pin stays and the real reason for it should be written down, because the comment above it currently gives a reason that does not hold on aarch64. ## Test plan What was checked before opening this: - x86_64, Windows and macOS cannot be affected. The changed step is gated on `inputs.architecture == 'aarch64'`, and `setup-binary-builds` is not touched. - The `26.3.2-3` release publishes `Miniforge3-Linux-aarch64.sh` under the same filename this step already uses. - The package build strings in that asset match the last green run, as listed above. - curl behaviour with and without `-f` on a 404 was reproduced locally against a local server. What still needs CI, and what a reviewer should look for: - One aarch64 wheel build on this branch. In the log, the installer should report conda 26.3.2, and `conda install -y conda=25.3.0` should solve. A 3.15 job is the most valuable one to include. - This has not been run through an aarch64 nightly yet, so that confirmation is still open. Authored with assistance from Claude Code (AI assistant).
1 parent cb4b1c6 commit 2f89f62

1 file changed

Lines changed: 6 additions & 1 deletion

File tree

.github/workflows/build_wheels_linux.yml

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -201,7 +201,12 @@ jobs:
201201
# when using Python version, less than the conda latest
202202
###############################################################################
203203
echo 'Installing conda-forge'
204-
curl -L -o /mambaforge.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-aarch64.sh
204+
# Pin the installer. The shared setup-binary-builds action installs
205+
# conda=25.3.0 into this base env, and conda 25.3.0 has no Python 3.14
206+
# build. Miniforge releases newer than this one put the base env on
207+
# Python 3.14, which makes that install unsatisfiable and fails every
208+
# aarch64 wheel build. Bump this tag only together with that pin.
209+
curl -fL -o /mambaforge.sh https://github.com/conda-forge/miniforge/releases/download/26.3.2-3/Miniforge3-Linux-aarch64.sh
205210
chmod +x /mambaforge.sh
206211
/mambaforge.sh -b -p /opt/conda
207212
rm /mambaforge.sh

0 commit comments

Comments
 (0)