Skip to content

Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds - #8602

Merged
atalman merged 1 commit into
pytorch:mainfrom
shoumikhin:fix-aarch64-miniforge-pin
Aug 21, 2026
Merged

Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds#8602
atalman merged 1 commit into
pytorch:mainfrom
shoumikhin:fix-aarch64-miniforge-pin

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

Summary

Linux aarch64 nightly wheel builds have been failing since 2026-08-21 for the domain
libraries (torchvision, torchaudio, ao, executorch). Nothing in any PyTorch repository
changed.

build_wheels_linux.yml bootstraps conda on aarch64 by downloading Miniforge from the
releases/latest pointer. That pointer is mutable. It moved, and the newer installer puts
the base environment on Python 3.14. The shared setup-binary-builds action then runs
conda install -y conda=25.3.0, and conda 25.3.0 has no Python 3.14 build, so the solve is
unsatisfiable:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package conda-25.3.0-py310h4c7bcd0_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed
...
Pins seem to be involved in the conflict. Currently pinned specs:
 - python=3.14

This step runs in the base environment before the target environment is created, so all
target Python versions fail identically. In the last torchvision aarch64 nightly, 80 of 81
jobs failed, every one of them at this step.

From the job logs:

when (UTC) what the installer produced conda install -y conda=25.3.0
2026-08-20 19:49 conda 26.3.2, base Python 3.13 succeeded, as a downgrade
2026-08-21 11:48 conda 26.5.3, base Python 3.14 fails

Miniforge 26.5.3-0 was published on 2026-08-15, but releases/latest still served 26.3.2 on
08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on its own, days after
that release already existed, which is why this broke with no commit behind it.

Fix

Pin the installer to 26.3.2-3, the release that was last green, and add -f to the curl.

This restores exactly the base environment the last green run had. Verified by downloading
the tagged aarch64 asset and comparing package build strings against the last green job log:

package last green run in the 26.3.2-3 asset
conda 26.3.2-py313hd81a959_1 yes
conda-libmamba-solver 26.4.2-pyhd8ed1ab_0 yes
libmambapy 2.5.0-py313h44afa9f_0 yes
mamba 2.5.0-hd347465_0 yes
openssl 3.6.2-h546c87b_0 yes
python 3.13.13-h11c0449_100_cp313 yes

The tagged release publishes the asset under the same unversioned filename, so only the path
segment of the URL changes.

The -f matters now that the URL carries a hand written tag. Without it, curl treats a 404 as
a successful transfer, writes the error body into the installer file and exits 0. chmod +x
then succeeds and the shell executes the error page, which reports a syntax error on line 1
and exit 127 instead of a download failure.

Why not change the conda pin instead

The guard in setup-binary-builds is if [[ "${RUNNER_OS}" != "macOS" ]], which keys on the
operating system rather than on whether the base environment actually needs the pin. aarch64
is now a counter-example, so that guard is worth revisiting. It should not be revisited in
this change:

  • On aarch64 the pin is a downgrade, so skipping it hands the Python 3.15 solve to conda
    26.5.3. Python 3.15 resolves from the conda-forge/label/python_dev and
    conda-forge/label/python_rc pre-release labels, and the only note in this repository about
    a newer base conda against those labels is the comment in the macOS branch saying it returns
    400. 20 of torchvision's 81 aarch64 jobs are 3.15 or 3.15t, and all of them were green with
    the pin in place.
  • Keeping the pin only for 3.15 does not work either, because that reinstalls conda 25.3.0 on
    a Python 3.14 base, which is the failure above.
  • Bumping the pin for everyone was already proposed in [macOS CI] Bump pinned conda to 26.5.3 to fix osx-arm64 nightly builds (base env now Python 3.14) #8284 and dropped before merge. The
    comment added there records that upgrading the Linux and Windows containers yields
    NoChannelsConfiguredError.

A good follow-up is to measure whether conda 26.5.3 can resolve python=3.15.0b4 from those
two labels on linux-aarch64. If it can, the pin can be skipped on aarch64 and the installer
can float again. If it cannot, the pin stays and the real reason for it should be written down,
because the comment above it currently gives a reason that does not hold on aarch64.

Test plan

What was checked before opening this:

  • x86_64, Windows and macOS cannot be affected. The changed step is gated on
    inputs.architecture == 'aarch64', and setup-binary-builds is not touched.
  • The 26.3.2-3 release publishes Miniforge3-Linux-aarch64.sh under the same filename this
    step already uses.
  • The package build strings in that asset match the last green run, as listed above.
  • curl behaviour with and without -f on a 404 was reproduced locally against a local server.

What still needs CI, and what a reviewer should look for:

  • One aarch64 wheel build on this branch. In the log, the installer should report conda
    26.3.2, and conda install -y conda=25.3.0 should solve. A 3.15 job is the most valuable
    one to include.
  • This has not been run through an aarch64 nightly yet, so that confirmation is still open.

Authored with assistance from Claude Code (AI assistant).

…ilds

The aarch64 bootstrap downloaded Miniforge from the releases/latest pointer.
That pointer moved to 26.5.3, whose base env is on Python 3.14, and the shared
setup-binary-builds action then fails installing conda=25.3.0, which has no
Python 3.14 build. Pin the installer to the release that was last green.
@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

@shoumikhin is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 21, 2026

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! cc @atalman if you think we need to cherry pick this into 2.14

@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
torchci Ignored Ignored Aug 21, 2026 6:11pm

Request Review

@atalman
atalman merged commit 2f89f62 into pytorch:main Aug 21, 2026
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants