Skip to content

Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds (#8602) - #8604

Open
atalman wants to merge 1 commit into
release/2.14from
atalman/cherry-pick-8602-2.14
Open

Pin the aarch64 Miniforge installer to unbreak Linux aarch64 wheel builds (#8602)#8604
atalman wants to merge 1 commit into
release/2.14from
atalman/cherry-pick-8602-2.14

Conversation

@atalman

@atalman atalman commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Linux aarch64 nightly wheel builds have been failing since 2026-08-21 for the domain
libraries (torchvision, torchaudio, ao, executorch). Nothing in any PyTorch repository
changed.

build_wheels_linux.yml bootstraps conda on aarch64 by downloading Miniforge from the
releases/latest pointer. That pointer is mutable. It moved, and the newer installer puts
the base environment on Python 3.14. The shared setup-binary-builds action then runs
conda install -y conda=25.3.0, and conda 25.3.0 has no Python 3.14 build, so the solve is
unsatisfiable:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package conda-25.3.0-py310h4c7bcd0_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed
...
Pins seem to be involved in the conflict. Currently pinned specs:
 - python=3.14

This step runs in the base environment before the target environment is created, so all
target Python versions fail identically. In the last torchvision aarch64 nightly, 80 of 81
jobs failed, every one of them at this step.

From the job logs:

when (UTC) what the installer produced conda install -y conda=25.3.0
2026-08-20 19:49 conda 26.3.2, base Python 3.13 succeeded, as a downgrade
2026-08-21 11:48 conda 26.5.3, base Python 3.14 fails

Miniforge 26.5.3-0 was published on 2026-08-15, but releases/latest still served 26.3.2 on
08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on its own, days after
that release already existed, which is why this broke with no commit behind it.

Fix

Pin the installer to 26.3.2-3, the release that was last green, and add -f to the curl.

This restores exactly the base environment the last green run had. Verified by downloading
the tagged aarch64 asset and comparing package build strings against the last green job log:

| package | last green run | in the 26.3.2-3 asset | | --- | --- | --- |
| conda | 26.3.2-py313hd81a959_1 | yes |
| conda-libmamba-solver | 26.4.2-pyhd8ed1ab_0 | yes | | libmambapy | 2.5.0-py313h44afa9f_0 | yes |
| mamba | 2.5.0-hd347465_0 | yes |
| openssl | 3.6.2-h546c87b_0 | yes |
| python | 3.13.13-h11c0449_100_cp313 | yes |

The tagged release publishes the asset under the same unversioned filename, so only the path
segment of the URL changes.

The -f matters now that the URL carries a hand written tag. Without it, curl treats a 404 as
a successful transfer, writes the error body into the installer file and exits 0. chmod +x
then succeeds and the shell executes the error page, which reports a syntax error on line 1
and exit 127 instead of a download failure.

Why not change the conda pin instead

The guard in setup-binary-builds is if [[ "${RUNNER_OS}" != "macOS" ]], which keys on the
operating system rather than on whether the base environment actually needs the pin. aarch64
is now a counter-example, so that guard is worth revisiting. It should not be revisited in
this change:

  • On aarch64 the pin is a downgrade, so skipping it hands the Python 3.15 solve to conda
    26.5.3. Python 3.15 resolves from the conda-forge/label/python_dev and conda-forge/label/python_rc pre-release labels, and the only note in this repository about
    a newer base conda against those labels is the comment in the macOS branch saying it returns
  1. 20 of torchvision's 81 aarch64 jobs are 3.15 or 3.15t, and all of them were green with
    the pin in place.

A good follow-up is to measure whether conda 26.5.3 can resolve python=3.15.0b4 from those
two labels on linux-aarch64. If it can, the pin can be skipped on aarch64 and the installer
can float again. If it cannot, the pin stays and the real reason for it should be written down,
because the comment above it currently gives a reason that does not hold on aarch64.

Test plan

What was checked before opening this:

  • x86_64, Windows and macOS cannot be affected. The changed step is gated on
    inputs.architecture == 'aarch64', and setup-binary-builds is not touched.
  • The 26.3.2-3 release publishes Miniforge3-Linux-aarch64.sh under the same filename this
    step already uses.
  • The package build strings in that asset match the last green run, as listed above.
  • curl behaviour with and without -f on a 404 was reproduced locally against a local server.

What still needs CI, and what a reviewer should look for:

  • One aarch64 wheel build on this branch. In the log, the installer should report conda
    26.3.2, and conda install -y conda=25.3.0 should solve. A 3.15 job is the most valuable
    one to include.
  • This has not been run through an aarch64 nightly yet, so that confirmation is still open.

Authored with assistance from Claude Code (AI assistant).

(cherry picked from commit 2f89f62)

…ilds (#8602)

## Summary

Linux aarch64 nightly wheel builds have been failing since 2026-08-21
for the domain
libraries (torchvision, torchaudio, ao, executorch). Nothing in any
PyTorch repository
changed.

`build_wheels_linux.yml` bootstraps conda on aarch64 by downloading
Miniforge from the
`releases/latest` pointer. That pointer is mutable. It moved, and the
newer installer puts
the base environment on Python 3.14. The shared `setup-binary-builds`
action then runs
`conda install -y conda=25.3.0`, and conda 25.3.0 has no Python 3.14
build, so the solve is
unsatisfiable:

```
LibMambaUnsatisfiableError: Encountered problems while solving:
  - package conda-25.3.0-py310h4c7bcd0_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed
...
Pins seem to be involved in the conflict. Currently pinned specs:
 - python=3.14
```

This step runs in the base environment before the target environment is
created, so all
target Python versions fail identically. In the last torchvision aarch64
nightly, 80 of 81
jobs failed, every one of them at this step.

From the job logs:

| when (UTC) | what the installer produced | `conda install -y
conda=25.3.0` |
| --- | --- | --- |
| 2026-08-20 19:49 | conda 26.3.2, base Python 3.13 | succeeded, as a
downgrade |
| 2026-08-21 11:48 | conda 26.5.3, base Python 3.14 | fails |

Miniforge 26.5.3-0 was published on 2026-08-15, but `releases/latest`
still served 26.3.2 on
08-20 at 19:49 and served 26.5.3 by 08-21 at 11:48. The pointer moved on
its own, days after
that release already existed, which is why this broke with no commit
behind it.

## Fix

Pin the installer to `26.3.2-3`, the release that was last green, and
add `-f` to the curl.

This restores exactly the base environment the last green run had.
Verified by downloading
the tagged aarch64 asset and comparing package build strings against the
last green job log:

| package | last green run | in the 26.3.2-3 asset |
| --- | --- | --- |
| conda | `26.3.2-py313hd81a959_1` | yes |
| conda-libmamba-solver | `26.4.2-pyhd8ed1ab_0` | yes |
| libmambapy | `2.5.0-py313h44afa9f_0` | yes |
| mamba | `2.5.0-hd347465_0` | yes |
| openssl | `3.6.2-h546c87b_0` | yes |
| python | `3.13.13-h11c0449_100_cp313` | yes |

The tagged release publishes the asset under the same unversioned
filename, so only the path
segment of the URL changes.

The `-f` matters now that the URL carries a hand written tag. Without
it, curl treats a 404 as
a successful transfer, writes the error body into the installer file and
exits 0. `chmod +x`
then succeeds and the shell executes the error page, which reports a
syntax error on line 1
and exit 127 instead of a download failure.

## Why not change the conda pin instead

The guard in `setup-binary-builds` is `if [[ "${RUNNER_OS}" != "macOS"
]]`, which keys on the
operating system rather than on whether the base environment actually
needs the pin. aarch64
is now a counter-example, so that guard is worth revisiting. It should
not be revisited in
this change:

- On aarch64 the pin is a downgrade, so skipping it hands the Python
3.15 solve to conda
26.5.3. Python 3.15 resolves from the `conda-forge/label/python_dev` and
`conda-forge/label/python_rc` pre-release labels, and the only note in
this repository about
a newer base conda against those labels is the comment in the macOS
branch saying it returns
400. 20 of torchvision's 81 aarch64 jobs are 3.15 or 3.15t, and all of
them were green with
  the pin in place.
- Keeping the pin only for 3.15 does not work either, because that
reinstalls conda 25.3.0 on
  a Python 3.14 base, which is the failure above.
- Bumping the pin for everyone was already proposed in #8284 and dropped
before merge. The
comment added there records that upgrading the Linux and Windows
containers yields
  `NoChannelsConfiguredError`.

A good follow-up is to measure whether conda 26.5.3 can resolve
`python=3.15.0b4` from those
two labels on linux-aarch64. If it can, the pin can be skipped on
aarch64 and the installer
can float again. If it cannot, the pin stays and the real reason for it
should be written down,
because the comment above it currently gives a reason that does not hold
on aarch64.

## Test plan

What was checked before opening this:

- x86_64, Windows and macOS cannot be affected. The changed step is
gated on
`inputs.architecture == 'aarch64'`, and `setup-binary-builds` is not
touched.
- The `26.3.2-3` release publishes `Miniforge3-Linux-aarch64.sh` under
the same filename this
  step already uses.
- The package build strings in that asset match the last green run, as
listed above.
- curl behaviour with and without `-f` on a 404 was reproduced locally
against a local server.

What still needs CI, and what a reviewer should look for:

- One aarch64 wheel build on this branch. In the log, the installer
should report conda
26.3.2, and `conda install -y conda=25.3.0` should solve. A 3.15 job is
the most valuable
  one to include.
- This has not been run through an aarch64 nightly yet, so that
confirmation is still open.

Authored with assistance from Claude Code (AI assistant).

(cherry picked from commit 2f89f62)
@vercel

vercel Bot commented Aug 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
torchci Ignored Ignored Aug 21, 2026 6:36pm

Request Review

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants