Skip to content

Fix Python 3.15 validation on Linux, Windows and macOS - #8443

Merged
atalman merged 3 commits into
pytorch:mainfrom
atalman:atalman/fix-py315-validation
Aug 5, 2026
Merged

Fix Python 3.15 validation on Linux, Windows and macOS#8443
atalman merged 3 commits into
pytorch:mainfrom
atalman:atalman/fix-py315-validation

Conversation

@atalman

@atalman atalman commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What

Makes Python 3.15 / 3.15t binary validation actually work, on every platform.
#8427 put 3.15 in the default matrix for all OSes, which exposed the uv-based
interpreter setup in validate_binaries.sh — previously Linux-only — to Windows
and macOS. Four bugs, all in that one script:

1. Wrong interpreter pinned (all platforms). import torch segfaults with
every published cp315 wheel under CPython 3.15.0b1, which is what the script
provisioned. The crash is pybind11 3.0.4's gil_scoped_acquire teardown in
python_tracer::init(), reached from torch._C._autograd_init():

torch/autograd/__init__.py:653          if not torch._C._autograd_init():
 └ THPAutograd_initExtension            torch/csrc/autograd/init.cpp:761
    └ python_tracer::init()             torch/csrc/autograd/profiler_python.cpp:1637
       └ pybind11::gil_scoped_acquire::~gil_scoped_acquire()
          └ ::dec_ref() -> PyThreadState_Clear -> SIGSEGV

The same wheel imports and runs fine under 3.15.0b4, so the pin moves to b4.
Verified locally on both interpreters, and with the oldest (dev20260724) and
newest (dev20260805) cp315 nightlies.

2. Windows could not activate the venv. uv lays a venv out as Scripts/, not
bin/ — it says so in the log line right above the failure:

Activate with: conda-env-31013667966\Scripts\activate
+ source conda-env-31013667966/bin/activate
validate_binaries.sh: line 327: .../bin/activate: No such file or directory

3. macOS silently validated the wrong Python. The macos-arm64 workaround
prepends ${CONDA_PREFIX}/bin to PATH. On the uv path no conda env is activated,
so CONDA_PREFIX still pointed at base (python 3.14) and shadowed the venv.
The job did not error — it installed and smoke-tested torch under 3.14, i.e. the
3.15 leg was reporting 3.14 results. The prepend is now skipped on the uv path.

4. Cleanup failed after a green run. ENV_NAME is a conda env name on the
conda path but a venv directory on the uv path, so conda env remove -n fails
with EnvironmentLocationNotFound and, under set -e, fails the job after the
tests passed. The uv path now deactivates and rm -rfs the directory.

5. numpy built from source on Windows. PyPI publishes no cp315/cp315t numpy
wheels, so the smoke-test step's pip install numpy --upgrade falls back to a
source build — slow on Linux/macOS, fatal on Windows:

numpy/_core/src/common/simd/simd.h(4): fatal error C1083:
    Cannot open include file: 'stdalign.h': No such file or directory

download.pytorch.org/whl/nightly already carries preview cp315 and cp315t numpy
wheels for every platform including win_amd64, so 3.15 pulls numpy from there
with --only-binary=:all: to forbid a source fallback. Other Python versions
keep using PyPI unchanged.

The conda path for 3.10–3.14t is untouched throughout.

Test plan

Validated end-to-end with a temporary commit that pointed the workflow checkouts
at this branch and forced PR builds to 3.15 (that commit has been removed;
this PR is now the three real commits, one file).

Final run — every 3.15 leg green:

platform 3.15 legs
linux 7/7 — cpu, cuda 12.6 / 13.0 / 13.2, rocm 7.1 / 7.2, xpu
linux-aarch64 4/4 — cpu, cuda 12.6 / 13.0 / 13.2
win 5/5 — cpu, cuda 12.6 / 13.0 / 13.2, xpu
mac-arm64 green

Also: bash -n clean; shellcheck -S warning reports only the pre-existing
line-181 finding; the 11 matrix unit tests pass.

Follow-ups (not in this PR)

  • None of the four validate-*-binaries.yml workflows pass test-infra-ref, so
    generate_binary_build_matrix.yml defaults it to main and the *_job.yml
    templates to "". Both the matrix generator and validate_binaries.sh are
    therefore always taken from mainno PR touching this script can test
    itself
    without a temporary commit like the one used above. Worth fixing
    properly.
  • PR-limited runs validate a single Python version, so 3.15t got no CI
    coverage
    here. It was verified locally on b4 (GIL genuinely disabled, matmul
    correct, and cp315t numpy wheels exist on the nightly index).
  • The pybind11 gil_scoped_acquire fragility behind the b1 segfault is real and
    still worth reporting upstream, even though b4 avoids it.

Now that 3.15/3.15t are in the default matrix on every OS (pytorch#8427), the
uv-based interpreter setup runs on Windows and macOS for the first time and
breaks in three places:

1. Windows: uv lays a venv out as Scripts/, not bin/, so
   `source ${ENV_NAME}/bin/activate` failed outright with
   "conda-env-<id>/bin/activate: No such file or directory".

2. macOS: the macos-arm64 workaround prepends ${CONDA_PREFIX}/bin to PATH.
   On the uv path no conda env was activated, so CONDA_PREFIX still pointed at
   base (python 3.14) and shadowed the venv's python3/pip3 -- the run installed
   and smoke-tested torch under conda base 3.14 instead of 3.15.

3. Cleanup: ENV_NAME is a conda env name on the conda path but a venv directory
   on the uv path, so `conda env remove -n` fails with
   EnvironmentLocationNotFound and, under set -e, fails the job after the tests
   have already passed.

The conda path for 3.10-3.14t is unchanged.
@vercel

vercel Bot commented Aug 5, 2026

Copy link
Copy Markdown

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 5, 2026
`import torch` segfaults with every published cp315 wheel under the 3.15.0b1
build. The crash is pybind11 3.0.4's gil_scoped_acquire teardown in
python_tracer::init(), reached from torch._C._autograd_init():

    THPAutograd_initExtension -> python_tracer::init()
      -> ~gil_scoped_acquire() -> dec_ref() -> PyThreadState_Clear -> SIGSEGV

The same wheel imports and runs cleanly under 3.15.0b4, so bump the pin. This
is what has been failing every Linux and aarch64 3.15 leg.
@atalman atalman changed the title Fix Python 3.15 validation on Windows and macOS Fix Python 3.15 validation on Linux, Windows and macOS Aug 5, 2026
@atalman
atalman force-pushed the atalman/fix-py315-validation branch from 3ef6a65 to 251ff5b Compare August 5, 2026 15:48
…ource

PyPI publishes no cp315/cp315t numpy wheels, so `pip install numpy` in the
smoke-test step falls back to a source build. That is merely slow on Linux and
macOS, but a hard failure on Windows:

    numpy/_core/src/common/simd/simd.h(4): fatal error C1083:
        Cannot open include file: 'stdalign.h': No such file or directory
    error: metadata-generation-failed

download.pytorch.org/whl/nightly carries preview cp315 and cp315t numpy wheels
for every platform (manylinux x86_64/aarch64, macosx arm64/x86_64, win_amd64),
so pull numpy from there for 3.15 and forbid a source fallback with
--only-binary=:all:. Other python versions keep using PyPI unchanged.
@atalman
atalman force-pushed the atalman/fix-py315-validation branch from 264be37 to 11056ed Compare August 5, 2026 16:36

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@atalman
atalman merged commit ee58c63 into pytorch:main Aug 5, 2026
33 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants