Skip to content

Isolate Poetry validation into its own release-only job - #8075

Merged
atalman merged 1 commit into
pytorch:mainfrom
atalman:main
May 13, 2026
Merged

Isolate Poetry validation into its own release-only job#8075
atalman merged 1 commit into
pytorch:mainfrom
atalman:main

Conversation

@atalman

@atalman atalman commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The Poetry-based validation was sourced inline inside the main manywheel matrix job in validate-linux-binaries.yml, gated by a 5-condition check, and Poetry itself was installed from its dev branch (poetry@main). When Poetry's main branch regressed, the whole binary-validation job went red for the stable-CUDA Python 3.11 combination, masking the actual binary signal. See this failed run for the most recent example.

Three coordinated changes:

1. Split Poetry into a standalone linux-poetry job

In .github/workflows/validate-linux-binaries.yml, the inline source validate_poetry.sh block (with its 5-condition gate) is removed from the linux: matrix job. A new linux-poetry: top-level job is added, modeled on the existing linux-amazon-2023: job:

linux-poetry:
  if: ${{ inputs.channel == 'release' }}
  uses: ./.github/workflows/linux_job_v2.yml
  name: poetry-test
  with:
    runner: "linux.g5.4xlarge.nvidia.gpu"
    repository: "pytorch/pytorch"
    ref: main
    job-name: "poetry-test"
    docker-image: 'pytorch/almalinux-builder:cpu-main'
    docker-build-dir: "skip-docker-build"
    timeout: 60

It runs only on inputs.channel == 'release' and exports a fixed Python 3.11 / stable-CUDA combination, matching what the old inline gate selected.

2. Pin Poetry to a released version

In .github/scripts/validate_poetry.sh:

-curl -sSL https://install.python-poetry.org | python3 - --git https://github.com/python-poetry/poetry.git@main
+POETRY_VERSION="${POETRY_VERSION:-2.4.1}"
+curl -sSL https://install.python-poetry.org | python3 - --version "${POETRY_VERSION}"

Installing from poetry@main has caused recurring CI flakes whenever Poetry's dev branch regresses. POETRY_VERSION is env-overridable; bump deliberately to upgrade.

3. Surface Poetry's actual error

poetry --quiet add ... silenced resolution errors (today's exit 1 had no further output). Replaced with poetry add --no-interaction ... so resolution errors print but Poetry still doesn't prompt.

Test plan

  • bash -n .github/scripts/validate_poetry.sh passes.
  • The new linux-poetry: job mirrors the existing linux-amazon-2023: job structure; both reuse linux_job_v2.yml with the same input shape.
  • Verify via dispatch on the release channel that the new linux-poetry job appears and runs; verify on nightly / test channels that it is skipped.
  • Verify that the next validate-linux-binaries run is no longer red when Poetry's main is broken.

Effect on vllm#40077-style flakes

Before After
Poetry-main breakage → red manywheel-py3_11-cuda13_0 job → looks like a binary regression Poetry-main breakage → red poetry-test job only; binary validation is unaffected
--quiet hides the real error --no-interaction keeps prompts off but lets errors print
Poetry installed from @main Poetry pinned to ${POETRY_VERSION:-2.4.1}
Poetry runs on every matrix entry matching 5 conditions Poetry runs once, on channel == 'release' only

Authored with Claude Code.

The Poetry-based validation was sourced inline inside the main
manywheel matrix job in validate-linux-binaries.yml, gated by a
5-condition check, and Poetry itself was installed from its dev
branch (poetry@main). When Poetry's main branch regressed, the
whole binary-validation job went red for the stable-CUDA Python
3.11 combination, masking the actual binary signal.

Three coordinated changes:

1. Split the Poetry test into a standalone linux-poetry job
   modeled on the existing linux-amazon-2023 job. The new job
   only runs on inputs.channel == 'release' and exports a fixed
   Python 3.11 / stable-CUDA combination, matching what the old
   inline gate selected.

2. Drop the inline `source validate_poetry.sh` block from the
   main matrix job so Poetry ecosystem churn can no longer flake
   binary validation.

3. In validate_poetry.sh, pin Poetry to a released version via
   POETRY_VERSION (default 2.4.1, env-overridable) instead of
   installing from poetry@main, and replace `--quiet` with
   `--no-interaction` so resolution errors are no longer
   silently swallowed.

Authored with Claude Code.
@vercel

vercel Bot commented May 13, 2026

Copy link
Copy Markdown

@atalman is attempting to deploy a commit to the Meta Open Source Team on Vercel.

A member of the Team first needs to authorize it.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 13, 2026

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@atalman
atalman merged commit 737cf7a into pytorch:main May 13, 2026
43 of 44 checks passed
atalman added a commit that referenced this pull request May 13, 2026
…ython (pytorch/vision#9307) (#8077)

## Summary

Fixes the `poetry-test` job's release-channel failure observed in [run
25819051911](https://github.com/pytorch/test-infra/actions/runs/25819051911/job/75855222112).
The root cause is the torchvision-side bug tracked in
[pytorch/vision#9307](pytorch/vision#9307):
torchvision 0.27.0's wheel metadata excludes Python 3.14.1 specifically
(`Requires-Python: !=3.14.1,>=3.10`). Combined with Poetry's strict
resolver, this blocks the smoke test entirely.

## Why this breaks

`validate_poetry.sh` previously called `poetry new test_poetry`, which
writes an **open-ended** `requires-python = ">=X.Y"` (e.g. `">=3.11"`)
into the generated `pyproject.toml`. Poetry's resolver then has to find
a torchvision version that's installable across the project's entire
declared Python range — including Python 3.14.1. Because torchvision
0.27.0 excludes that exact patch
([pytorch/vision#9307](pytorch/vision#9307)),
no version is lockable:

```
The current project's supported Python range (>=3.11) is not compatible
with some of the required packages Python requirement:
  - torchvision requires Python !=3.14.1,>=3.10, so it will not be
    installable for Python 3.14.1

Because no versions of torchvision match >0.27.0,<0.28.0
 and torchvision (0.27.0) requires Python !=3.14.1,>=3.10,
 torchvision is forbidden.
So, because test-poetry depends on torchvision (^0.27.0), version
 solving failed.
```

(This error only became visible after #8075 dropped `--quiet` from the
`poetry add` calls.)

## Fix

Replace `poetry new test_poetry` with `poetry init --no-interaction
--python ">=X.Y,<X.(Y+1)"`, which writes a **narrow** Python range into
the generated `pyproject.toml` directly. For
`MATRIX_PYTHON_VERSION=3.11` the project ends up at `requires-python =
">=3.11,<3.12"`. That range excludes 3.14.1 entirely, so torchvision's
`!=3.14.1` constraint no longer applies and resolution succeeds.

This is also more accurate: the test now reflects the runtime we
actually set up via `conda create python=${MATRIX_PYTHON_VERSION}`,
rather than the open-ended `">=X.Y"` Poetry defaults to.

No workflow-file changes — `linux-poetry:` is already present on `main`
from #8075. The cosmetic display of `linux-poetry:` in this PR's diff is
a GitHub artifact (the merge-base predates #8075's squash merge); the
workflow file content is byte-identical to `pytorch/test-infra:main` and
the merge will be a no-op on it.

## Test plan

- [x] Diagnosed against [run
25819051911](https://github.com/pytorch/test-infra/actions/runs/25819051911/job/75855222112),
the first run that surfaced Poetry's real error after `--quiet` was
removed.
- [ ] CI on this PR's release-channel `poetry-test` job re-runs `poetry
add torch torchvision` cleanly.

## Upstream tracking

- [pytorch/vision#9307](pytorch/vision#9307) —
torchvision excludes Python 3.14.1 in its wheel metadata.

Authored with Claude Code.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants