Skip to content

Commit a30fc53

Browse files
authored
Add release-update-docker-image-pin skill (#8169)
Adds a Claude Code skill that documents and drives pinning the Linux manywheel builder docker images for a PyTorch release branch: derive the .ci/docker tree hash (git rev-parse HEAD:.ci/docker, the same tag binary-docker-build publishes), set DOCKER_IMAGE_PIN in generate_binary_build_matrix.py, regenerate the binary build workflows, and verify only the linux manywheel files change and stay idempotent. The skill takes the release version (e.g. 2.13) as its input and operates on a pytorch/pytorch release branch checkout. This skill was authored with the assistance of Claude Code.
1 parent 74c852e commit a30fc53

1 file changed

Lines changed: 210 additions & 0 deletions

File tree

  • .claude/skills/release-update-docker-image-pin
Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
---
2+
name: release-update-docker-image-pin
3+
description: Pin (or re-pin) the Linux manywheel builder docker images used by the nightly/release binary build workflows in pytorch/pytorch to a fixed .ci/docker build. Takes the release version (e.g. 2.13) as input. Updates DOCKER_IMAGE_PIN in .github/scripts/generate_binary_build_matrix.py and regenerates the workflows. Triggered by mentions of "pin docker image", "update docker pin", "docker image pin", "pin builder images", or "bump the manywheel image pin" for a release branch.
4+
---
5+
6+
# Release: Update Docker Image Pin
7+
8+
Pins the Linux manywheel builder docker images that the binary build workflows
9+
in **pytorch/pytorch** use, so a release branch builds against a fixed,
10+
reproducible toolchain instead of the floating tags that `main` tracks (e.g.
11+
`pytorch/manylinux2_28-builder:cuda12.6`). The pin lives in the workflow
12+
generator, so it survives `regenerate.sh` and the lint "generated files are up
13+
to date" check.
14+
15+
This skill operates on a **pytorch/pytorch** checkout (usually a `release/X.Y`
16+
branch), even though the skill itself lives in test-infra.
17+
18+
## Inputs
19+
20+
| Input | Required | Example | Notes |
21+
|-------|----------|---------|-------|
22+
| **Release version** | yes | `2.13` | The `release/X.Y` being pinned. Drives `RELEASE_VERSION_TAG` when regenerating and is used to sanity-check the checkout. |
23+
| **pytorch/pytorch path** | only if ambiguous | `~/pytorch` | A checkout with the matching `release/X.Y` branch checked out. |
24+
25+
The **release version is the primary argument**. If it was not supplied when the
26+
skill was invoked, ask for it before making any changes -- do not guess. Confirm
27+
it matches the checkout with `cut -d'.' -f1-2 version.txt`.
28+
29+
## When to use this skill
30+
31+
Use when the user asks to:
32+
- Pin / re-pin the manywheel (builder) docker images for a release
33+
- Update or bump `DOCKER_IMAGE_PIN`
34+
- Freeze the docker image the nightly/release binaries build with
35+
- Refresh the docker image pin after the release branch's `.ci/docker` changed
36+
37+
## Background: how the pin maps to a published image
38+
39+
When the builder images are built (`.github/workflows/build-manywheel-images.yml`
40+
and `build-manywheel-images-s390x.yml` via the `binary-docker-build` action),
41+
each image is pushed with several tags, including:
42+
43+
```
44+
docker.io/pytorch/<image>:<prefix>-${CI_FOLDER_SHA}
45+
```
46+
47+
where `CI_FOLDER_SHA="$(git rev-parse HEAD:.ci/docker)"` -- the git tree hash of
48+
the `.ci/docker` directory. This is exactly the value
49+
`test-infra/.github/actions/calculate-docker-image` computes at runtime. Pinning
50+
simply freezes that hash as a literal so the release stops tracking floating
51+
tags. Example published image:
52+
53+
```
54+
pytorch/manylinux2_28_aarch64-builder:cpu-aarch64-<CI_FOLDER_SHA>
55+
```
56+
57+
Only **linux** manywheel builds run inside these containers, so only the linux
58+
images are pinned. Windows and macOS keep the plain tag prefix.
59+
60+
## Target
61+
62+
Repo: **pytorch/pytorch** (the release branch you are pinning).
63+
64+
| Path | Role |
65+
|------|------|
66+
| `.github/scripts/generate_binary_build_matrix.py` | Holds `DOCKER_IMAGE_PIN`; the only file you edit by hand |
67+
| `.github/workflows/generated-linux-binary-manywheel-nightly.yml` | Regenerated output (x86) |
68+
| `.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml` | Regenerated output (aarch64) |
69+
| `.github/workflows/generated-linux-s390x-binary-manywheel-nightly.yml` | Regenerated output (s390x) |
70+
71+
## Instructions
72+
73+
### Step 1: Resolve inputs
74+
75+
Resolve the **Release version** and **pytorch/pytorch path** from the Inputs
76+
table above. If the release version was not provided, ask for it now. Confirm
77+
the checkout is on the matching branch and the version agrees:
78+
79+
```bash
80+
cut -d'.' -f1-2 version.txt # must equal the release version, e.g. 2.13
81+
git rev-parse --abbrev-ref HEAD # should be release/<version>
82+
```
83+
84+
### Step 2: Compute the pin hash
85+
86+
From the root of the pytorch/pytorch checkout, on the release branch:
87+
88+
```bash
89+
git rev-parse HEAD:.ci/docker
90+
```
91+
92+
This 40-char hash is the new `DOCKER_IMAGE_PIN`. It is the same value the
93+
builder images are published under. Do NOT invent or hand-edit it.
94+
95+
If the user instead points at a specific published image tag (e.g. from a known
96+
good nightly on https://hud.pytorch.org), use the suffix from that tag and
97+
verify it equals the `git rev-parse` output above; they should match on a fresh
98+
release cut.
99+
100+
### Step 3 (optional): Verify the image exists
101+
102+
If the user wants confirmation, check that the tag exists on Docker Hub (a human
103+
can open the URL; automated egress may be restricted):
104+
105+
```
106+
https://hub.docker.com/r/pytorch/manylinux2_28_aarch64-builder/tags?name=cpu-aarch64-<HASH>
107+
```
108+
109+
### Step 4: Update or add `DOCKER_IMAGE_PIN`
110+
111+
Open `.github/scripts/generate_binary_build_matrix.py`.
112+
113+
**If the pin block already exists** (re-pinning), just replace the hash:
114+
115+
```python
116+
DOCKER_IMAGE_PIN = "<NEW_40_CHAR_HASH>"
117+
```
118+
119+
**If pinning for the first time on this release branch**, add the block right
120+
after the `WHEEL_CONTAINER_IMAGES` dict, and route the wheel tag prefix through
121+
the helper. Add:
122+
123+
```python
124+
# RELEASE-ONLY: pin the manywheel builder images to a fixed build so the release
125+
# uses a reproducible toolchain instead of main's floating tags. The suffix is
126+
# the .ci/docker tree hash (`git rev-parse HEAD:.ci/docker`), i.e. the same tag
127+
# .github/actions/binary-docker-build publishes. Only linux manywheel builds run
128+
# inside these containers, so only those images are pinned.
129+
DOCKER_IMAGE_PIN = "<NEW_40_CHAR_HASH>"
130+
MANYWHEEL_OSES = ("linux", "linux-aarch64", "linux-s390x")
131+
132+
133+
def wheel_container_image_tag_prefix(arch_version: str, os: str) -> str:
134+
tag_prefix = WHEEL_CONTAINER_IMAGES[arch_version].split(":")[1]
135+
if os in MANYWHEEL_OSES:
136+
return f"{tag_prefix}-{DOCKER_IMAGE_PIN}"
137+
return tag_prefix
138+
```
139+
140+
Then replace BOTH occurrences of the inline tag-prefix computation in
141+
`generate_wheels_matrix` with a call to the helper:
142+
143+
```python
144+
"container_image_tag_prefix": wheel_container_image_tag_prefix(
145+
arch_version, os
146+
),
147+
```
148+
149+
(The original reads `WHEEL_CONTAINER_IMAGES[arch_version].split(":")[1]`.)
150+
151+
### Step 5: Regenerate the workflows
152+
153+
Run the generator in release mode from the repo root:
154+
155+
```bash
156+
RELEASE_VERSION_TAG=<RELEASE_VERSION> python3 .github/scripts/generate_ci_workflows.py
157+
```
158+
159+
(Equivalent: `RELEASE_VERSION_TAG=<ver> ./.github/regenerate.sh`.)
160+
161+
### Step 6: Verify
162+
163+
```bash
164+
# Only the three linux manywheel files should change:
165+
git status --short .github/workflows/generated-*.yml
166+
167+
# Every linux builder image should carry the new hash; none floating:
168+
grep -rhE "image: pytorch/manylinux" .github/workflows/generated-linux-*.yml | grep -v "<HASH>" || echo "none floating (good)"
169+
170+
# Windows/macOS must be unchanged (plain cpu / cuda12.6 prefixes):
171+
grep -nE "docker_image_tag_prefix:" .github/workflows/generated-macos-arm64-binary-wheel-nightly.yml | grep -v '\${{'
172+
```
173+
174+
Then confirm idempotency -- regenerating a second time must produce
175+
byte-identical output (this is what the lint up-to-date check enforces):
176+
177+
```bash
178+
md5sum .github/workflows/generated-linux-*.yml > /tmp/a
179+
RELEASE_VERSION_TAG=<ver> python3 .github/scripts/generate_ci_workflows.py
180+
md5sum .github/workflows/generated-linux-*.yml > /tmp/b
181+
diff /tmp/a /tmp/b && echo "idempotent"
182+
```
183+
184+
### Step 7: Lint and commit
185+
186+
Run `lintrunner -a` on the changed files, then commit. The commit message
187+
should record the hash and that it equals `git rev-parse HEAD:.ci/docker`.
188+
This is a release-only change; do not port it to `main`.
189+
190+
## Common pitfalls
191+
192+
- **Wrong hash**: always derive it with `git rev-parse HEAD:.ci/docker` on the
193+
release branch. A hand-typed or stale hash points at a nonexistent tag and
194+
every linux binary build will fail to pull its container.
195+
- **Pinning windows/macOS**: those builds do not run in these containers. Keep
196+
the pin scoped to `MANYWHEEL_OSES`; a pinned windows/macOS prefix is wrong.
197+
- **Editing the generated YAML directly**: the lint check regenerates and
198+
asserts no diff, so hand edits get reverted. Always change the generator.
199+
- **Forgetting `RELEASE_VERSION_TAG`**: without it the regenerated reusable
200+
workflow refs/version may differ from the committed release files.
201+
202+
## Example usage
203+
204+
```
205+
Pin the manywheel docker images for release/2.13
206+
```
207+
208+
```
209+
Re-pin the builder images, .ci/docker changed on the release branch
210+
```

0 commit comments

Comments
 (0)