Skip to content

Commit 267d7da

Browse files
atalmanclaude
andauthored
Add daily nightly wheel-size regression detector (wraps wheel-size-analyzer skill) (#8317)
## Summary Daily detector for **significant increases in pytorch nightly wheel sizes**, as a thin cron wrapper around the **existing `wheel-size-analyzer` skill** (`.claude/skills/wheel-size-analyzer`). The existing `validate-pypi-wheel-binary-size.yml` enforces a **fixed ceiling** (fail if a wheel exceeds N MB). It misses a *relative* regression — a wheel that jumps vs. its recent size while still under the ceiling. This adds day-over-day detection + root-cause. ## Why a wrapper (not new scrapers) The `wheel-size-analyzer` skill already: - reads **real per-artifact sizes** from the GitHub Actions **artifacts API** (`size_in_bytes` on the `linux-binary-manywheel` nightly runs) — a more precise source than scraping the published index; - flags **day-over-day jumps** against a threshold; and - **investigates the cause** by diffing the nightly base SHAs and reading the git log. So this PR does not reinvent any of that — it just runs the skill **on a schedule** and alerts. > Note: neither `workflow_job` nor `workflow_run` stores artifact size in ClickHouse (checked — no size columns; GH job/run webhooks don't carry it). Artifact sizes live in the artifacts API, which the skill uses. A ClickHouse+Grafana-alert version is possible but would require standing up an ingestion table first; this wrapper needs none. ## What's in it - **`.github/workflows/nightly-wheel-size-regression.yml`** — daily cron (`environment: bedrock`, `gha_workflow_claude_code` OIDC). Runs the `wheel-size-analyzer` skill (Claude on Bedrock, read-only) over a short recent window for the configured manywheel variants, and files an alert issue **only** when a variant crosses the day-over-day MB threshold on the most recent day (the skill also fills in the likely cause). - Inputs: `variants`, `threshold_mb` (default 50), `lookback_days` (default 3), `dry_run`. *(Earlier revisions of this PR added a bespoke index-scraper + S3 history + Python delta; those are removed in favor of reusing the skill.)* ## Test plan - Workflow YAML validates. - Recommended: run via `workflow_dispatch` with `dry_run: true` and inspect the produced report / whether `/tmp/alert.md` is created, before enabling the schedule. ## Notes / for review - **Thresholds:** `threshold_mb` is a flat MB delta (matches the skill). A percentage rule might suit the wide size range better (CPU ~175 MB vs ROCm ~4 GB) — easy to switch. - **`prompt:` input** — confirm against the pinned `claude-code-action` version / Treehugger's autonomous invocation. - **Variants** default to the current nightly set; could be derived from `generate_binary_build_matrix.py` if you want zero-maintenance. This PR was authored with the assistance of an AI coding agent. --------- Signed-off-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: Andrey Talman <atalman@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent c4580ef commit 267d7da

1 file changed

Lines changed: 136 additions & 0 deletions

File tree

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
name: Nightly Wheel-Size Regression
2+
3+
# Daily detector for significant increases in pytorch nightly wheel sizes.
4+
# Wraps the existing `wheel-size-analyzer` skill (.claude/skills/wheel-size-analyzer),
5+
# which reads real per-artifact sizes from the GitHub Actions artifacts API
6+
# (size_in_bytes on the linux-binary-manywheel nightly runs), flags day-over-day
7+
# jumps, and can investigate the cause via the nightly base-SHA git log.
8+
#
9+
# Complements validate-pypi-wheel-binary-size.yml (a fixed ceiling): this catches
10+
# *relative* regressions -- a wheel that jumps vs. its recent size even while
11+
# still under the ceiling.
12+
13+
on:
14+
schedule:
15+
# Daily at 16:00 UTC (after nightlies publish + artifacts are up)
16+
- cron: '0 16 * * *'
17+
workflow_dispatch:
18+
inputs:
19+
variants:
20+
description: "comma-separated manywheel variants; blank = derive from generate_binary_build_matrix.py"
21+
type: string
22+
default: ""
23+
threshold_mb:
24+
description: "flag a variant if its day-over-day size delta exceeds this many MB"
25+
type: string
26+
default: "50"
27+
lookback_days:
28+
description: "days of history to analyze (default: the past week)"
29+
type: string
30+
default: "7"
31+
dry_run:
32+
description: "run analysis but do not file an alert issue"
33+
type: boolean
34+
default: false
35+
36+
concurrency:
37+
group: nightly-wheel-size-${{ github.ref }}
38+
cancel-in-progress: true
39+
40+
jobs:
41+
wheel-size:
42+
if: github.repository == 'pytorch/test-infra'
43+
runs-on: ubuntu-latest
44+
timeout-minutes: 30
45+
environment: bedrock
46+
permissions:
47+
contents: read
48+
issues: write
49+
id-token: write
50+
steps:
51+
- uses: actions/checkout@v4
52+
with:
53+
fetch-depth: 1
54+
55+
- name: Configure AWS credentials via OIDC
56+
uses: aws-actions/configure-aws-credentials@v4
57+
with:
58+
role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_claude_code
59+
aws-region: us-east-1
60+
61+
# Derive the CUDA/ROCm variant suffixes from the current build matrix
62+
# (tools/scripts/generate_binary_build_matrix.py) so this stays in sync as
63+
# arches are added/dropped, rather than hardcoding version numbers.
64+
- name: Derive manywheel variants from build matrix
65+
id: variants
66+
run: |
67+
LIST=$(python3 - <<'PY'
68+
import sys
69+
sys.path.insert(0, "tools/scripts")
70+
import generate_binary_build_matrix as gbm
71+
variants = ["cpu", "xpu"]
72+
variants += [f"cuda{a}".replace(".", "_") for a in gbm.CUDA_ARCHES]
73+
variants += [f"rocm{a}".replace(".", "_") for a in gbm.ROCM_ARCHES]
74+
print(",".join(dict.fromkeys(variants)))
75+
PY
76+
)
77+
echo "list=${LIST}" >> "$GITHUB_OUTPUT"
78+
echo "Derived variants: ${LIST}"
79+
80+
# Run the existing wheel-size-analyzer skill over a small recent window and
81+
# only write /tmp/alert.md if a variant crosses the threshold today.
82+
- name: Run wheel-size-analyzer
83+
uses: anthropics/claude-code-action@593d7a5c4e0073569f74772c2b7b64c30ec14707 # v1.0.141
84+
env:
85+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
86+
with:
87+
use_bedrock: "true"
88+
claude_args: "--model global.anthropic.claude-opus-4-8 --allowedTools Read,Write,Bash(gh api:*),Bash(git log:*)"
89+
prompt: |
90+
Use the `wheel-size-analyzer` skill (.claude/skills/wheel-size-analyzer)
91+
to analyze pytorch nightly wheel sizes over the PAST WEEK (the last
92+
${{ inputs.lookback_days || '7' }} days, through today) for these
93+
manywheel variants: ${{ inputs.variants || steps.variants.outputs.list }}.
94+
95+
Write the full report to /tmp/report.md, including the per-day size
96+
table and the significant-jumps table for the week.
97+
98+
Then flag any variant whose MOST RECENT day's size exceeds its
99+
baseline for the week — the median of the earlier days in the window —
100+
by more than ${{ inputs.threshold_mb || '50' }} MB. (Using the weekly
101+
baseline catches both a day-over-day jump and a slower increase across
102+
the week, and is robust to a single missing day.)
103+
104+
If (and only if) at least one variant crosses that threshold:
105+
investigate the cause using the skill's "Investigating size jumps"
106+
procedure (diff the nightly base SHAs across the increase and read the
107+
git log), and write /tmp/alert.md summarizing each flagged variant, its
108+
MB/% delta vs. the weekly baseline, the day the jump happened, and the
109+
most likely cause. Do NOT create /tmp/alert.md if nothing crosses the
110+
threshold.
111+
112+
Read-only: do not modify code, PRs, or issues.
113+
114+
- name: File alert issue
115+
if: ${{ !inputs.dry_run }}
116+
env:
117+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
118+
run: |
119+
if [ -s /tmp/alert.md ]; then
120+
DAY=$(date -u +%F)
121+
{
122+
echo "## Nightly wheel-size increase — ${DAY}"
123+
echo
124+
cat /tmp/alert.md
125+
} > /tmp/issue_body.md
126+
gh issue create --repo pytorch/test-infra \
127+
--title "Nightly wheel-size increase detected (${DAY})" \
128+
--body-file /tmp/issue_body.md \
129+
--label "pytorch-alert"
130+
else
131+
echo "No significant nightly wheel-size increase detected today."
132+
fi
133+
134+
- name: Upload usage metrics
135+
if: always()
136+
uses: pytorch/test-infra/.github/actions/upload-claude-usage@main

0 commit comments

Comments
 (0)