Skip to content

Commit ca0ed61

Browse files
atalmanatalman
authored
Update inductor expected accuracy in pytorch/pytorch (#7627)
Workflow to Create a PR to Update inductor expected accuracy in pytorch/pytorch 1. Triggers: - Scheduled: Runs weekly on Sundays at 3:00 AM UTC - Manual: Can be triggered manually via workflow_dispatch with a specific PyTorch commit SHA 2. What it does: - Checks out the pytorch/pytorch repository - Sets up Python 3.10 and installs dependencies (pandas, requests) - Runs the update_expected.py script with the specified SHA - Checks if there are any changes to the CSV files - If changes exist, creates a new branch and opens a PR from pytorchbot 3. PR Details: - Branch name includes timestamp for uniqueness - Commits are signed by pytorchbot - PR includes a detailed description explaining what changed and why - Includes a test plan checklist for reviewers
1 parent 087c9c6 commit ca0ed61

File tree

1 file changed

+131
-0
lines changed

1 file changed

+131
-0
lines changed

‎.github/workflows/update-inductor-expected-accuracy.yml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
name: Update Inductor Expected Accuracy
2+
3+
on:
4+
# Run weekly on Sunday at 3:00 AM UTC
5+
schedule:
6+
- cron: "0 3 * * 0"
7+
# Allow manual triggering with SHA parameter
8+
workflow_dispatch:
9+
inputs:
10+
pytorch_sha:
11+
description: 'PyTorch commit SHA to use for updating expected accuracy'
12+
required: true
13+
type: string
14+
15+
jobs:
16+
update-expected-accuracy:
17+
runs-on: ubuntu-latest
18+
environment: trigger-nightly
19+
steps:
20+
- name: Checkout PyTorch repository
21+
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
22+
with:
23+
repository: pytorch/pytorch
24+
token: ${{ secrets.GH_PYTORCHBOT_TOKEN }}
25+
fetch-depth: 0
26+
27+
- name: Set up Python 3.10
28+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
29+
with:
30+
python-version: '3.10'
31+
32+
- name: Install dependencies
33+
run: |
34+
python -m pip install --upgrade pip
35+
python -m pip install pandas==2.3.3 requests==2.32.5
36+
37+
- name: Determine PyTorch SHA
38+
id: determine-sha
39+
run: |
40+
# shellcheck disable=SC2086
41+
if [ -n "${{ github.event.inputs.pytorch_sha }}" ]; then
42+
SHA=${{ github.event.inputs.pytorch_sha }}
43+
else
44+
# Use the latest commit SHA from main branch if not manually specified
45+
SHA=$(git rev-parse HEAD)
46+
fi
47+
# shellcheck disable=SC2086
48+
echo "sha=${SHA}" >> $GITHUB_OUTPUT
49+
echo "Using PyTorch SHA: ${SHA}"
50+
51+
- name: Run update_expected.py
52+
env:
53+
CH_KEY_ID: ${{ secrets.CH_KEY_ID }}
54+
CH_KEY_SECRET: ${{ secrets.CH_KEY_SECRET }}
55+
run: |
56+
python benchmarks/dynamo/ci_expected_accuracy/update_expected.py ${{ steps.determine-sha.outputs.sha }}
57+
58+
- name: Check for changes
59+
id: check-changes
60+
run: |
61+
# shellcheck disable=SC2086
62+
if git diff --quiet benchmarks/dynamo/ci_expected_accuracy/; then
63+
echo "has_changes=false" >> $GITHUB_OUTPUT
64+
echo "No changes detected"
65+
else
66+
echo "has_changes=true" >> $GITHUB_OUTPUT
67+
echo "Changes detected"
68+
fi
69+
70+
- name: Create Pull Request
71+
if: steps.check-changes.outputs.has_changes == 'true'
72+
env:
73+
GH_TOKEN: ${{ secrets.GH_PYTORCHBOT_TOKEN }}
74+
run: |
75+
# Configure git
76+
git config user.name "pytorchbot"
77+
git config user.email "[email protected]"
78+
79+
# Create a new branch
80+
BRANCH_NAME="update-inductor-expected-accuracy-$(date +%Y%m%d-%H%M%S)"
81+
git checkout -b "${BRANCH_NAME}"
82+
83+
# Add and commit changes
84+
git add benchmarks/dynamo/ci_expected_accuracy/
85+
git commit -m "$(cat <<'EOF'
86+
Update inductor expected accuracy files
87+
88+
This PR updates the expected accuracy CSV files for inductor benchmarks
89+
based on CI results from commit ${{ steps.determine-sha.outputs.sha }}.
90+
91+
These files are used as reference points for dynamo/inductor CI to detect
92+
regressions in graph breaks and accuracy.
93+
EOF
94+
)"
95+
96+
# Push the branch
97+
git push -u origin "${BRANCH_NAME}"
98+
99+
# Create pull request
100+
gh pr create \
101+
--title "Update inductor expected accuracy files" \
102+
--body "$(cat <<'EOF'
103+
## Summary
104+
105+
This PR updates the expected accuracy CSV files for inductor benchmarks based on CI results from PyTorch commit ${{ steps.determine-sha.outputs.sha }}.
106+
107+
These files serve as reference points for dynamo/inductor CI to track:
108+
- Graph breaks
109+
- Model accuracy
110+
111+
## Changes
112+
113+
- Updated CUDA expected accuracy files in `benchmarks/dynamo/ci_expected_accuracy/`
114+
- Updated ROCm expected accuracy files in `benchmarks/dynamo/ci_expected_accuracy/rocm/`
115+
116+
## Test Plan
117+
118+
- [ ] Verify that the CI jobs pass with the updated expected accuracy files
119+
- [ ] Review the diff to ensure changes are reasonable and expected
120+
- [ ] Check that no unexpected regressions are being marked as "expected"
121+
122+
cc: @pytorch/pytorch-dev-infra
123+
EOF
124+
)" \
125+
--base main \
126+
--head "${BRANCH_NAME}"
127+
128+
- name: No changes to commit
129+
if: steps.check-changes.outputs.has_changes == 'false'
130+
run: |
131+
echo "No changes detected in expected accuracy files. Skipping PR creation."

0 commit comments

Comments
 (0)