Skip to content

Commit 2d317b1

Browse files
authored
[alerts] HUD failing job alerts handles failures that move across shards and a lot of other changes (#7011)
HUD failing job alerts handles failures that move across shards and remove logic that supports features that I don't think are really used effectively Changes * Swap runner to 24.04 which has default python version 3.12 so I can take advantage of 3.11 features (StrEnum) * Upgrade setuptools for 3.12? * Alert can now track failures that jump shards * failure -> pending -> failure is now considered a continuous failure * Remove the flaky stuff (it wasn't doing anything I think) * Remove tracking which sha it started it (didn't show up in the comment, only the issue body -> doesn't get sent in chat -> basically ignored) * Remove requirement that failure captures need to be the same for a failure to be considered continuous (I think this is ok to remove because I don't think we see that many different failures close to each other, and sometimes log classifier is wrong) * Use classes so I can group functions (pros: type checking, cons: annoying to mock, need to transform from API/hud output to classes) * Idk what DISABLED_JOB_NAMES is for so I got rid of the test there * Maybe for filtering out rerun + mem leak? But I don't see any logic for that and it's probably better off in the regex or something Testing: 1044 python tools/torchci/check_alerts.py --dry-run=true --branch=db32b60662b2f2bdcad980127d5dc4b66b02a7e4 1045 python tools/torchci/check_alerts.py --dry-run=true --branch=ba47821f524eee50a214ed39fa2e7765d54aabf4 And check for jobs that I know to be bad on these commits
1 parent 6789d56 commit 2d317b1

4 files changed

Lines changed: 256 additions & 294 deletions

File tree

.github/workflows/check-alerts.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,14 @@ jobs:
3232
JOB_NAME_REGEX: ${{ matrix.job_filter_regex }}
3333
# Don't do actual work on pull request
3434
DRY_RUN: ${{ github.event_name == 'pull_request'}}
35-
runs-on: ubuntu-22.04
35+
runs-on: ubuntu-24.04
3636
permissions:
3737
issues: write
3838
steps:
3939
- name: Checkout
4040
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
4141
- name: Install Dependencies
42-
run: pip3 install requests setuptools==61.2.0
42+
run: pip3 install requests setuptools==80.9.0
4343
- name: Check for alerts and creates issue
4444
run: |
4545
cd tools
@@ -51,14 +51,14 @@ jobs:
5151
update-queue-alert:
5252
env:
5353
DRY_RUN: ${{ github.event_name == 'pull_request' }}
54-
runs-on: ubuntu-22.04
54+
runs-on: ubuntu-24.04
5555
permissions:
5656
issues: write
5757
steps:
5858
- name: Checkout
5959
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
6060
- name: Install Dependencies
61-
run: pip3 install requests setuptools==61.2.0
61+
run: pip3 install requests setuptools==80.9.0
6262
- name: Check for alerts and creates issue
6363
run: |
6464
cd tools

0 commit comments

Comments
 (0)