Commit 2d317b1
authored
[alerts] HUD failing job alerts handles failures that move across shards and a lot of other changes (#7011)
HUD failing job alerts handles failures that move across shards and
remove logic that supports features that I don't think are really used
effectively
Changes
* Swap runner to 24.04 which has default python version 3.12 so I can
take advantage of 3.11 features (StrEnum)
* Upgrade setuptools for 3.12?
* Alert can now track failures that jump shards
* failure -> pending -> failure is now considered a continuous failure
* Remove the flaky stuff (it wasn't doing anything I think)
* Remove tracking which sha it started it (didn't show up in the
comment, only the issue body -> doesn't get sent in chat -> basically
ignored)
* Remove requirement that failure captures need to be the same for a
failure to be considered continuous (I think this is ok to remove
because I don't think we see that many different failures close to each
other, and sometimes log classifier is wrong)
* Use classes so I can group functions (pros: type checking, cons:
annoying to mock, need to transform from API/hud output to classes)
* Idk what DISABLED_JOB_NAMES is for so I got rid of the test there
* Maybe for filtering out rerun + mem leak? But I don't see any logic
for that and it's probably better off in the regex or something
Testing:
1044 python tools/torchci/check_alerts.py --dry-run=true
--branch=db32b60662b2f2bdcad980127d5dc4b66b02a7e4
1045 python tools/torchci/check_alerts.py --dry-run=true
--branch=ba47821f524eee50a214ed39fa2e7765d54aabf4
And check for jobs that I know to be bad on these commits1 parent 6789d56 commit 2d317b1
4 files changed
Lines changed: 256 additions & 294 deletions
File tree
- .github/workflows
- tools/torchci
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
| 61 | + | |
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| |||
0 commit comments