Skip to content

Commit 592772d

Browse files
ezyangezyang
authored
Log classifier grouping and coverage improvements (#8404)
- **log-classifier: fix over-broad grouping + close no-match gaps** - **log-classifier: link fixtures to source jobs + fixture-aware diff tool** - **log-classifier: document git difffix for whole-branch fixture review** - **log-classifier: surface ninja FAILED: edge over generic build-stopped** - **log-classifier: dedupe two noisy catch-all capture groups**
1 parent 36a455c commit 592772d

33 files changed

Lines changed: 1515 additions & 36 deletions

‎aws/lambda/log-classifier/CLAUDE.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,27 @@ with the expected verdict recorded **in-band**:
1212
- `#=MATCH=# ` prefixes the line the classifier surfaces; each captured span is
1313
wrapped in `‹ ›` (so the group key is visible in context).
1414
- A fixture with no `#=MATCH=#` line records that nothing classifies.
15+
- An optional `#=SOURCE=#` line at the top links to the source job (metadata:
16+
never classified, preserved across re-blessing; written by `pull_fixture.py`).
1517

1618
The marker snapshots **current** behavior. Full format: `fixtures/classify/FIXTURES.md`.
1719

20+
## Reviewing fixtures (`git showfix`)
21+
22+
New fixtures are all-additions, so `git show` paints every log line green and
23+
buries the `#=MATCH=#` line. `tools/color_fixture_diff.py` recolors a
24+
`git show`/`git diff` stream: it dims the added log body, highlights only the
25+
match line (and its `‹ ›` captures) and the `#=SOURCE=#` link, and leaves
26+
non-fixture files (ruleset.toml, log.rs) in normal green. Wire up two aliases —
27+
`showfix` for a single commit, `difffix` for a range/working tree (use
28+
`git difffix main...HEAD` to review a whole branch; `git show` would list each
29+
commit separately, so a fixture touched by two commits shows up twice):
30+
31+
```
32+
git config alias.showfix '!f() { root="$(git rev-parse --show-toplevel)"; cd "${GIT_PREFIX:-.}" && git show --color=never "$@" | "$root/aws/lambda/log-classifier/tools/color_fixture_diff.py" | less -R; }; f'
33+
git config alias.difffix '!f() { root="$(git rev-parse --show-toplevel)"; cd "${GIT_PREFIX:-.}" && git diff --color=never "$@" | "$root/aws/lambda/log-classifier/tools/color_fixture_diff.py" | less -R; }; f'
34+
```
35+
1836
- Run: `cargo test --test classify`
1937
- Re-bless after a ruleset/engine change (verify the diff before committing!):
2038
`UPDATE_FIXTURES=1 cargo test --test classify`

‎aws/lambda/log-classifier/fixtures/classify/FIXTURES.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,14 @@ The expectation lives **in-band** in each fixture (see `tests/classify.rs`):
1010
the line the classifier currently lands on is prefixed with `#=MATCH=# ` and each
1111
captured span is wrapped in `‹ ›` (so the group key is visible in context); a
1212
fixture with no `#=MATCH=#` line records that nothing classifies. The marker
13-
records the **current** verdict — which for the bucket-(A) cases below is
14-
deliberately wrong (the *ideal* answer is noted in the table and per-fixture
15-
notes), documenting the bug rather than fixing it. Re-bless after a ruleset change with
13+
records the **current** verdict. Re-bless after a ruleset change with
1614
`UPDATE_FIXTURES=1 cargo test --test classify`.
1715

16+
An optional `#=SOURCE=#` line at the very top links back to the originating job
17+
(so a reviewer can open the live log during CR — these S3/GHA logs eventually
18+
expire). It is metadata: never fed to the classifier and preserved verbatim
19+
across re-blessing. `pull_fixture.py` writes it automatically.
20+
1821
Note on the classifier's ignore-list: `src/log.rs` strips a few generic noise
1922
lines *before* classification — notably `##[error]Process completed with exit
2023
code N` and `##[error]Executing the custom container implementation failed`.
@@ -42,11 +45,9 @@ trimmed off, re-run with a larger `--context`, or pin the window with
4245
the numbered log to help you pick). `--no-bless` writes the window offline
4346
(anchoring on the last `##[error]` / exit-code line instead of the classifier).
4447

45-
After generating: rename to something descriptive and add a row to the table
46-
above (note the ideal answer there if it's a known misclassification).
47-
48-
TODO: As of right now, some of these fixtures are arguably misclassified! We
49-
will be working on improving them soon.
48+
After generating: rename to something descriptive. If the classifier lands on
49+
the wrong line (or nothing) for a real failure, prefer fixing the ruleset so the
50+
fixture blesses to the correct line, rather than checking in a wrong marker.
5051

5152
## Verification
5253

‎aws/lambda/log-classifier/fixtures/classify/action_archive_download_failure.txt‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#=SOURCE=# https://github.com/pytorch/pytorch/actions/runs/30533012812/job/90845795592
2+
2026-07-30T10:33:37.0347135Z Current runner version: '2.336.0'
3+
2026-07-30T10:33:37.0352906Z Runner name: 'gnr816793.jf.intel.com_b60_card_7'
4+
2026-07-30T10:33:37.0353675Z Runner group name: 'linux.idc.xpu.group'
5+
2026-07-30T10:33:37.0354739Z Machine name: 'gnr816793'
6+
2026-07-30T10:33:37.0356799Z ##[group]GITHUB_TOKEN Permissions
7+
2026-07-30T10:33:37.0359288Z Contents: read
8+
2026-07-30T10:33:37.0359718Z Metadata: read
9+
2026-07-30T10:33:37.0360152Z ##[endgroup]
10+
2026-07-30T10:33:37.0363158Z Secret source: Actions
11+
2026-07-30T10:33:37.0364371Z Prepare workflow directory
12+
2026-07-30T10:33:38.1304832Z Prepare all required actions
13+
2026-07-30T10:33:38.1474530Z Getting action download info
14+
2026-07-30T10:33:38.6176600Z Download action repository 'pytorch/pytorch@main' (SHA:050ce9e1226b95d6b5938bdd5e245491f7219fd8)
15+
2026-07-30T10:35:18.6494363Z ##[warning]Failed to download action 'https://codeload.github.com/pytorch/pytorch/tar.gz/050ce9e1226b95d6b5938bdd5e245491f7219fd8'. Error: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
16+
2026-07-30T10:35:18.6503557Z ##[warning]Back off 23.854 seconds before retry.
17+
2026-07-30T10:37:22.5234937Z ##[warning]Failed to download action 'https://codeload.github.com/pytorch/pytorch/tar.gz/050ce9e1226b95d6b5938bdd5e245491f7219fd8'. Error: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
18+
2026-07-30T10:37:22.5238024Z ##[warning]Back off 18.171 seconds before retry.
19+
2026-07-30T10:39:20.7109104Z ##[error]Action 'https://codeload.github.com/pytorch/pytorch/tar.gz/050ce9e1226b95d6b5938bdd5e245491f7219fd8' download has timed out. Error: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
20+
#=MATCH=# 2026-07-30T10:39:20.7140484Z ##[error]‹Failed to download archive› 'https://codeload.github.com/pytorch/pytorch/tar.gz/050ce9e1226b95d6b5938bdd5e245491f7219fd8' after 3 attempts.

‎aws/lambda/log-classifier/fixtures/classify/artifact_not_found.txt‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
#=SOURCE=# https://github.com/pytorch/pytorch/actions/runs/30387633773/job/90739196341
2+
2026-07-29T23:48:08.0720611Z Downloading single artifact
3+
#=MATCH=# 2026-07-29T23:48:08.2790307Z ##[error]‹Unable to download artifact(s): Artifact not found› for name: manywheel-py3_10-cuda13_0
4+
Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact.
5+
For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md
6+
2026-07-29T23:48:08.5576871Z ##[error][OSDC] Step script exited with code 1. This is a script/workflow error, not an infrastructure issue. Check the step logs above for the actual failure.
7+
2026-07-29T23:48:08.5604057Z ##[error]Process completed with exit code 1.
8+
2026-07-29T23:48:08.5705723Z ##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
9+
2026-07-29T23:48:08.5821898Z Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
10+
2026-07-29T23:48:08.5823480Z ##[group]Run actions/upload-artifact@v4.4.0
11+
2026-07-29T23:48:08.5823814Z with:
12+
2026-07-29T23:48:08.5824092Z name: libtorch-cuda13_0-shared-with-deps-release
13+
2026-07-29T23:48:08.5824450Z retention-days: 14
14+
2026-07-29T23:48:08.5824739Z if-no-files-found: error
15+
2026-07-29T23:48:08.5825061Z path: /home/runner/_work/_temp/libtorch_output/
16+
2026-07-29T23:48:08.5825418Z compression-level: 6
17+
2026-07-29T23:48:08.5825735Z overwrite: false
18+
2026-07-29T23:48:08.5825995Z include-hidden-files: false
19+
2026-07-29T23:48:08.5826272Z env:
20+
2026-07-29T23:48:08.5826612Z ALPINE_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/tool/alpine
21+
2026-07-29T23:48:08.5827096Z AWS_DEFAULT_REGION: us-east-1
22+
2026-07-29T23:48:08.5827392Z BINARY_ENV_FILE: /tmp/env
23+
2026-07-29T23:48:08.5827697Z BUILD_ENVIRONMENT: linux-binary-manywheel
24+
2026-07-29T23:48:08.5830897Z GITHUB_TOKEN: ***
25+
2026-07-29T23:48:08.5831154Z PR_NUMBER:
26+
2026-07-29T23:48:08.5831410Z PYTORCH_FINAL_PACKAGE_DIR: /artifacts
27+
2026-07-29T23:48:08.5831736Z PYTORCH_ROOT: /pytorch
28+
2026-07-29T23:48:08.5832039Z SHA1: 7a0bb5187068d69dac9688c8724c8bef72f0971a
29+
2026-07-29T23:48:08.5832378Z SKIP_ALL_TESTS: 0
30+
2026-07-29T23:48:08.5832626Z DESIRED_CUDA: cu130
31+
2026-07-29T23:48:08.5832899Z LIBTORCH_VARIANT: shared-with-deps
32+
2026-07-29T23:48:08.5833498Z ##[endgroup]
33+
2026-07-29T23:48:08.6252995Z ##[group]Run '/home/runner/hook-extensions/wrapper.js'
34+
2026-07-29T23:48:08.6254701Z shell: /home/runner/externals/node20/bin/node {0}
35+
2026-07-29T23:48:08.6255079Z ##[endgroup]
36+
2026-07-29T23:48:10.0842737Z (node:996) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
37+
2026-07-29T23:48:10.0843942Z (Use `node --trace-deprecation ...` to show where the warning was created)
38+
2026-07-29T23:48:10.0844514Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
39+
2026-07-29T23:48:10.0845007Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
40+
2026-07-29T23:48:10.0845497Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
41+
2026-07-29T23:48:10.0845982Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
42+
2026-07-29T23:48:10.0849640Z ##[error]No files were found with the provided path: /__w/_temp/libtorch_output/. No artifacts will be uploaded.
43+
2026-07-29T23:48:10.3850220Z ##[error][OSDC] Step script exited with code 1. This is a script/workflow error, not an infrastructure issue. Check the step logs above for the actual failure.
44+
2026-07-29T23:48:10.3874614Z ##[error]Process completed with exit code 1.
45+
2026-07-29T23:48:10.3894351Z ##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
46+
2026-07-29T23:48:10.4071002Z Node 20 is being deprecated. This workflow is running with Node 24 by default. If you need to temporarily use Node 20, you can set the ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true environment variable. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/
47+
2026-07-29T23:48:10.4072514Z Post job cleanup.
48+
2026-07-29T23:48:10.4096763Z ##[group]Run '/home/runner/hook-extensions/wrapper.js'
49+
2026-07-29T23:48:10.4098131Z shell: /home/runner/externals/node20/bin/node {0}
50+
2026-07-29T23:48:10.4098482Z ##[endgroup]
51+
2026-07-29T23:48:11.7482216Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
52+
2026-07-29T23:48:11.7482832Z GITHUB_EVENT_PATH /github/workflow/event.json does not exist
53+
2026-07-29T23:48:11.7483605Z [command]/usr/bin/git version
54+
2026-07-29T23:48:11.7483902Z git version 2.47.3
55+
2026-07-29T23:48:11.7486703Z Copying '/root/.gitconfig' to '/__w/_temp/d329ee26-40ae-457a-b55c-7dbe7a3d8843/.gitconfig'
56+
2026-07-29T23:48:11.7487599Z Temporarily overriding HOME='/__w/_temp/d329ee26-40ae-457a-b55c-7dbe7a3d8843' before making global git config changes
57+
2026-07-29T23:48:11.7488410Z Adding repository directory to the temporary git global config as a safe directory
58+
2026-07-29T23:48:11.7489078Z [command]/usr/bin/git config --global --add safe.directory /__w/pytorch/pytorch
59+
2026-07-29T23:48:11.7490112Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
60+
2026-07-29T23:48:11.7491494Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
61+
2026-07-29T23:48:11.7492970Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
62+
2026-07-29T23:48:11.7493650Z http.https://github.com/.extraheader
63+
2026-07-29T23:48:11.7494560Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
64+
2026-07-29T23:48:11.7496174Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
65+
2026-07-29T23:48:12.0235258Z ##[group]Run '/home/runner/hook-extensions/wrapper.js'
66+
2026-07-29T23:48:12.0236883Z shell: /home/runner/externals/node20/bin/node {0}
67+
2026-07-29T23:48:12.0237239Z ##[endgroup]
68+
2026-07-29T23:48:12.8889015Z Cleaning up orphan processes
69+
2026-07-29T23:48:12.8948504Z ##[warning]Node.js 20 is deprecated. The following actions target Node.js 20 but are being forced to run on Node.js 24: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683, actions/download-artifact@v4.1.7, actions/upload-artifact@v4.4.0. For more information see: https://github.blog/changelog/2025-09-19-deprecation-of-node-20-on-github-actions-runners/

0 commit comments

Comments
 (0)