Skip to content

Commit cc1468d

Browse files
ezyangezyang
authored
log-classifier: marker-based regression fixture harness (#8394)
**Impact:** HUD / Dr.CI (test-only; no classifier behavior change) **Risk:** low Human: I was monkeying around with Dr. CI classifiers and I realized that I had no idea if I was silently causing regressions. This PR is a PoC for a more robust test suite based on real log snippets and then classifier'ing them. I iterated with Claude on it and I quite like this new result. Please take a look and see if you like. If we like we can build out way more samples now. Rest is agent description. ## What A marker-based regression-test harness for the log classifier, plus an initial set of 5 real-world fixtures. **No ruleset / classifier rule changes** — this only adds tests. - **`tests/classify.rs`** runs the production ruleset over each `fixtures/classify/*.txt` (real, verbatim CI-log fragments, timestamps + ANSI intact, trimmed to keep the surrounding *confusers*) and asserts which line the classifier surfaces and what it captures. The expectation lives in-band in the fixture: - `#=MATCH=# ` prefixes the line the classifier lands on, and each captured span is wrapped in `‹ ›` — so both the surfaced line and its group key are visible in context. - `#=NO-MATCH=#` records that nothing classifies; `#=WANT=#` notes document the ideal answer for known misclassifications. - Sentinels/delimiters are stripped before the classifier sees the log. Re-bless after a ruleset change with `UPDATE_FIXTURES=1 cargo test --test classify`, which places the marker on the line the classifier now lands on and draws `‹ ›` around its live captures (offsets mapped onto the raw line via a new shared `log::strip_line`, so timestamps/ANSI don't misalign them). ## Why The classifier picks the single line that explains a job's failure. This locks that behavior against *real* logs — with the distractors that make the priority arbitration non-trivial — instead of hand-authored strings. Putting the capture delimiters in-band also makes grouping quality reviewable: e.g. `infra_osdc_pod_failure` visibly wraps a random pod name, showing that rule groups badly across runs. ## Notes - Two fixtures are intentional, documented misclassifications left **unfixed** (a to-do list for follow-up rule work): `pr_sanity_exit_code_only` (NO MATCH) and `distributed_test_wrapper` (generic run_test.py wrapper wins over the real "test class should extend TestCase" cause). - Adds `log::strip_line` (shared by `Log::new`); no dependency changes. - `cargo test` green (lib + bin + `classify`). --------- Signed-off-by: Edward Yang <ezyang@meta.com>
1 parent ca5f51e commit cc1468d

11 files changed

Lines changed: 1567 additions & 6 deletions

File tree

‎aws/lambda/log-classifier/CLAUDE.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# log-classifier — notes for agents
2+
3+
Rust AWS Lambda that classifies pytorch/pytorch CI logs. Rules live in
4+
`ruleset.toml`; the engine is in `src/` (`log.rs` preprocesses, `engine.rs`
5+
arbitrates by priority). See `README.md` for deployment.
6+
7+
## Regression fixtures (`tests/classify.rs`)
8+
9+
`fixtures/classify/*.txt` are real CI-log fragments (timestamps + ANSI intact)
10+
with the expected verdict recorded **in-band**:
11+
12+
- `#=MATCH=# ` prefixes the line the classifier surfaces; each captured span is
13+
wrapped in `‹ ›` (so the group key is visible in context).
14+
- A fixture with no `#=MATCH=#` line records that nothing classifies.
15+
16+
The marker snapshots **current** behavior. Full format: `fixtures/classify/FIXTURES.md`.
17+
18+
- Run: `cargo test --test classify`
19+
- Re-bless after a ruleset/engine change (verify the diff before committing!):
20+
`UPDATE_FIXTURES=1 cargo test --test classify`
21+
22+
## Adding a fixture from a failing job
23+
24+
Prefer `./pull_fixture.py` over hand-collecting a log. Pass anything carrying the
25+
job id — a bare id, a GitHub Actions job URL (`.../job/<id>`), or a raw-log URL:
26+
27+
```
28+
./pull_fixture.py <job-id | job-URL | log-URL> --name <case>
29+
```
30+
31+
It downloads the raw log from public S3, runs *this crate's* classifier to find
32+
the line it surfaces, trims to that line ± `--context` (default 60), writes
33+
`fixtures/classify/<case>.txt`, blesses the markers, and prints where the
34+
classifier landed.
35+
36+
Then: **confirm the marked line is the real failure**. If the real cause was
37+
trimmed off, re-run with a larger `--context`, or pin the window with
38+
`--grep <regex>` / `--lines <A-B>` (1-based raw line numbers; `--stdout` prints
39+
the numbered log to help pick a range). `--no-bless` writes offline, anchoring on
40+
the last `##[error]` / exit-code line instead of the classifier.
41+
42+
Finish by adding a row in `fixtures/classify/FIXTURES.md` (note the ideal answer
43+
there if it's a known misclassification).

‎aws/lambda/log-classifier/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,13 @@ You can edit `fixtures/request.json` to test different inputs.
3737
**Note that this will write to S3!** You can pass a different value for
3838
`ShouldWriteS3` if you don't want to do that.
3939

40+
## Classification regression fixtures
41+
42+
`fixtures/classify/*.txt` snapshot the classifier's verdict on real CI logs, so a
43+
ruleset or engine change that moves a match fails `cargo test`. See
44+
[`fixtures/classify/FIXTURES.md`](fixtures/classify/FIXTURES.md) for the marker
45+
format and how to add a fixture from a failing job with `pull_fixture.py`.
46+
4047
## Lambda deployment
4148

4249
The lambda is deployed from main (see:

‎aws/lambda/log-classifier/fixtures/classify/FIXTURES.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Log-classifier regression fixtures
2+
3+
Real, verbatim CI-log fragments (GitHub-Actions timestamps + ANSI intact) from
4+
failing `pytorch/pytorch` jobs, trimmed to a window that keeps the surrounding
5+
*confusers* (generic `run_test.py` summaries, exit-code lines, teardown
6+
tracebacks, OSDC/runner spew). Each fixture asserts "given this noisy log, does
7+
the *correct* line win?"
8+
9+
The expectation lives **in-band** in each fixture (see `tests/classify.rs`):
10+
the line the classifier currently lands on is prefixed with `#=MATCH=# ` and each
11+
captured span is wrapped in `‹ ›` (so the group key is visible in context); a
12+
fixture with no `#=MATCH=#` line records that nothing classifies. The marker
13+
records the **current** verdict — which for the bucket-(A) cases below is
14+
deliberately wrong (the *ideal* answer is noted in the table and per-fixture
15+
notes), documenting the bug rather than fixing it. Re-bless after a ruleset change with
16+
`UPDATE_FIXTURES=1 cargo test --test classify`.
17+
18+
Note on the classifier's ignore-list: `src/log.rs` strips a few generic noise
19+
lines *before* classification — notably `##[error]Process completed with exit
20+
code N` and `##[error]Executing the custom container implementation failed`.
21+
Two fixtures below hinge on this behavior.
22+
23+
Raw-log URLs (public, gzip-stored — pipe through `gunzip`):
24+
`https://ossci-raw-job-status.s3.amazonaws.com/log/<jobId>`
25+
26+
## Adding a fixture
27+
28+
Use `../../pull_fixture.py` (in the crate root) to turn a failing job into a
29+
fixture. Pass anything that carries the job id — a bare id, a GitHub Actions job
30+
URL, or a raw-log URL:
31+
32+
```
33+
./pull_fixture.py https://github.com/pytorch/pytorch/actions/runs/<run>/job/<jobId> --name my_case
34+
```
35+
36+
It downloads the raw log, runs *this crate's* classifier to find the line it
37+
surfaces, trims to that line ± `--context` (default 60), writes
38+
`fixtures/classify/my_case.txt`, and blesses the markers — then prints where the
39+
classifier landed. Confirm that line is the real failure; if the real cause got
40+
trimmed off, re-run with a larger `--context`, or pin the window with
41+
`--grep <regex>` / `--lines <A-B>` (1-based raw line numbers; `--stdout` prints
42+
the numbered log to help you pick). `--no-bless` writes the window offline
43+
(anchoring on the last `##[error]` / exit-code line instead of the classifier).
44+
45+
After generating: rename to something descriptive and add a row to the table
46+
above (note the ideal answer there if it's a known misclassification).
47+
48+
TODO: As of right now, some of these fixtures are arguably misclassified! We
49+
will be working on improving them soon.
50+
51+
## Verification
52+
53+
`cargo test --test classify` passes (each fixture's `#=MATCH=#` marker — or its
54+
absence — agrees with the live classifier).

‎aws/lambda/log-classifier/fixtures/classify/distributed_test_wrapper.txt‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
2026-07-24T18:47:00.4262407Z Running distributed/test_c10d_torchcomms 1/1 ... [2026-07-24 18:47:00.269144][5722.385406931]
2+
2026-07-24T18:47:00.4262902Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
3+
2026-07-24T18:47:00.4264023Z Executing ['/opt/python-3.10-venv/bin/python', '-bb', 'distributed/test_c10d_torchcomms.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=0', '--import-slow-tests', '--import-disabled-tests'] ... [2026-07-24 18:47:00.269458]
4+
2026-07-24T18:47:03.6568011Z
5+
2026-07-24T18:47:03.6568821Z PRINTING LOG FILE of distributed/test_c10d_torchcomms 1/1 (test/test-reports/distributed.test_c10d_torchcomms_1.1_57d5e8b26f6005c4_.log)
6+
2026-07-24T18:47:03.6570425Z TestC10dTorchCommsBackendConfig - failed. This test class should extend from torch.testing._internal.common_utils.TestCase but it doesn't.
7+
2026-07-24T18:47:03.6571251Z Got exit code 1
8+
2026-07-24T18:47:03.6571812Z No stepcurrent file found. Either pytest didn't get to run (e.g. import error) or file got deleted (contact dev infra)
9+
2026-07-24T18:47:03.6572299Z
10+
2026-07-24T18:47:03.6572801Z FINISHED PRINTING LOG FILE of distributed/test_c10d_torchcomms 1/1 (test/test-reports/distributed.test_c10d_torchcomms_1.1_57d5e8b26f6005c4_.log)
11+
2026-07-24T18:47:03.6573421Z
12+
2026-07-24T18:47:03.6573743Z Finished distributed/test_c10d_torchcomms 1/1 ... [2026-07-24 18:47:03.538120][5725.654382616], took 0.05min
13+
2026-07-24T18:47:04.6668511Z distributed/test_c10d_torchcomms 1/1 failed!
14+
2026-07-24T18:47:04.6668899Z Emitting td_test_failure_stats_v2
15+
2026-07-24T18:47:05.6765866Z /__w/pytorch/pytorch/tools/stats/upload_metrics.py:156: UserWarning: Error uploading metric td_test_failure_stats_v2 to DynamoDB: Unable to locate credentials
16+
2026-07-24T18:47:05.6766792Z warn(f"Error uploading metric {metric_name} to DynamoDB: {e}")
17+
2026-07-24T18:47:05.6767201Z Traceback (most recent call last):
18+
2026-07-24T18:47:05.6774802Z File "/__w/pytorch/pytorch/test/run_test.py", line 2397, in <module>
19+
2026-07-24T18:47:05.6775253Z main()
20+
2026-07-24T18:47:05.6775596Z File "/__w/pytorch/pytorch/test/run_test.py", line 2348, in main
21+
2026-07-24T18:47:05.6775976Z run_tests(
22+
2026-07-24T18:47:05.6776324Z File "/__w/pytorch/pytorch/test/run_test.py", line 2169, in run_tests
23+
2026-07-24T18:47:05.6776819Z Failed to upload artifacts: Unable to locate credentials
24+
2026-07-24T18:47:05.6777192Z Uploading artifacts took 1.08 seconds
25+
2026-07-24T18:47:05.6777850Z Writing 1 documents to S3 ossci-raw-job-status/ossci_uploaded_metrics/td_test_failure_stats_v2_1784918824_1072106c879011f19b01dec92ee577a0
26+
2026-07-24T18:47:05.6778657Z raise RuntimeError(failure.message + keep_going_message)
27+
#=MATCH=# 2026-07-24T18:47:05.6779110Z ‹RuntimeError: distributed/test_c10d_torchcomms 1/1 failed!›
28+
2026-07-24T18:47:05.6779385Z
29+
2026-07-24T18:47:05.6779655Z Tip: You can keep running tests even on failure by passing --keep-going to run_test.py.
30+
2026-07-24T18:47:05.6780239Z If running on CI, add the 'keep-going' label to your PR and rerun your jobs.
31+
2026-07-24T18:47:06.2821510Z
32+
2026-07-24T18:47:06.2821871Z real 0m9.377s
33+
2026-07-24T18:47:06.2822123Z user 0m6.972s
34+
2026-07-24T18:47:06.2822325Z sys 0m0.981s
35+
2026-07-24T18:47:06.2822520Z + sccache_epilogue
36+
2026-07-24T18:47:06.2823276Z + echo '::group::Sccache Compilation Log'
37+
2026-07-24T18:47:06.2823667Z + echo '=================== sccache compilation log ==================='
38+
2026-07-24T18:47:06.2824266Z + python /__w/pytorch/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
39+
2026-07-24T18:47:06.2825009Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
40+
2026-07-24T18:47:06.2825530Z + sccache --show-stats
41+
2026-07-24T18:47:06.2825773Z + sccache --stop-server
42+
2026-07-24T18:47:06.2826026Z + echo ::endgroup::
43+
2026-07-24T18:47:06.2826481Z ##[group]Sccache Compilation Log
44+
2026-07-24T18:47:06.2826807Z =================== sccache compilation log ===================
45+
2026-07-24T18:47:06.2827643Z storage write check failed: PermissionDenied (permanent) at write => S3Error { code: "AccessDenied", message: "Access Denied", resource: "", request_id: "4DGERBD62D4JFCZB" }
46+
2026-07-24T18:47:06.2828337Z
47+
2026-07-24T18:47:06.2828343Z
48+
2026-07-24T18:47:06.2828354Z
49+
2026-07-24T18:47:06.2828435Z Context:
50+
2026-07-24T18:47:06.2828546Z
51+
2026-07-24T18:47:06.2828854Z uri: https://s3.us-east-1.amazonaws.com/ossci-compiler-cache-circleci-v2/.sccache_check
52+
2026-07-24T18:47:06.2829264Z
53+
2026-07-24T18:47:06.2830913Z response: Parts { status: 403, version: HTTP/1.1, headers: {"x-amz-request-id": "4DGERBD62D4JFCZB", "x-amz-id-2": "VMMd199Eqe1rBr0hhGExkpxcTO5wtSeQLYBSvU1jBAqgTbCg2psrSCwoqL3yVlBHzOYXwFq0Wv0qejSlo4ZyiQuj4sJP5wLs", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 24 Jul 2026 18:44:29 GMT", "connection": "close", "server": "AmazonS3"} }
54+
2026-07-24T18:47:06.2832452Z
55+
2026-07-24T18:47:06.2832532Z service: s3
56+
2026-07-24T18:47:06.2832662Z
57+
2026-07-24T18:47:06.2832759Z path: .sccache_check
58+
2026-07-24T18:47:06.2832910Z
59+
2026-07-24T18:47:06.2832993Z written: 13
60+
2026-07-24T18:47:06.2833117Z
61+
2026-07-24T18:47:06.2833122Z
62+
2026-07-24T18:47:06.2833126Z
63+
2026-07-24T18:47:06.2833420Z =========== If your build fails, please take a look at the log above for possible reasons ===========
64+
2026-07-24T18:47:06.2833925Z Compile requests 197
65+
2026-07-24T18:47:06.2834231Z Compile requests executed 97
66+
2026-07-24T18:47:06.2834526Z Cache hits 62
67+
2026-07-24T18:47:06.2834814Z Cache hits (C/C++) 62
68+
2026-07-24T18:47:06.2835102Z Cache misses 29
69+
2026-07-24T18:47:06.2835392Z Cache misses (C/C++) 29
70+
2026-07-24T18:47:06.2835688Z Cache hits rate 68.13 %
71+
2026-07-24T18:47:06.2835995Z Cache hits rate (C/C++) 68.13 %
72+
2026-07-24T18:47:06.2836297Z Cache timeouts 0
73+
2026-07-24T18:47:06.2836587Z Cache read errors 0
74+
2026-07-24T18:47:06.2836884Z Forced recaches 0
75+
2026-07-24T18:47:06.2837174Z Cache write errors 29
76+
2026-07-24T18:47:06.2837462Z Cache errors 2
77+
2026-07-24T18:47:06.2837752Z Cache errors (C/C++) 2
78+
2026-07-24T18:47:06.2838046Z Compilations 29
79+
2026-07-24T18:47:06.2838343Z Compilation failures 4
80+
2026-07-24T18:47:06.2838650Z Non-cacheable compilations 0
81+
2026-07-24T18:47:06.2838955Z Non-cacheable calls 34
82+
2026-07-24T18:47:06.2839254Z Non-compilation calls 66
83+
2026-07-24T18:47:06.2839557Z Unsupported compiler calls 0
84+
2026-07-24T18:47:06.2839861Z Average cache write 0.000 s
85+
2026-07-24T18:47:06.2840168Z Average compiler 17.852 s
86+
2026-07-24T18:47:06.2840499Z Average cache read hit 0.026 s
87+
2026-07-24T18:47:06.2840808Z Failed distributed compilations 0
88+
2026-07-24T18:47:06.2841020Z
89+
2026-07-24T18:47:06.2841113Z Non-cacheable reasons:
90+
2026-07-24T18:47:06.2841361Z multiple input files 34
91+
2026-07-24T18:47:06.2841563Z
92+
2026-07-24T18:47:06.2841797Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: /
93+
2026-07-24T18:47:06.2842443Z Base directories (none)
94+
2026-07-24T18:47:06.2842728Z Version (client) 0.16.0
95+
2026-07-24T18:47:06.2843014Z Stopping sccache server...
96+
2026-07-24T18:47:06.2843273Z Compile requests 197
97+
2026-07-24T18:47:06.2843570Z Compile requests executed 97
98+
2026-07-24T18:47:06.2843900Z Cache hits 62
99+
2026-07-24T18:47:06.2844181Z Cache hits (C/C++) 62
100+
2026-07-24T18:47:06.2844460Z Cache misses 29
101+
2026-07-24T18:47:06.2844753Z Cache misses (C/C++) 29
102+
2026-07-24T18:47:06.2845042Z Cache hits rate 68.13 %
103+
2026-07-24T18:47:06.2845339Z Cache hits rate (C/C++) 68.13 %
104+
2026-07-24T18:47:06.2845633Z Cache timeouts 0
105+
2026-07-24T18:47:06.2845922Z Cache read errors 0
106+
2026-07-24T18:47:06.2846208Z Forced recaches 0
107+
2026-07-24T18:47:06.2846496Z Cache write errors 29
108+
2026-07-24T18:47:06.2846790Z Cache errors 2
109+
2026-07-24T18:47:06.2847075Z Cache errors (C/C++) 2
110+
2026-07-24T18:47:06.2847362Z Compilations 29
111+
2026-07-24T18:47:06.2847652Z Compilation failures 4
112+
2026-07-24T18:47:06.2847959Z Non-cacheable compilations 0
113+
2026-07-24T18:47:06.2848258Z Non-cacheable calls 34
114+
2026-07-24T18:47:06.2848552Z Non-compilation calls 66
115+
2026-07-24T18:47:06.2849005Z Unsupported compiler calls 0
116+
2026-07-24T18:47:06.2849329Z Average cache write 0.000 s
117+
2026-07-24T18:47:06.2849630Z Average compiler 17.852 s
118+
2026-07-24T18:47:06.2849931Z Average cache read hit 0.026 s
119+
2026-07-24T18:47:06.2850240Z Failed distributed compilations 0
120+
2026-07-24T18:47:06.2850444Z
121+
2026-07-24T18:47:06.2850536Z Non-cacheable reasons:
122+
2026-07-24T18:47:06.2850783Z multiple input files 34
123+
2026-07-24T18:47:06.2851100Z
124+
2026-07-24T18:47:06.2851336Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: /
125+
2026-07-24T18:47:06.2851776Z Base directories (none)
126+
2026-07-24T18:47:06.2852060Z Version (client) 0.16.0
127+
2026-07-24T18:47:06.2852502Z ##[endgroup]
128+
2026-07-24T18:47:06.7107531Z ##[error][OSDC] Step script exited with code 1. This is a script/workflow error, not an infrastructure issue. Check the step logs above for the actual failure.
129+
2026-07-24T18:47:06.7128750Z ##[error]Process completed with exit code 1.
130+
2026-07-24T18:47:06.7147309Z ##[error]Executing the custom container implementation failed. Please contact your self hosted runner administrator.
131+
2026-07-24T18:47:06.7216637Z ##[group]Run cat test/**/*_toprint.log || true

0 commit comments

Comments
 (0)