Commit 4efff15
authored
[log-classifier] Remove generic OSDC rule, use Bedrock inference profiles, widen LLM context (#8391)
## Summary
Three related fixes to the log-classifier Lambda:
1. **Remove the generic `OSDC step script failure` rule.** `[OSDC] Step
script exited...` is a generic wrapper message that hides the real
failure, so classifying on it produced useless results. Removing it lets
these logs fall through to the generic `GHA error` catch-all (and then
the LLM).
2. **Keep `GHA error` as the lowest-priority rule.** Rules in
`ruleset.toml` are ordered by priority (first = highest), and `main.rs`
only kicks off the LLM classifier when the best match is the *last*
rule. So the generic `^##[error](.*)` catch-all must stay last — `New
modules are not documented correctly` is moved above it.
3. **Use `us.*` cross-region inference profile IDs for Bedrock.** The
bare `anthropic.claude-*` on-demand IDs are not directly invokable for
these models; the `us.anthropic.*` inference profiles are required.
4. **Widen the LLM error-context window from 100 → 500 lines.**
`make_query` centers the snippet on the matched line: `[error_line -
num_lines/2, error_line + num_lines/2]`. The catch-all match often lands
on a wrapper line that sits well above the real cause, so a narrow
window misses it. See the 90075305031 case below: the match is at line
600 but the root cause is at line 427 (173 lines up) — with 100 the
window is only `[550, 650]` and the LLM settles for a nearby symptom
line; with 500 (`[350, 850]`) it reaches the real error.
| context | surfaced line for 90075305031 | correct? |
|---|---|---|
| 100 | line 453: `##[error]TypeError: Cannot read properties of null
(reading 'jobPod')` (a symptom/boilerplate line) | no |
| 500 | line 427: `##[error]Error: pod failed to come online ... backoff
timeout` | yes |
Caveat: 500 is empirically fitted (`±250` lines), not principled — a log
whose real cause is even further from the wrapper line would still be
missed. A more robust fix would anchor the snippet on the last error
region rather than widening symmetrically, but that is a larger change.
Cost/latency of ~5x prompt tokens per fallback call is negligible for
Haiku and the fallback only fires on the catch-all rule.
## Relationship to #8382
This is an alternative to @jeanschmidt's #8382, which drops the
CI-wrapper boilerplate lines at preprocessing time. This PR reaches the
same outcome via a different mechanism: every log falls through to the
`GHA error` catch-all and the **LLM fallback** re-locates the meaningful
error line.
## Test plan
- `cargo check` passes (only pre-existing deprecation warnings,
unrelated to these changes).
Re-ran the four real prod jobs from [Jean's validation
table](#8382 (comment))
through this branch (read-only, no DynamoDB write):
| job_id | Jean's AFTER (#8382) | This branch's FINAL | verdict |
|---|---|---|---|
| 90151654008 (P1 exit-code) | `Action failed... non-human actor:
pytorch-bot` | `##[error]Action failed with error: Workflow initiated by
non-human actor: pytorch-bot (type: Bot). Add bot to allowed_bots
list...` (line 472, Bedrock haiku) | same |
| 90075305031 (P2 OSDC) | `Error: pod failed to come online ... backoff
timeout` | `##[error]Error: pod failed to come online ... is unhealthy
with phase status Pending: backoff timeout` (line 427, Bedrock haiku) |
same |
| 90103895526 (P3 container) | `[OSDC] Step cancelled by GitHub Actions
(received SIGINT)` |
`inductor/test_cpu_repro.py::CPUReproTests::test_vec_compare_op_cpu_only
Command took >60min, returning 124` (line 9532, Bedrock haiku) | better
— surfaced the hung test that caused the SIGINT |
| 90131159825 (P3 container) | `Error: pod failed to come online ...
backoff timeout` | `Pod lf-l-... phase=Pending ... waiting=[job:
ErrImagePull] (2596s/7200s)` (line 220, Bedrock haiku) | better —
pinpointed `ErrImagePull` |
| P4 jobPod | n/a (0 prod rows in 120d) | not run — no real log exists |
matches Jean's note |
All four surface the real failure instead of the useless `exit code 1` /
OSDC boilerplate; two cases trace past the generic pod/SIGINT symptom to
the actual root cause.1 parent f163b03 commit 4efff15
3 files changed
Lines changed: 5 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
318 | | - | |
| 317 | + | |
| 318 | + | |
319 | 319 | | |
320 | 320 | | |
321 | 321 | | |
322 | 322 | | |
323 | | - | |
324 | | - | |
325 | | - | |
326 | | - | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
123 | | - | |
| 122 | + | |
| 123 | + | |
124 | 124 | | |
125 | 125 | | |
126 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
54 | | - | |
| 54 | + | |
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
| |||
0 commit comments