Skip to content

Commit 4efff15

Browse files
authored
[log-classifier] Remove generic OSDC rule, use Bedrock inference profiles, widen LLM context (#8391)
## Summary Three related fixes to the log-classifier Lambda: 1. **Remove the generic `OSDC step script failure` rule.** `[OSDC] Step script exited...` is a generic wrapper message that hides the real failure, so classifying on it produced useless results. Removing it lets these logs fall through to the generic `GHA error` catch-all (and then the LLM). 2. **Keep `GHA error` as the lowest-priority rule.** Rules in `ruleset.toml` are ordered by priority (first = highest), and `main.rs` only kicks off the LLM classifier when the best match is the *last* rule. So the generic `^##[error](.*)` catch-all must stay last — `New modules are not documented correctly` is moved above it. 3. **Use `us.*` cross-region inference profile IDs for Bedrock.** The bare `anthropic.claude-*` on-demand IDs are not directly invokable for these models; the `us.anthropic.*` inference profiles are required. 4. **Widen the LLM error-context window from 100 → 500 lines.** `make_query` centers the snippet on the matched line: `[error_line - num_lines/2, error_line + num_lines/2]`. The catch-all match often lands on a wrapper line that sits well above the real cause, so a narrow window misses it. See the 90075305031 case below: the match is at line 600 but the root cause is at line 427 (173 lines up) — with 100 the window is only `[550, 650]` and the LLM settles for a nearby symptom line; with 500 (`[350, 850]`) it reaches the real error. | context | surfaced line for 90075305031 | correct? | |---|---|---| | 100 | line 453: `##[error]TypeError: Cannot read properties of null (reading 'jobPod')` (a symptom/boilerplate line) | no | | 500 | line 427: `##[error]Error: pod failed to come online ... backoff timeout` | yes | Caveat: 500 is empirically fitted (`±250` lines), not principled — a log whose real cause is even further from the wrapper line would still be missed. A more robust fix would anchor the snippet on the last error region rather than widening symmetrically, but that is a larger change. Cost/latency of ~5x prompt tokens per fallback call is negligible for Haiku and the fallback only fires on the catch-all rule. ## Relationship to #8382 This is an alternative to @jeanschmidt's #8382, which drops the CI-wrapper boilerplate lines at preprocessing time. This PR reaches the same outcome via a different mechanism: every log falls through to the `GHA error` catch-all and the **LLM fallback** re-locates the meaningful error line. ## Test plan - `cargo check` passes (only pre-existing deprecation warnings, unrelated to these changes). Re-ran the four real prod jobs from [Jean's validation table](#8382 (comment)) through this branch (read-only, no DynamoDB write): | job_id | Jean's AFTER (#8382) | This branch's FINAL | verdict | |---|---|---|---| | 90151654008 (P1 exit-code) | `Action failed... non-human actor: pytorch-bot` | `##[error]Action failed with error: Workflow initiated by non-human actor: pytorch-bot (type: Bot). Add bot to allowed_bots list...` (line 472, Bedrock haiku) | same | | 90075305031 (P2 OSDC) | `Error: pod failed to come online ... backoff timeout` | `##[error]Error: pod failed to come online ... is unhealthy with phase status Pending: backoff timeout` (line 427, Bedrock haiku) | same | | 90103895526 (P3 container) | `[OSDC] Step cancelled by GitHub Actions (received SIGINT)` | `inductor/test_cpu_repro.py::CPUReproTests::test_vec_compare_op_cpu_only Command took >60min, returning 124` (line 9532, Bedrock haiku) | better — surfaced the hung test that caused the SIGINT | | 90131159825 (P3 container) | `Error: pod failed to come online ... backoff timeout` | `Pod lf-l-... phase=Pending ... waiting=[job: ErrImagePull] (2596s/7200s)` (line 220, Bedrock haiku) | better — pinpointed `ErrImagePull` | | P4 jobPod | n/a (0 prod rows in 120d) | not run — no real log exists | matches Jean's note | All four surface the real failure instead of the useless `exit code 1` / OSDC boilerplate; two cases trace past the generic pod/SIGINT symptom to the actual root cause.
1 parent f163b03 commit 4efff15

3 files changed

Lines changed: 5 additions & 9 deletions

File tree

aws/lambda/log-classifier/ruleset.toml

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -314,13 +314,9 @@ name = 'ghstack merge check error'
314314
pattern = '^Mergeability Error: .*'
315315

316316
[[rule]]
317-
name = 'OSDC step script failure'
318-
pattern = '\[OSDC\] Step script exited.*'
317+
name = 'New modules are not documented correctly'
318+
pattern = 'You added the following module\(s\) to the PyTorch namespace .* but they'
319319

320320
[[rule]]
321321
name = 'GHA error'
322322
pattern = '^##\[error\](.*)'
323-
324-
[[rule]]
325-
name = 'New modules are not documented correctly'
326-
pattern = 'You added the following module\(s\) to the PyTorch namespace .* but they'

aws/lambda/log-classifier/src/bedrock.rs

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,8 +119,8 @@ pub async fn make_query(
119119
error_line: &usize,
120120
num_lines: usize,
121121
) -> Option<SerializedMatch> {
122-
let model_id_primary = String::from("anthropic.claude-haiku-4-5-20251001-v1:0");
123-
let model_id_secondary = String::from("anthropic.claude-sonnet-4-6");
122+
let model_id_primary = String::from("us.anthropic.claude-haiku-4-5-20251001-v1:0");
123+
let model_id_secondary = String::from("us.anthropic.claude-sonnet-4-6");
124124
let line_numbers = vec![*error_line];
125125

126126
let log_snippet = get_snippets(log, line_numbers, num_lines / 2, num_lines + 1);

aws/lambda/log-classifier/src/main.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ async fn handle(
5151
// check if match has the lowest priority in the ruleset
5252
if best_match.rule.name == ruleset.rules.last().unwrap().name {
5353
// kick off the llm to get the rule
54-
match make_query(&log, &best_match.line_number, 100).await {
54+
match make_query(&log, &best_match.line_number, 500).await {
5555
Some(llm_match_json) => {
5656
body = serde_json::to_string_pretty(&llm_match_json)?;
5757
match_json = llm_match_json;

0 commit comments

Comments
 (0)