Skip to content

fix: sub-classify the dominant patch 'other' error bucket (Closes #1537) - #1539

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1537-patch-other-bucket-2
Jul 31, 2026
Merged

fix: sub-classify the dominant patch 'other' error bucket (Closes #1537)#1539
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1537-patch-other-bucket-2

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Summary

Automated evolution PR for issue #1537 — the patch tool's other error bucket was 81% of all patch failures (70/86 in a 7d introspection window): opaque to the digest and giving the agent no recovery signal, driving retry spirals up to 6 deep.

Root cause (verified against the real code path)

The fuzzy matcher's dominant failure message — "Could not find a match for old_string in the file" (tools/fuzzy_match.py:197, fired on every fuzzy-match miss) — matched neither classifier:

  • Introspection (scripts/introspection_extract.py): _NO_MATCH_RE requires the literal phrase "no match", but the message says "Could not find a match" → fell to the opaque other bucket (the measured 81%).
  • Runtime (tools/file_operations.py::classify_file_error): the fuzzy_match branch checked "did not match" / "no match" / ("match" + "block") — none match "Could not find a match" → fell to the generic error class with the unhelpful recovery "The operation failed. ... CHANGE the call.", giving the model no signal to re-read the file.

The recovery directive mechanism (hard refuse + write_file at threshold 3) already existed (#507/#1037) — the gap was purely the misclassification, plus surfacing the write_file escape one step earlier.

Changes (3 targeted fixes)

  1. Introspection (scripts/introspection_extract.py): add a patch-no-match sub-category regex for "could not find (a) match", placed before _NO_MATCH_RE. The 70 previously-other failures now become a trackable, actionable signal in the digest.

  2. Runtime classifier (tools/file_operations.py::classify_file_error): extend the fuzzy_match branch to recognize "could not find" / "not find a match", so the model gets the "Re-read the file and copy the EXACT current text" recovery directive instead of the generic one.

  3. Earlier write_file nudge (tools/file_tools.py): on the 2nd consecutive no-match failure on the same file, surface write_file as the recommended escape BEFORE the hard 3rd-failure refuse-gate fires. The existing gate already refuses at threshold 3; this biases the model toward the reliably-terminating rewrite path one step earlier. Deliberately NOT gated on has_diagnostic — the consecutive-failure count + write_file signal is complementary to (not mutually exclusive with) the structured diagnostic.

Tests

  • test_patch_no_match_classified_not_other — introspection classifies "Could not find a match" as patch-no-match, not other.
  • test_could_not_find_match_is_fuzzy — runtime classifier routes it to fuzzy_match recovery.
  • test_second_consecutive_no_match_nudges_write_file — the [FIX] patch tool 'other' error bucket still dominant — 70/86 failures (81%) after decomposition fix #1537 nudge fires on attempt 2 (not 1), recommends write_file, and is not the hard refuse-gate.
  • test_first_two_failures_no_hard_escalation (existing, updated) — tightened to assert on PERMANENT FAILURE so the new soft nudge remains valid.

Validation

  • Pre-PR targeted test shard: ✅ PASS (3 shards)
  • ruff check: ✅ All checks passed
  • Full related suites (324 tests across patch/file/fuzzy/introspection/recovery): ✅ all pass
  • Diff: 141 insertions, 3 deletions across 6 files — within the 200-line self-merge cap.

Closes #1537

The patch tool's 'other' error bucket was 81% of all patch failures
(70/86 in a 7d introspection window) — opaque to the digest and giving
the agent no recovery signal, driving retry spirals up to 6 deep.

Root cause: the fuzzy matcher's dominant failure message ("Could not
find a match for old_string in the file") matched NEITHER classifier:
  - introspection _NO_MATCH_RE needs literal "no match", the message
    says "Could not find a match" → fell to 'other'
  - runtime classify_file_error checked "did not match"/"no match"
    → fell to the generic 'error' class with the unhelpful "The
    operation failed" recovery

Three targeted fixes, each verified against the real code path:

1. Introspection (scripts/introspection_extract.py): add a
   patch-no-match sub-category for "could not find (a) match", placed
   before _NO_MATCH_RE so the 70 previously-'other' failures become a
   trackable, actionable signal in the digest.

2. Runtime classifier (tools/file_operations.py:classify_file_error):
   extend the fuzzy_match branch to recognize "could not find" / "not
   find a match" so the model gets the "re-read the file and copy the
   EXACT current text" recovery directive instead of the generic one.

3. Earlier write_file nudge (tools/file_tools.py): on the 2nd
   consecutive no-match failure on the same file, surface write_file
   as the recommended escape BEFORE the hard 3rd-failure refuse-gate
   fires. The existing gate already refuses at threshold 3; this
   biases the model toward the reliably-terminating rewrite path one
   step earlier. Deliberately not gated on has_diagnostic — the count
   + write_file signal is complementary to the structured diagnostic,
   not mutually exclusive.

Tests: introspection patch-no-match classification, runtime classifier
routing, and the new 2nd-failure write_file nudge (the existing
test_first_two_failures test is tightened to assert on PERMANENT
FAILURE so the new soft nudge remains valid).

Closes #1537

Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
@github-actions github-actions Bot added fix Bug or fix introspection Found by session introspection labels Jul 31, 2026
@Lexus2016
Lexus2016 merged commit c494f79 into main Jul 31, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug or fix introspection Found by session introspection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FIX] patch tool 'other' error bucket still dominant — 70/86 failures (81%) after decomposition fix

1 participant