fix: sub-classify the dominant patch 'other' error bucket (Closes #1537) - #1539
Merged
Merged
Conversation
The patch tool's 'other' error bucket was 81% of all patch failures
(70/86 in a 7d introspection window) — opaque to the digest and giving
the agent no recovery signal, driving retry spirals up to 6 deep.
Root cause: the fuzzy matcher's dominant failure message ("Could not
find a match for old_string in the file") matched NEITHER classifier:
- introspection _NO_MATCH_RE needs literal "no match", the message
says "Could not find a match" → fell to 'other'
- runtime classify_file_error checked "did not match"/"no match"
→ fell to the generic 'error' class with the unhelpful "The
operation failed" recovery
Three targeted fixes, each verified against the real code path:
1. Introspection (scripts/introspection_extract.py): add a
patch-no-match sub-category for "could not find (a) match", placed
before _NO_MATCH_RE so the 70 previously-'other' failures become a
trackable, actionable signal in the digest.
2. Runtime classifier (tools/file_operations.py:classify_file_error):
extend the fuzzy_match branch to recognize "could not find" / "not
find a match" so the model gets the "re-read the file and copy the
EXACT current text" recovery directive instead of the generic one.
3. Earlier write_file nudge (tools/file_tools.py): on the 2nd
consecutive no-match failure on the same file, surface write_file
as the recommended escape BEFORE the hard 3rd-failure refuse-gate
fires. The existing gate already refuses at threshold 3; this
biases the model toward the reliably-terminating rewrite path one
step earlier. Deliberately not gated on has_diagnostic — the count
+ write_file signal is complementary to the structured diagnostic,
not mutually exclusive.
Tests: introspection patch-no-match classification, runtime classifier
routing, and the new 2nd-failure write_file nudge (the existing
test_first_two_failures test is tightened to assert on PERMANENT
FAILURE so the new soft nudge remains valid).
Closes #1537
Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automated evolution PR for issue #1537 — the patch tool's
othererror bucket was 81% of all patch failures (70/86 in a 7d introspection window): opaque to the digest and giving the agent no recovery signal, driving retry spirals up to 6 deep.Root cause (verified against the real code path)
The fuzzy matcher's dominant failure message —
"Could not find a match for old_string in the file"(tools/fuzzy_match.py:197, fired on every fuzzy-match miss) — matched neither classifier:scripts/introspection_extract.py):_NO_MATCH_RErequires the literal phrase"no match", but the message says "Could not find a match" → fell to the opaqueotherbucket (the measured 81%).tools/file_operations.py::classify_file_error): thefuzzy_matchbranch checked"did not match"/"no match"/("match" + "block")— none match "Could not find a match" → fell to the genericerrorclass with the unhelpful recovery "The operation failed. ... CHANGE the call.", giving the model no signal to re-read the file.The recovery directive mechanism (hard refuse + write_file at threshold 3) already existed (
#507/#1037) — the gap was purely the misclassification, plus surfacing the write_file escape one step earlier.Changes (3 targeted fixes)
Introspection (
scripts/introspection_extract.py): add apatch-no-matchsub-category regex for"could not find (a) match", placed before_NO_MATCH_RE. The 70 previously-otherfailures now become a trackable, actionable signal in the digest.Runtime classifier (
tools/file_operations.py::classify_file_error): extend thefuzzy_matchbranch to recognize"could not find"/"not find a match", so the model gets the "Re-read the file and copy the EXACT current text" recovery directive instead of the generic one.Earlier write_file nudge (
tools/file_tools.py): on the 2nd consecutive no-match failure on the same file, surface write_file as the recommended escape BEFORE the hard 3rd-failure refuse-gate fires. The existing gate already refuses at threshold 3; this biases the model toward the reliably-terminating rewrite path one step earlier. Deliberately NOT gated onhas_diagnostic— the consecutive-failure count + write_file signal is complementary to (not mutually exclusive with) the structured diagnostic.Tests
test_patch_no_match_classified_not_other— introspection classifies "Could not find a match" aspatch-no-match, notother.test_could_not_find_match_is_fuzzy— runtime classifier routes it tofuzzy_matchrecovery.test_second_consecutive_no_match_nudges_write_file— the [FIX] patch tool 'other' error bucket still dominant — 70/86 failures (81%) after decomposition fix #1537 nudge fires on attempt 2 (not 1), recommends write_file, and is not the hard refuse-gate.test_first_two_failures_no_hard_escalation(existing, updated) — tightened to assert onPERMANENT FAILUREso the new soft nudge remains valid.Validation
ruff check: ✅ All checks passedCloses #1537