feat(introspection): track refusal-recovery rate, not just refusal count (Closes #1366) - #1367
Merged
Merged
Conversation
…unt (#1366) Child B of #1327. Child A (#1356, merged bb0a325) told the agent to propose an alternative path before refusing for a missing capability. Nothing measured whether it worked: _REFUSAL_RE matches "I can't do that" and "I can't do that directly, but I can via terminal" identically, so the 51/week signal could not move even if the guidance were perfect, and a realized-impact verdict on #1356 would have been forced to no-signal for lack of a comparable rate. Adds _RECOVERY_RE — a turn counts as recovered when it refuses AND offers a concrete alternative. Deliberately conservative: pivot phrases that introduce a different path (instead / alternatively / workaround / however-I-can / what I can do), not mere hedging. Over-matching would inflate the rate and make #1356 look effective when it is not, which is worse than under-counting. Emitted additively, per session and in the digest: - refusals_with_recovery — count - refusal_recovery_rate — recovered / refusals The rate's denominator is refusals, not sessions: it measures the quality of the refusals that happened, not how often the agent refuses. Both numbers move independently and both matter. refusals / refusals_or_access_denied keep their exact current meaning; a back-compat test asserts the aggregate is unchanged. Measured on the real session corpus: 4629 sessions, 143 refusals, 51 recovered — a 35.7% recovery rate. So #1356 fires on roughly a third of refusals today, which is the baseline Child C (recovery dispatcher) needs in order to show it improved anything. 43 passing in test_introspection_extract.py (9 new), 82 across the introspection and realized-impact suites; ruff clean. Closes #1366
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1366 — Child B of #1327.
The gap
Child A (#1356, merged
bb0a325cc) addedRECOVERY_BEFORE_REFUSAL_GUIDANCE: before refusing for a missing capability, propose an alternative path.Nothing measured whether it worked.
_REFUSAL_REmatches these identically:Both bumped one counter, so the 51/week signal could not move even if the guidance were perfect — and a realized-impact verdict on #1356 would have been forced to
no-signalfor lack of a comparable rate.Change
_RECOVERY_RE— a turn counts as recovered when it refuses AND offers a concrete alternative. Deliberately conservative: pivot phrases that introduce a different path (instead,alternatively,workaround,however I can,what I can do), not mere hedging. Over-matching would inflate the rate and make #1356 look effective when it is not — worse than under-counting.Emitted additively, per session and in the digest:
refusals_with_recoveryrefusal_recovery_rateThe denominator is refusals, not sessions — this measures the quality of the refusals that happened, not how often the agent refuses. Both numbers move independently and both matter.
refusals/refusals_or_access_deniedkeep their exact current meaning; a back-compat test asserts the aggregate is unchanged.Measured on the real corpus
So #1356 currently fires on roughly a third of refusals — two thirds are still bare. That is the baseline Child C (recovery dispatcher, ~150 lines, still unfiled) needs in order to demonstrate it improved anything.
Success criteria
refusal_recovery_rateappears in the digestrefusalscounts are unchanged43 passing in
test_introspection_extract.py(9 new), 82 across the introspection and realized-impact suites; ruff clean.