Skip to content

feat(introspection): track refusal-recovery rate, not just refusal count (Closes #1366) - #1367

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1366-refusal-recovery-rate
Jul 27, 2026
Merged

feat(introspection): track refusal-recovery rate, not just refusal count (Closes #1366)#1367
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1366-refusal-recovery-rate

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Closes #1366 — Child B of #1327.

The gap

Child A (#1356, merged bb0a325cc) added RECOVERY_BEFORE_REFUSAL_GUIDANCE: before refusing for a missing capability, propose an alternative path.

Nothing measured whether it worked. _REFUSAL_RE matches these identically:

Both bumped one counter, so the 51/week signal could not move even if the guidance were perfect — and a realized-impact verdict on #1356 would have been forced to no-signal for lack of a comparable rate.

Change

_RECOVERY_RE — a turn counts as recovered when it refuses AND offers a concrete alternative. Deliberately conservative: pivot phrases that introduce a different path (instead, alternatively, workaround, however I can, what I can do), not mere hedging. Over-matching would inflate the rate and make #1356 look effective when it is not — worse than under-counting.

Emitted additively, per session and in the digest:

Field Meaning
refusals_with_recovery count of refusals that offered a path forward
refusal_recovery_rate recovered ÷ refusals

The denominator is refusals, not sessions — this measures the quality of the refusals that happened, not how often the agent refuses. Both numbers move independently and both matter.

refusals / refusals_or_access_denied keep their exact current meaning; a back-compat test asserts the aggregate is unchanged.

Measured on the real corpus

sessions_scanned: 4629
refusals:         143
with_recovery:     51
recovery_rate:   0.3566

So #1356 currently fires on roughly a third of refusals — two thirds are still bare. That is the baseline Child C (recovery dispatcher, ~150 lines, still unfiled) needs in order to demonstrate it improved anything.

Success criteria

  • A refusal that proposes an alternative is counted as recovered
  • A bare refusal is not
  • refusal_recovery_rate appears in the digest
  • Existing refusals counts are unchanged

43 passing in test_introspection_extract.py (9 new), 82 across the introspection and realized-impact suites; ruff clean.

…unt (#1366)

Child B of #1327. Child A (#1356, merged bb0a325) told the agent to propose an
alternative path before refusing for a missing capability. Nothing measured
whether it worked: _REFUSAL_RE matches "I can't do that" and "I can't do that
directly, but I can via terminal" identically, so the 51/week signal could not
move even if the guidance were perfect, and a realized-impact verdict on #1356
would have been forced to no-signal for lack of a comparable rate.

Adds _RECOVERY_RE — a turn counts as recovered when it refuses AND offers a
concrete alternative. Deliberately conservative: pivot phrases that introduce a
different path (instead / alternatively / workaround / however-I-can / what I
can do), not mere hedging. Over-matching would inflate the rate and make #1356
look effective when it is not, which is worse than under-counting.

Emitted additively, per session and in the digest:
- refusals_with_recovery — count
- refusal_recovery_rate  — recovered / refusals

The rate's denominator is refusals, not sessions: it measures the quality of the
refusals that happened, not how often the agent refuses. Both numbers move
independently and both matter.

refusals / refusals_or_access_denied keep their exact current meaning; a
back-compat test asserts the aggregate is unchanged.

Measured on the real session corpus: 4629 sessions, 143 refusals, 51 recovered
— a 35.7% recovery rate. So #1356 fires on roughly a third of refusals today,
which is the baseline Child C (recovery dispatcher) needs in order to show it
improved anything.

43 passing in test_introspection_extract.py (9 new), 82 across the introspection
and realized-impact suites; ruff clean.

Closes #1366
@github-actions github-actions Bot added feature introspection Found by session introspection labels Jul 27, 2026
@Lexus2016
Lexus2016 merged commit 64b7b88 into main Jul 27, 2026
37 checks passed
@Lexus2016
Lexus2016 deleted the evolution/issue-1366-refusal-recovery-rate branch July 27, 2026 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature introspection Found by session introspection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[INCREMENT] Refusal-recovery rate tracking (Child B of #1327)

1 participant