Skip to content

feat: add quota_exhausted failure category for sub-tool paid backends (Closes #1489) - #1527

Merged
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1489-quota-exhausted
Jul 31, 2026
Merged

feat: add quota_exhausted failure category for sub-tool paid backends (Closes #1489)#1527
Lexus2016 merged 1 commit into
mainfrom
evolution/issue-1489-quota-exhausted

Conversation

@Lexus2016

Copy link
Copy Markdown
Owner

Summary

Add a distinct non-retriable quota_exhausted category to the cross-tool failure classifier, separating hard quota/billing exhaustion from retriable rate limits.

Problem (issue #1489)

Quota/billing exhaustion from sub-tool paid backends (vision_analyze, search_files, tool_call, terminal) surfaced as generic tool failures indistinguishable from a tool bug. The agent then retried the failing tool instead of recognizing "quota exhausted — switch approach." The existing rate_limited category was should_retry=True, so "quota exceeded" errors triggered retry spirals on tools that could never succeed until quota resets.

Changes

tools/tool_failure_classifier.py (core):

  • Add ToolFailureCategory.quota_exhausted — non-retriable, distinct from rate_limited
  • Add 7 billing-specific regex patterns BEFORE the generic rate_limited rules (first match wins):
    • quota exceeded, billing.*exhausted|limit|exceeded|disabled|failed, insufficient/out of credit|quota|balance, no credits remaining|left|available, usage limit exceeded|reached, 402, payment required
  • Set _CATEGORY_RETRYABLE[quota_exhausted] = False
  • Add actionable hint: "Stop retrying and surface the quota state to the user"

tools/recovery_strategy_dispatcher.py (recovery):

  • Map quota_exhaustedRecoveryStrategy.surface_blocker (stop retrying, report the blocker)

Tests (both files):

  • TestQuotaExhausted — 9 parametrized billing error patterns + priority test
  • test_recover_from_failure_quota_exhausted_surfaces_blocker — end-to-end classify+dispatch
  • Updated TestRateLimited — removed "quota exceeded" from rate_limited test cases (now classified as quota_exhausted)
  • Updated dispatcher parametrized test — added quota_exhaustedsurface_blocker row

Verification

  • ruff check
  • ruff format --check
  • pytest tests/tools/test_tool_failure_classifier.py tests/tools/test_recovery_strategy_dispatcher.py — 101 passed

Diff size

4 files changed, 135 insertions(+), 18 deletions(-) — within the 200-line self-merge cap.

Closes #1489

Add a distinct non-retriable  category to the cross-tool
failure classifier, separating hard quota/billing exhaustion (out of credits,
billing plan cap reached, 402 Payment Required) from retriable rate limits
(429, rate-limit throttles). The recovery dispatcher maps the new category
to  so the agent stops retrying and surfaces the quota state
instead of looping on a tool that cannot succeed until quota resets.

Closes #1489

Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
@github-actions github-actions Bot added the bug Something isn't working label Jul 31, 2026
@Lexus2016
Lexus2016 merged commit e8ecc51 into main Jul 31, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[UX] Quota/billing errors surface as generic tool failures in 4 downstream tools — 26 events/7d, agent retries instead of recognizing quota exhaustion

1 participant