ci(backend): widen the AutoPilot tool-schema budget to 65,000 chars - #14455
ci(backend): widen the AutoPilot tool-schema budget to 65,000 chars#14455Pwuts wants to merge 1 commit into
Conversation
dev alone measures 60,984 of the 61,000 budget — sixteen characters of headroom, against a comment claiming ~1,400. The previous bump's headroom was consumed in 17 days by nine tools growing 50-400 chars each, with no single PR at fault, and the next PR to add anything (#14434, +53) was ejected from the merge queue. 65,000 is sized against what concurrent in-flight PRs add in aggregate — the ten v0.7.5 PRs add 1,763 chars — because each branch's CI only sees its own delta, so a global counter can only ever fail in the merge queue. It leaves 2,253 chars over the 62,747 the registry measures with all ten merged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
github-actions
Bot
added
cla: signed
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 10 included reviews per hour; 2 remain after this review. 📜 Recent review details🧰 Additional context used📓 Path-based instructions (1)Format Python code with `poetry run format`📄 CodeRabbit inference engine (AGENTS.md) Files:
🔇 Additional comments (1)
WalkthroughThe tool schema regression test raises ChangesTool schema budget
Priority: ⬇️ Low Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to This updates the tool-schema test budget to accommodate measured schema growth without changing product behavior. No current merge-readiness risk remains. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 Full measurement tablesEvery row is that PR merged into Per-PR totals against the current 61,000 budget
Two cross the line. #14434 by 37, and #14424 by 1,694 — though #14424 carries its own bump to 63,000, so on its own head it passes with 306 to spare. All nine merges applied cleanly; there are no conflicts in the set other than the intended one this PR creates with #14424 on the budget line. What actually growsBetween
Nothing else moves by a single character. #14412 and #14420 touch The ten largest tools in the merged state
Top ten: 19,293 chars, 30.7% of the total. Across all 73 tools, description text is 33,164 chars — 53% of the serialized total. The biggest tools are big because of nested parameter structure, not prose: ReconciliationPer-tool figures are each schema serialised alone, and they close against the list total exactly — Test outputOne follow-up, not part of this PRThe assertion prints a total and nothing else, so whoever hits it next has to rebuild the attribution above from scratch. Having the failure print the ten largest tools and the delta against the last recorded total would turn that into reading the CI log. Worth a small PR after the release. |
Four comments ran to four and five lines where two or three carry the same constraint. No behaviour change. Also record, above `_CHAR_BUDGET` where a resolver will actually see it, that a merge conflict on that constant resolves to the higher value: #14424 and #14455 are bumping the same line concurrently, and each branch's CI only measures its own delta against dev, so taking the incoming side lowers a ceiling that has to cover every in-flight PR at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #14455 +/- ##
==========================================
- Coverage 81.48% 81.47% -0.01%
==========================================
Files 3553 3553
Lines 265706 265704 -2
Branches 24618 24618
==========================================
- Hits 216514 216491 -23
+ Misses 43830 43765 -65
- Partials 5362 5448 +86
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
1 participant
Why / What / How
Raises
_CHAR_BUDGETintool_schema_test.pyfrom 61,000 to 65,000 characters, and replaces the stale comment above it.This widens a gate; it does not fix a problem. The tool-schema budget is a drift alarm — it serialises every schema in
TOOL_REGISTRYand fails if the total passes a threshold — and nothing about 61,000 is physically meaningful. What it caught is real drift, and the honest response is a deliberately sized new threshold rather than a claim that anything is now smaller.devalone measures 60,984 of 61,000 — sixteen characters of headroom, against a comment claiming ~1,400. That is why #14434 was ejected from the merge queue at 22:34Z: it is green on its own head, green ondev, and the merged combination measures 61,037. It adds 53 characters and had the bad luck to be first.This is accumulation, not one bad PR. The last bump (#14099, 2026-08-23) granted 1,376 characters of headroom, and the comment recording that was accurate when written — I measured
devat that commit at 59,624. Seventeen days later it is gone, spread across nine tools with no single contribution over 411 characters:list_teamraise_expertask_questionhire_expertlist_workspace_filesread_workspace_filedelegate_to_expertwrite_workspace_filerun_sub_sessionstore_skillEvery one of those authors was reading a comment that promised 1,400 characters of room and adding something that plainly fitted. A wrong comment next to the constant is most of how this reached a release cut unnoticed, so it is corrected in the same diff.
Why 65,000 and not 63,000
The instinct is to pick the smaller number, and one of the ten release PRs (#14424) already proposes 63,000. 63,000 leaves 253 characters over the final state — less than one new parameter description, and about three days at the drift rate measured above (~80 chars/day). The next tool anyone adds ejects a PR from the merge queue again.
65,000 leaves 2,253 characters, sized against what concurrent in-flight PRs add in aggregate rather than against whatever sits on
devtoday. That is the number that matters, because each branch's CI only ever sees its own delta — a global counter can only fail once the branches are combined, which is the merge queue, which is the most expensive place to discover it. The ten v0.7.5 PRs add 1,763 characters between them and every one of them is individually green. Headroom smaller than that aggregate guarantees another ejection. 2,253 also buys roughly 28 days at the observed drift rate, against the 17 days the last bump bought.It remains a real ceiling: 3.5% above today's merged total, and any single PR adding a substantial tool without thinking still trips it.
#11220 is a twelfth PR, and it sharpens the ordering question
#11220 is not in the release set I measured, but it touches two registry tools (
manage_presets,setup_agent_webhook_trigger) and sets the budget to 62,000. Measured: its head merged intodevis 61,162, a net +178 — it addstrigger_configandconstant_inputson top of a trim to the same two tools. All ten release PRs plus #11220 measures 62,925, which leaves 2,075 against this PR's 65,000. So it does not change the recommendation.It does change what the smaller ceilings are worth. That same end state leaves 75 characters under 63,000, and exceeds 62,000 by 925. Neither of the other two proposed values survives the set of PRs currently in flight, whichever order they land in — and each is carried by a PR whose own CI is green, because no branch can see the sum. Landing this one first is what makes the order stop mattering.
Why not trim instead
Trimming is a legitimate answer and it is not available tonight at this size. The merged set is 1,747 characters over the current budget, against copy that twenty-one separate bump comments each argue is load-bearing, with no test that would notice if the model behaved worse after a cut.
Two measurements bound how much a trim could yield. Only 53% of the serialized total is description text at all — 33,164 of 62,747 characters; the rest is JSON structure: keys, braces, type names, parameter names, enum values. And the clearest genuine redundancy in the whole registry is
connect_integration, which states its supported-providers rule three times ("ONLY call this tool for one of the supported providers listed above — do NOT call it for Google, Gmail, Slack…", "Double-check that the provider is in the supported list above before calling.", "Theproviderparameter must match what the failing CLI/API actually needs."). Collapsing those to one recovers about 230 characters. Closing 1,747 would need eight such wins, and afterconnect_integrationthere are not eight —schedule_followup's 850-character description, the next largest, spends every sentence on a distinct rule.A trim pass is worth doing on its own merits, with someone watching model behaviour. It is not a release-night lever.
Changes 🏗️
_CHAR_BUDGET: 61,000 → 65,000 inautogpt_platform/backend/backend/copilot/tools/tool_schema_test.py.No product code changes; the diff is one file and eight lines.
Verified
I measured every one of the ten v0.7.5 PRs merged into
devindividually, and all ten merged together, then ran the gate against the combined tree.The measurement is validated two ways before any number below was trusted. My script's total is byte-identical to the test function's own assertion message on
dev(60,984), and merging #14434 intodevlocally reproduces the CI failure exactly — 61,037, matching run 34284792185. Local and CI agree, so the note in the file about env-flagged registrations pushing CI higher no longer applies.Executed on this branch:
backend/copilot/tools/tool_schema_test.py(217 passed),backend/util/architecture_test.py(3 passed),backend/blocks/test/test_block.py(1,647 passed, 84 skipped).Executed on this branch with all ten release PRs merged in locally:
tool_schema_test.py, 223 passed, registry measuring 62,747 across 73 tools — the new budget holds against the actual end state, which is the claim this PR rests on. #14424 conflicts on the_CHAR_BUDGETline in that merge, as it must, since it changes the same line to 63,000; that conflict is the intended outcome and makes the newer number visible to whoever resolves it, rather than silently regressing.Reasoned about but not executed: nothing bearing on the number. The gate is a pure function of the registry.
Full measurement tables, per-PR and per-tool, are in a comment below.
Agents and large language models used
Claude Code with Claude Opus 5
Checklist 📋
For code changes:
tool_schema_test.pypasses on this branch (217 passed)tool_schema_test.pypasses on this branch with all ten v0.7.5 PRs merged in (223 passed, 62,747 chars vs the 65,000 budget)architecture_test.pyandblocks/test/test_block.pypass, as they scan the whole tree and no diff points at themTool schemas use 60984 chars … exceeding budget of 1 chars