feat: tool spend as a first-class primitive with reserve/settle parit…#38
Conversation
…y (py 0.5.0 / js 0.4.0).
|
Warning Review limit reached
Next review available in: 38 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughFloe Guard now enforces a shared ceiling across LLM tokens and paid tools in Python and JavaScript, adding reservation/settlement APIs, per-tool accounting, unified ledger events, validation, concurrency tests, documentation, and an example. ChangesTool budget enforcement
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant BudgetGuard
participant PaidTool
participant SpendLedger
Agent->>BudgetGuard: reserve tool estimate
BudgetGuard-->>Agent: reservation or BudgetExceeded
Agent->>PaidTool: execute paid call
PaidTool-->>Agent: actual cost
Agent->>BudgetGuard: settle tool cost
BudgetGuard->>SpendLedger: record tool event and update totals
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/floe_guard/guard.py`:
- Around line 375-391: Validate tool estimates in reserve_tool and
js/src/guard.ts before delegating to reserve: reject None/undefined, non-finite,
and negative values, while preserving valid exact-cost reservations. Apply the
corresponding validation at both listed sites: src/floe_guard/guard.py lines
375-391 and js/src/guard.ts lines 333-350.
- Around line 393-423: The numeric reserved handle in settle_tool can release
unrelated in-flight reservations and be reused; replace numeric reservation
tracking with opaque, single-use reservation IDs registered when reservations
are created, and atomically validate and consume the matching ID during
settle_tool or release in src/floe_guard/guard.py lines 393-423. Implement the
same single-use reservation-token contract in js/src/guard.ts lines 365-380 for
parity.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 3e2227bf-b63a-4b3c-b4af-1c25987b8bc6
⛔ Files ignored due to path filters (1)
js/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (11)
CHANGELOG.mdREADME.mdexamples/tool_budget.pyjs/README.mdjs/package.jsonjs/src/guard.tsjs/test/tool-costs.test.tspyproject.tomlsrc/floe_guard/__init__.pysrc/floe_guard/guard.pytests/test_tool_costs.py
Signed-off-by: shivamfloe <shivam@floelabs.xyz>
There was a problem hiding this comment.
Pull request overview
Adds “tool spend” as a first-class part of Floe Guard’s shared USD ceiling, so paid API/tool calls can be reserved/settled like LLM calls and attributed per tool name.
Changes:
- Introduces tool reservation + settlement APIs (
reserve_tool/settle_toolandreserveTool/settleTool) and per-tool totals (tool_costs/toolCosts) under the same ceiling as token spend. - Updates accounting/logging so tool spend lands in the spend ledger and can participate in “stop-before” enforcement behavior.
- Adds cross-language tests plus docs/examples and bumps package versions + changelog.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tool_costs.py | New Python tests covering reserve/settle tool flows, attribution, and mixed concurrency. |
| src/floe_guard/guard.py | Implements tool reservations/settlements, tool attribution totals, and related docs. |
| src/floe_guard/init.py | Updates package messaging and bumps Python __version__. |
| README.md | Documents tool spend under the shared ceiling and the new APIs. |
| pyproject.toml | Bumps Python package version. |
| js/test/tool-costs.test.ts | New TS tests for tool spend APIs, attribution, and mixed interleaving. |
| js/src/guard.ts | Implements TS tool reservation/settlement + per-tool totals and spend-log updates. |
| js/README.md | Documents tool spend under the shared ceiling for the TS package. |
| js/package.json | Bumps JS package version. |
| js/package-lock.json | Updates lockfile version metadata to match the bumped package version. |
| examples/tool_budget.py | New example demonstrating reserve/settle + record_tool + attribution. |
| CHANGELOG.md | Adds release notes for the tool-spend feature and behavior changes. |
Files not reviewed (1)
- js/package-lock.json: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
a few comments to check @shivamfloe |
Signed-off-by: shivamfloe <shivam@floelabs.xyz>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
README.md (1)
11-14: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winClarify that paid-tool hard stops require pre-call reservation.
record_tool()is post-hoc and cannot block the call that already ran. Precise pre-call enforcement requiresreserve_tool(...)followed bysettle_tool(...)(or an adapter that performs that flow). The current intro may lead users to userecord_tool()alone while assuming the paid call is protected.Proposed wording
- next LLM or paid tool call + next LLM call or paid tool call when protected with reserve_tool()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 11 - 14, Clarify the README introduction that paid-tool hard stops require reserving budget before the call and settling afterward, using reserve_tool(...) and settle_tool(...) or an adapter that performs both; distinguish this from record_tool(), which only records completed calls and cannot prevent an already-started call.
♻️ Duplicate comments (1)
src/floe_guard/guard.py (1)
412-422: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftNumeric reservation handles are reusable and can release unrelated in-flight holds.
As previously flagged, the tool reservation APIs continue to use raw numbers as reservation handles. This allows an unrelated in-flight hold to be released, or a handle to be reused to decrement the reserved total multiple times, breaking the budget cap.
src/floe_guard/guard.py#L412-L422: Replace numeric reservation tracking with opaque, single-use reservation IDs forreserve_toolandsettle_tool.js/src/guard.ts#L350-L362: Implement the same single-use reservation-token contract here for parity.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/floe_guard/guard.py` around lines 412 - 422, The tool reservation APIs currently use numeric handles, allowing unrelated or reused handles to release holds multiple times. Update reserve_tool and settle_tool in src/floe_guard/guard.py (lines 412-422) to generate opaque, single-use reservation IDs, track each ID to its reservation, and reject unknown or already-settled IDs. Apply the same single-use reservation-token contract to reserve_tool and settle_tool in js/src/guard.ts (lines 350-362) for parity.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@README.md`:
- Around line 11-14: Clarify the README introduction that paid-tool hard stops
require reserving budget before the call and settling afterward, using
reserve_tool(...) and settle_tool(...) or an adapter that performs both;
distinguish this from record_tool(), which only records completed calls and
cannot prevent an already-started call.
---
Duplicate comments:
In `@src/floe_guard/guard.py`:
- Around line 412-422: The tool reservation APIs currently use numeric handles,
allowing unrelated or reused handles to release holds multiple times. Update
reserve_tool and settle_tool in src/floe_guard/guard.py (lines 412-422) to
generate opaque, single-use reservation IDs, track each ID to its reservation,
and reject unknown or already-settled IDs. Apply the same single-use
reservation-token contract to reserve_tool and settle_tool in js/src/guard.ts
(lines 350-362) for parity.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: e96a8f9f-a745-4cff-a28e-3869d514cc03
⛔ Files ignored due to path filters (1)
js/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (7)
README.mdjs/package.jsonjs/src/guard.tsjs/test/tool-costs.test.tssrc/floe_guard/__init__.pysrc/floe_guard/guard.pytests/test_tool_costs.py
settle and tests.
…y (py 0.5.0 / js 0.4.0).
Summary
New Features
tool_costs/toolCosts.Documentation
Release
Testing
How was this tested?
pytestpasses (Python changes)ruff check .andruff format .are clean (Python changes)npm testandnpm run typecheckpass injs/(TypeScript changes)src/floe_guard/cost_map.json==js/src/cost_map.json)Summary by CodeRabbit
New Features
Documentation
Tests
Chores