Track and enforce cost budgets for LLM inference traffic based on token consumption.
Scope:
- Input token estimation before forwarding (pre-request cost check)
- Per-client/per-route token budgets with configurable windows
- Cost attribution: map token usage to user, session, model, and endpoint
- Budget exceeded behavior: reject with 429 or queue
- Cost data exposed to access logging and metrics
Track and enforce cost budgets for LLM inference traffic based on token consumption.
Scope: