Epic: Token Rate Limiting

[shaneutt]

Overview

Rate limiting based on token consumption rather than request count. Requires token counts from the Token Counting filter (#20) in the filter chain.

Algorithm

Token bucket (preferred for burstiness tolerance) with configurable refill rate and burst capacity. Independent buckets per token type: input, output, and total. Token bucket allows short bursts while enforcing sustained rate limits.

Bucket Keys

Rate limit buckets keyed by: per-IP, per-header (API key, tenant ID), per-model, or composite keys combining multiple dimensions.

Hierarchical Quotas

Cascading limits at org > team > user levels. Each level enforced independently. A user within their personal quota can still be blocked by an exhausted org-level bucket.

Response Format

• 429 status code when token quota exceeded
• Retry-After header calculated from token consumption rate and bucket refill
• X-RateLimit-Limit-Tokens: configured token limit
• X-RateLimit-Remaining-Tokens: tokens remaining in current window
• X-RateLimit-Reset: time until bucket refills

Distributed Counters

• In-memory counters for single-instance deployments
• Redis/Valkey backend for multi-instance consistency via KV store registry

Filter Design

Separate token_rate_limit filter, distinct from the existing request-count rate_limit filter. Depends on token counting filter earlier in the chain.