Consumer-Based Rate Limiting for LLM Providers

[Saadha123]

Overview

• Consumer-based rate limiting allow each GenAI application (consumer) to have its own independent rate limit counter, instead of all applications sharing a single counter against the provider.
• Currently, rate limits are applied at the provider level. If App A exhausts the token budget, App B gets blocked too. With consumer-based rate limiting, each app gets its own independent budget. The limit value is the same for all apps, but the counters are tracked separately per app.

---

How It Works

• A GenAI application is identified by x-wso2-application-id, which is set by the api-key-auth policy after successfully authenticating an API key.
• The rate limiters (token-based-ratelimit and llm-cost-based-ratelimit) use key extraction to decide what to count against. Currently the key is just routename.
• For consumer-based limiting, the key becomes routename + x-wso2-application-id.

---

Design Decision: Reuse Existing Policies with a Toggle

• Instead of creating new policies (consumer-token-based-ratelimit, consumer-llm-cost-based-ratelimit), we extend the existing token-based-ratelimit and llm-cost-based-ratelimit policies with a consumerBased boolean parameter.
• When a user configures both a backend limit and a consumer limit, the same policy appears twice in the deployment YAML. Once without consumerBased (backend, shared counter) and once with consumerBased: true (consumer, per-app counters). The gateway processes each as an independent instance.

Note: x-wso2-application-id is Not in SharedContext.Metadata

• The rate limiting metadata key extraction reads from SharedContext.Metadata. Currently, api-key-auth writes the application ID to two places after authentication:
  • SharedContext.AuthContext.Properties["ApplicationID"]
  • AnalyticsMetadata["x-wso2-application-id"]
• Neither of these is SharedContext.Metadata, so the rate limiter cannot find the application ID.
• We need to add a write to SharedContext.Metadata in api-key-auth as part of this change.

---

Changes Required

gateway-controllers

• api-key-auth/apikey.go - after a successful auth, write x-wso2-application-id into SharedContext.Metadata so downstream rate limiting policies can read it.

• token-based-ratelimit - add consumerBased boolean to the policy definition yaml, and in token_based_ratelimit.go switch the key extraction to include the application ID metadata when it's true.

• llm-cost-based-ratelimit -

  1. Same as above, add consumerBased to the policy definition yaml and update llm_cost_based_ratelimit.go accordingly.
  2. Fix a metadata key collision where both the backend and consumer instances stored their delegate reference under the same key llm_cost_delegate. Fix by using separate keys: llm_cost_delegate for backend and llm_cost_delegate_consumer for consumer.

• advanced-ratelimit/ratelimit.go - two fixes:

  1. Fallback key for missing app ID - add a fallback field to KeyComponent. When x-wso2-application-id is absent from metadata (e.g. no api-key-auth in the chain), the key used is routename:default. This prevents any accidental collision with the backend key (routename vs routename:default are always distinct).
  2. Metadata key collision between instances — fix a metadata key collision that surfaces when a backend and a consumer instance are both in the request chain:
  • advanced-ratelimit passes state between its request and response phases via hardcoded metadata key names (ratelimit:keys, ratelimit:result, etc.)
  • These names are constants ( not derived from key extraction config ) so when two instances run, the second one blindly overwrites the first's metadata entry
  • This is harmless when both instances have the same key extraction (they write the same value), but breaks when they differ. The backend writes routename and the consumer overwrites with routename:app-id, so the backend's response phase reads the consumer's key and deducts tokens from the wrong counter
  • Fix by adding an instanceID field to RateLimitPolicy( a short hash of the quota names and key extraction config) and routing all metadata reads and writes through p.metaKey(base) which appends the instance ID
  • This namespaces each instance's metadata slots without affecting the limiter objects or limiter cache keys, so existing limiter sharing between APIs is unaffected

api-platform

• platform-api/src/internal/service/llm_deployment.go - three things to fix here:

  1. addOrAppendPolicyPath currently merges all entries with the same policy name, so a backend and a consumer token-based-ratelimit would get collapsed into one. Need to make it scope-aware by checking whether consumerBased differs between entries.
  2. Fill in the consumer level rate limits - there's currently a TODO for request, token, and cost.
  3. Add a hasPolicy() helper to avoid adding llm-cost twice when both backend and consumer cost limits are configured. Without this we'd end up with two llm-cost entries in the chain.

• gateway/gateway-controller/default-policies/ - update both token-based-ratelimit.yaml and llm-cost-based-ratelimit.yaml which are copies of the gateway-controllers definitions.

apim-saas

• ServiceProviderRateLimitingTab.tsx - the Per Consumer card is currently disabled with a "Coming Soon" chip. Need to remove those.

---

Runtime Flow

Via the Rate Limit tab

Consumer limit only

1. User sets a consumer token limit of 10,000 tokens/hour in the Rate Limit tab
2. llm_deployment.go generates one token-based-ratelimit entry with consumerBased: true (per-app counter)
3. A request arrives with an API key
4. api-key-auth authenticates it and writes the application ID into SharedContext.Metadata
5. Consumer token-based-ratelimit checks the per-app counter (keyed by routename + application-id)
6. The request is blocked if the counter is exhausted

Both backend and consumer limits configured

1. User sets both a backend token limit and a consumer token limit in the Rate Limit tab
2. llm_deployment.go generates two token-based-ratelimit entries - one without consumerBased (shared counter) and one with consumerBased: true (per-app counter)
3. A request arrives with an API key
4. api-key-auth authenticates it and writes the application ID into SharedContext.Metadata
5. Backend token-based-ratelimit checks the shared counter (keyed by routename)
6. Consumer token-based-ratelimit checks the per-app counter (keyed by routename + application-id)
7. The request is blocked if either counter is exhausted

Via the Guardrails tab

The Guardrails tab lets users configure policies directly, bypassing llm_deployment.go. The consumerBased parameter is set explicitly in the policy params rather than being inferred from the rate limit configuration.

Consumer cost limit only

1. User adds an llm-cost-based-ratelimit policy in the Guardrails tab with consumerBased: true and a budget (e.g. $0.1/hour)
2. The policy is stored directly in the provider's policy list with consumerBased: true in params
3. A request arrives with an API key
4. api-key-auth authenticates it and writes the application ID into SharedContext.Metadata
5. llm-cost reads the response and writes the request cost into SharedContext.Metadata under x-llm-cost
6. Consumer llm-cost-based-ratelimit checks the per-app cost counter (keyed by routename + application-id)
7. The request is blocked on the next call once the budget is exhausted

Both backend and consumer cost limits configured

1. User adds two llm-cost-based-ratelimit entries in the Guardrails tab — one without consumerBased (shared budget) and one with consumerBased: true (per-app budget)
2. Both policies are stored in the provider's policy list
3. A request arrives with an API key
4. api-key-auth authenticates it and writes the application ID into SharedContext.Metadata
5. llm-cost reads the response cost into SharedContext.Metadata
6. Backend llm-cost-based-ratelimit checks the shared cost counter (keyed by routename)
7. Consumer llm-cost-based-ratelimit checks the per-app cost counter (keyed by routename + application-id)
8. The request is blocked if either budget is exhausted

Requests where app ID is unavailable

1. User adds an llm-cost-based-ratelimit policy with consumerBased: true
2. At runtime, x-wso2-application-id is missing/empty in SharedContext.Metadata (for example, no api-key-auth in the chain, or any flow where app ID is not populated)
3. The fallback key "default" is used, so the effective key is routename:default
4. All requests that do not have an app ID share this same "default" cost bucket
5. Once the "default" budget is exhausted, further requests without an app ID are blocked

[malinthaprasan]

x-wso2-application-id becomes "default" for unauthenticated calls only? It will happen for both unauthenticated and any API calls with API Keys which are not bound to any applications?

[Saadha123]

default is used whenever x-wso2-application-id is missing/empty in SharedContext.Metadata (not only for unauthenticated requests).
Updated the discussion stating this clearly.