Skip to content

feat(filter): add external metering filter for usage reporting and balance checks #577

Description

@noyitz

Description

Add an external_metering filter that integrates with an external metering/billing service for:

  1. Pre-request balance check: Queries the metering service to verify the tenant has available token budget before forwarding the request to the LLM provider. Rejects with 429 if budget is exhausted.
  2. Post-response usage reporting: After the response completes, reads token counts from filter_metadata (set by the existing token_count filter) and sends a CloudEvents-formatted usage report to the metering service.

Key design points

  • Single filter (not two like the Go/ext_proc predecessor) — leverages the existing token_count filter for usage extraction from all providers (OpenAI, Anthropic, Google, Bedrock, Azure)
  • Identity header capture: Captures configurable identity headers (default prefix: x-tenant-) containing username, group, subscription. Strips them before forwarding upstream.
  • Fail-open by default: If the metering service is unavailable, requests proceed normally (configurable to fail-closed)
  • Fire-and-forget usage reporting: Usage events are sent asynchronously via tokio::spawn — never blocks the response path
  • CloudEvents format: Standard CloudEvents 1.0 envelope with inference.tokens.used and inference.request.error event types
  • Uses CalloutClient from praxis-core for HTTP calls with circuit breaking and timeout

API contract

Balance check (pre-request):

GET {metering_url}/api/v1/customers/{username}/entitlements/{feature_key}/value?model={model}
Response: { "hasAccess": true, "balance": 9000.0, "usage": 1000.0, "overage": 0.0 }

Usage report (post-response):

POST {metering_url}/api/v1/events
Body: CloudEvent JSON with token usage data

Configuration

filter: external_metering
metering_url: "http://metering-service:8080"
timeout_seconds: 5
feature_key: "inference-tokens"
source: "ai-gateway"
fail_open: true
identity_header_prefix: "x-tenant-"

Pipeline ordering

filters:
  - filter: external_metering   # request: first, response: LAST
  - filter: token_count          # response: writes metadata before metering reads it
  - filter: router
  - filter: load_balancer

Motivation

Production AI gateway deployments need per-tenant token quota enforcement and usage tracking for billing/chargeback. Praxis has token extraction (token_count) but no way to report usage to an external metering or billing system.

This capability exists in a Go-based ext_proc predecessor and needs to be ported to native Praxis for the data plane migration.

Alternatives Considered

  1. Two separate filters (metering + headers-guard): Rejected because the identity headers are only consumed by metering, and combining them reduces pipeline complexity.
  2. External CloudEvents crate: Rejected — the CloudEvent envelope is simple enough to serialize inline with serde_json::json!().
  3. Synchronous usage reporting: Rejected — would add latency to every response. Fire-and-forget via tokio::spawn is appropriate since usage report failures should never affect the client response.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions