feat(demos): ai-gateway demo with tiered token budgets and span-derived dashboards

[Ladas]

Base

#13 has merged, so this branch is rebased directly onto main and its diff is
exactly its own 16 files — everything under demos/ai-gateway/ plus enabling
the token-rate-limit-filter feature on the experimental server.

Note: pr-size-check counts 1,511 added lines against a 750 limit, because
its exemption list covers examples/, tests/, benchmarks/ and docs/
but not demos/. #13 carried skip/pr-conventions for the same reason.
Adding demos/** to that list would be the better fix.

Summary

demos/ai-gateway/ — the experimental gateway in front of a real local
model (Ollama), enforcing per-tier token budgets, with dashboards built
from both Prometheus counters and Tempo span metrics. Runs on its own KIND
cluster with ports offset from otel-benchmark, so both demos can be up at
once.

Also enables the token-rate-limit-filter cargo feature on
praxis-experimental-server. This image exists to expose feature-gated
upstream work, and previously enabled only one of the four experimental AI
filters.

Demo	Question it answers
otel-benchmark (#13)	What does OTel tracing cost?
ai-gateway (this)	What does a real AI gateway do, and which filter costs what?

The rate limit is real, not rigged

Two tiers matched on a header, with reserved_tokens: 200 — a realistic
estimate for a short chat completion, not a number tuned to trip on request
six:

Tier	Match	Budget	~requests/min
premium	X-Tier: premium	20,000 tokens/min	~100
free	catch-all	5,000 tokens/min	~25

scripts/rate-limit-demo.sh sends the same 40-request burst as each tier.
Captured from a real run:

  40 requests as tier: free
  200 OK            25
  429 rate limited  15
  -> 38% of requests hit the free tier's token budget

  40 requests as tier: premium
  200 OK            40
  429 rate limited   0

25 admitted is exactly 5,000 / 200. Identical load, different outcome.

Two telemetry planes, and why it matters

This is the design constraint the dashboards are built around.

Plane	Coverage	Owns
Prometheus (praxis_*)	100% of requests, sampling-independent	counts, rates, budget state
Span metrics (traces_spanmetrics_*)	the sampled fraction only	latency distributions, per-filter breakdown

Random sampling is unbiased for percentiles, so span-derived latency is
trustworthy even at sampling_rate: 0.1. Counts are not — they are low by
exactly the sampling factor.

This demo sets sampling_rate: 1.0 explicitly. It pushes a handful of requests
per minute through a local model, and at 0.1 the span panels would be built
from a couple of traces. Leaving the field unset would have reached the same
sampler by accident (the default is ParentBased(AlwaysOn)) while the README
talked about a sampled fraction, so the choice is now written down with its
reason. ../otel-benchmark runs at 0.1 under ~500 RPS, which is where the two
planes visibly diverge.

So the "Weighted Filter Cost" panel takes its quantile from span metrics and
its request rate from praxis_http_requests_total. Multiplying two sampled
series would under-report 10x while looking entirely plausible — the first
version of this dashboard did exactly that.

A Sampling Ratio Cross-Check panel divides one plane by the other. It
should approximate the configured sampling rate — ~1.0 here, measured at 1.06
over a controlled 10-request window; a drop means spans are being
lost rather than sampled, and it plots otelcol_exporter_send_failed_spans
alongside so the two are distinguishable.

Dashboards

Dashboard	Source
Praxis AI Gateway Overview	both planes, explicitly labelled per row
Praxis Token Budget & Rate Limiting	Prometheus only
Praxis Filter Latency (from spans)	span metrics only
Praxis OTel Traces	Tempo

Per-filter latency is the panel a trace waterfall cannot give you: praxis
emits one span per filter per phase, so span_name becomes the breakdown key
across all requests rather than one at a time.

Four traps that produce plausible-but-wrong results, and what the demo does instead

1. Trap: declaring token_rate_limit after token_count. Response
   hooks run in reverse declared order, and it reconciles against the
   token.total metadata token_count writes. Declared after, it charges
   every request at its flat estimate — the metrics look reasonable and are
   wrong. The demo declares it before. Verified: correct ordering gives estimated 300 / actual 279 / refunded 22 / overage 1; wrong ordering gives actual == estimated.

2. Trap: OLLAMA_HOST=0.0.0.0. It is the obvious first guess for letting
   a container reach the host, and it publishes the model server to the
   whole local network. It is also unnecessary: a KIND pod reaches
   host.docker.internal, which the container runtime proxies from the host
   side, with Ollama on plain loopback. The demo leaves Ollama on its
   default 127.0.0.1 binding — verified from inside a pod.

3. Trap: a slow model makes the limit unreachable. The limiter decides at
   admission, before the upstream call, so the model is irrelevant to what is
   demonstrated — but the burst must land inside the sliding window. qwen3.8:27b takes 74-78s per request under concurrency (1.7s
   idle), so 40 requests span ~12 minutes and the budget ages out faster than
   it is consumed. qwen3.5:0.8b answers in ~0.3s warm. The rate-limit
   step uses the small model; the quality step (7) keeps qwen3.8:27b.

4. Trap: reading budget / reserved_tokens as the effective limit. It is
   not, because reservations reconcile: measured over 110 requests,
   estimated 22,000 / actual 7,511 / refunded 14,489, or ~68 real tokens
   each. The refund lands quickly, so the sustained limit follows real usage
   (~70 requests/min on a 5,000-token free tier), not the ~25 that
   5,000 / 200 suggests. What the estimate bounds is concurrency: every
   in-flight request holds 200 tokens, so 25 simultaneous requests exhaust the
   budget on reservations alone while the same 25 spread out cost ~1,700
   tokens. The demo states the measured limit, drives enough traffic to
   actually reach it, and charts estimated-vs-actual-vs-refunded so the gap is
   visible.

   This one bit the PR itself. The walkthrough previously reported
   25 admitted / 15 denied from 40 requests — a number produced by the
   broken wait -n throttle firing all 40 at once, so the denials came from
   reservation pressure, not from the budget. With the throttle fixed, 40
   requests never reach the limit. The demo now sends 100 and the README
   shows the re-measured 53 / 47.

Caveats

This is a demo configuration, and two insecure_options say so:

• allow_public_admin: true binds the admin listener to the pod IP so
  Prometheus can scrape /metrics. That listener also serves
  /api/log-level (PUT/DELETE) and /api/kv, so every pod in the cluster
  can change the gateway's log level. Fine on a throwaway KIND cluster; a
  shared cluster needs a NetworkPolicy on :9901. Metrics on a port separate
  from the mutating admin API is the proper fix and is worth an upstream
  issue — praxis does not offer it today.

• allow_private_endpoints: true disables SSRF / DNS-rebinding hardening so
  the gateway can reach host.docker.internal. Drop it for a public upstream.

• token_rate_limit is experimental and its parent proposal is not
  accepted (ai#121, open questions in ai#796). The config surface may
  change.

• It does not authenticate. A header-matched rule trusts whatever set the
  header, so a real deployment needs an auth filter populating X-Tier and
  stripping client-supplied copies (grid#101).

• The memory backend is per-replica; backend.kind: valkey shares one
  budget across replicas.

Review fixes in the latest push

Six of these produced dashboards or scripts that looked fine and were not:

• Every span-derived panel queried the wrong label. Tempo's span-metrics
  processor emits the service dimension as service, not service_name, so
  nine panels across two dashboards matched zero series and showed their
  "no span metrics yet" placeholder forever. Verified against the live series:
  service, span_kind, span_name, status_code. Fixed; 22 span names now
  resolve.
• The collector's self-telemetry port was never opened. forge.yaml
  scraped otel-collector.otel.svc:8888 with a comment explaining that
  without it span loss is silent — but this demo's collector manifest was
  copied from before that fix, so the target was down from the start. Now up.
• wait -n silently defeated the concurrency cap. It is bash 4.3+, macOS
  ships 3.2, and || true swallowed the error, so all 40 requests fired at
  once. Replaced with xargs -P.
• The README pulled only qwen3.8:27b, while the rate-limit script
  defaults to qwen3.5:0.8b; Ollama's HTTP API does not auto-pull, so every
  admitted request would have 404'd.
• A memory_limiter and the exporter="otlp/tempo" filter from feat: add OTel observability benchmark demo #13, plus
  feat: add OTel observability benchmark demo #13's Span Export Failures panel, which this demo's copy predated.
• exec steps name the kubectl context; the praxis.yaml header comment no
  longer points at benchmark scripts this demo does not have; the README gained
  the forge install command; .gitignore uses one demos/*/.forge/ glob.

Test plan

• praxis-forge up creates ai-gw-local alongside otel-bench-local
• KIND pod reaches host Ollama via host.docker.internal
• qwen3.8:27b answers through the gateway
• Tiered rate limit: 53/47 free, 100/0 premium — matches the README, re-measured after the throttle fix
• Both planes reporting: denied{rule="free"} 15, 22 span-metric series
• cargo check --locked, make lint, make test, taplo, markdownlint
• rebased onto main after feat: add OTel observability benchmark demo #13 merged; 16 files, 0 overlap

praxis-proxy/forge#16 (extraPortMappings) and #13 have both merged, so this
branch has no open dependencies — forge installs from main.