feat: add OTel observability benchmark demo

[Ladas]

Summary

A KIND-based observability stack for measuring what OTel tracing costs the
Praxis experimental AI gateway, plus the otel cargo feature on
praxis-experimental-server and a FEATURES build-arg so the image can be
built with or without it.

Two scenarios, each running three configurations against the same filter
chain so only tracing varies:

Run	Cargo features	Trace sampling	What it isolates
A Baseline	none	n/a	the chain with no tracing compiled in
B OTel noop	otel	AlwaysOn	cost of creating spans, nothing exported
C OTel full	otel	0.1	creation plus export to the collector

• AI scenario: 500 RPS, POST /v1/chat/completions against llm-d
  inference-sim, 8 filters, 23 spans per request
• Core scenario: 2000 RPS, GET / against Fortio echo

Five Grafana dashboards, Tempo traces, Loki logs, Prometheus metrics.

Results

Clean run: fresh cluster, 8 vCPUs, nothing else on the machine.

Config	P50 (us)	P99 (us)	P99 range (us)	P50 delta
AI Baseline	842	1464	1464-1497	--
AI OTel (noop)	847	1418	1391-1635	+0.6%
AI OTel (full)	843	1764	1420-46119	+0.1%

P50 overhead is +0.1% to +0.6%, which is at or below what this setup can
resolve. Per-run P50 spread within a config is 27-47us, comparable to the
between-config difference. The honest reading is "no measurable median
overhead", not a specific figure. Export is batched and off the request path;
what remains on the hot path is span creation.

P99 is not resolvable at three runs. With one outlier the median becomes
the larger of the two clean values, which is exactly how OTel (full) reports
1764us and a +20.5% delta. The P99 range column exists so this is visible
rather than hidden. P50 is trustworthy; the tail number is not.

Tail-latency signal worth following up. Extreme outliers (90ms and 208ms
max) appeared in both otel-compiled builds across runs, but never in the
baseline (max 13ms). It shows up with no endpoint configured, so it is not the
export path -- something in the otel build itself. Two events across three
runs, so this is a flag for investigation, not a finding.

Methodology, and two bugs the results exposed

Runs are interleaved (A B C, A B C, ...), not grouped. Grouped ordering
aliases drift over time onto the config comparison and whichever config runs
first absorbs it: in an earlier grouped run the baseline's three measurements
fell monotonically 815 -> 775 -> 735us purely from running first, which biased
it slow and made OTel look free -- at one point reporting OTel as faster
than baseline. Every config now sees the same distribution of machine states,
each with its own rollout and warmup.

The report medians rather than means. A single cold run had previously
dragged the mean P99 to a reported +546% regression that did not exist.

98.7% of spans were being silently dropped. Tempo's OTLP gRPC receiver
inherits grpc-go's 4 MiB default message size. The collector's 8192-span
batches exceeded it, Tempo answered ResourceExhausted, and because that code
is non-retryable without server-supplied RetryInfo the collector discarded
entire batches as permanent errors -- 832,514 of 843,156 spans. Every
request still returned HTTP 200, so the only symptom was a suspiciously empty
Tempo.

Fixed at the root by raising the receiver limit, with the batch cap as defence
in depth (both are span-count caps, so neither alone can guarantee a byte
limit):

tempo:
  receivers:
    otlp:
      protocols:
        grpc:
          max_recv_msg_size_mib: 32

Verified after: 0 dropped batches, 0 permanent errors, and a controlled probe
measured exactly 23.0 spans per request exported.

And made detectable. The incident was invisible because nothing scraped
the collector's own counters. Its self-telemetry is now exposed and scraped,
with a "Span Export Failures (collector -> Tempo)" panel on
otelcol_exporter_send_failed_spans. The panel's series are filtered to
exporter="otlp/tempo": the pipeline also has a debug exporter, so an
unfiltered sum() counts every span twice and a debug-exporter hiccup would
read as a Tempo failure.

A memory_limiter runs first in the pipeline. The collector container is
capped at 512Mi. Under exporter backpressure the queue grew until the kernel
killed it and every buffered span went with it; the limiter refuses new spans
near the limit instead and counts them in otelcol_processor_refused_spans,
which is now a series on the same panel. Verified live: Memory limiter
configured in the collector log, pod stable, counter present.

Sampling. 500 RPS x 23 spans is ~11,500 spans/sec unsampled, which no real
deployment exports in full. Run C uses sampling_rate: 0.1 in both
scenarios -- the core config had been left unsampled, which at 2000 RPS is the
larger of the two firehoses. The sampler is parent-based, so it samples 10% of
traces and each sampled trace arrives complete -- fewer traces, not partial
ones.

Reports are self-describing. benchmark.sh records rate, duration, runs,
connections and per-config cargo features (read off the image's
io.praxis.build.features label rather than assumed), trace sampling and the
actual filter chain into scenario.env. report.sh renders it as a Scenario
table, so a report cannot drift from the configs that produced it.

Dependencies

Requires praxis-proxy/forge#16 (extraPortMappings, to expose KIND
NodePorts to the host). Once that merges, the README's install line drops
--branch feat/extra-port-mappings-v2.

Demo structure

demos/otel-benchmark/
├── forge.yaml          # Forge environment: 1 cluster, 7 stacks
├── README.md           # AI gateway benchmark, step by step
├── README-core.md      # Core proxy benchmark
├── configs/            # baseline/otel-full x core/AI
├── manifests/          # Praxis deployment + ServiceMonitor
├── scripts/            # benchmark.sh + report.sh (both scenario-parameterized)
└── stacks/             # mock backends + observability

Datasources are provisioned declaratively with fixed uids. They were
previously POSTed to Grafana's API, which meant a pod restart silently dropped
them and every panel rendered "No data".

Review fixes in the latest push

• forge.yaml's exec steps now name the kubectl context. forge scopes
  manifest/helm/wait steps to the cluster but runs exec with the
  ambient context, so with two KIND clusters up the namespace and dashboard
  ConfigMaps could land on whichever was created last.
• Removed the ai-extended benchmark scenario: it referenced configs that
  never existed, so selecting it aborted on the first kubectl create.
• Removed the per-run kubectl top pod snapshot. No stack installs
  metrics-server, so it always failed into || true and wrote nine empty
  *-resources.txt files; the dashboards already plot CPU and memory from
  cAdvisor.
• make container FEATURES=otel now plumbs the build-arg through, so the
  repo's own tooling can build the image the demos ask for instead of
  hand-rolled docker build lines. Added a cargo git cache mount alongside
  the registry one, since the ai dependency is a git source and was re-cloned
  every build.
• shellcheck in make lint/CI now covers demos/*/scripts/*.sh, which it
  never did; the first run found a masked exit status in report.sh.
• Dropped two stale comments in Cargo.toml and deny.toml describing a
  [patch.crates-io] table that the 0.5.4 bump removed.

Test plan

• praxis-forge up creates the cluster with all four host port mappings
• All 7 stacks deploy on a from-scratch cluster
• 23 spans per request visible in Tempo, verified against the live trace
• Dashboards populated (aggregation fixed: unaggregated selectors were
  rendering ~27 indistinguishable "P50"/"P99" series and half-zero stat tiles)
• Benchmark completes and the report renders with a Scenario table
• Zero span drops under full benchmark load
• make build, make lint, make test, taplo, markdownlint all clean

Known limitations

• praxis-ai-proxy is not published to crates.io, so it stays a git
  dependency, pinned to the commit tagged v0.3.0 rather than to the tag
  itself: a tag can be force-moved upstream and a commit cannot. The praxis
  core crates come from crates.io at 0.5.4.
• praxis-protocol is declared with the admin-api feature because
  praxis-ai-proxy uses AdminEndpointOptions without enabling it, and the
  praxis-main feature is on because the default path in ai v0.3.0 omits the
  log_level field 0.5.4 requires. Both are upstream gaps worth filing.
• MLflow was removed from this demo; it showed no data. Revisiting separately.
• Slow Traces (>100ms) and the two ai#92-blocked token panels are
  legitimately empty.

[github-actions]

Unsigned commits: 74990af. Please sign your commits.

[github-actions]

PR too large: 1500 lines added (limit: 750, excludes Cargo files, tests, docs, examples, and benchmarks). Please split into smaller PRs. Add skip/pr-conventions label to override.

[praxis-bot]

4 findings (1 Large, 3 Medium)

Overall: well-structured demo with good reproducibility design -- pinned chart versions, configurable benchmark parameters, multiple runs with warmup. The OTel collector pipeline, Tempo integration, and benchmark methodology are sound. The three-way comparison (baseline / noop / full) is the right approach for isolating overhead. A few issues to address.

[praxis-bot]

PR Review (re-review)

Summary: Previous findings addressed -- ARG FEATURES replaces the fragile sed approach, vegeta prerequisite check added, report table formatting fixed. One new issue found.

Severity	Count
Critical	0
Large	0
Medium	1

[praxis-bot]

PR Review (re-review)

Summary: All five previous findings addressed -- Containerfile.ai-otel removed in favor of unified ARG FEATURES build-arg, vegeta prerequisite check added, report table now uses Python (no trailing ||), REPORT_SCRIPT fixed. Two new issues.

Severity	Count
Critical	0
Large	0
Medium	2

[nerdalert]

LGTM 🎉