fix(demos): harden the OTel benchmark demo after review - #24
Draft
Ladas wants to merge 1 commit into
Draft
Conversation
Ladas
marked this pull request as draft
September 4, 2026 09:18
Ladas
force-pushed
the
feat/otel-benchmark-followup
branch
from
September 4, 2026 13:50
aa473b9 to
0ec8139
Compare
Follow-ups to praxis-proxy#13, found by a review pass after it merged. Each one is a defect with an observable consequence; nothing here changes what the benchmark measures. Broken or dead: - The `ai-extended` scenario referenced two config files that do not exist, so selecting it aborted on the first `kubectl create` under `set -euo pipefail`, and report.sh had no branch for its result prefix. - The per-run `kubectl top pod` snapshot always failed into `|| true`, because no stack installs metrics-server and KIND does not ship it. It wrote nine empty `*-resources.txt` files per 3x3 run; the dashboards already plot CPU and memory from cAdvisor. - `export BRANCH=$(...)` in report.sh masked the command's exit status (SC2155), which shellcheck never saw because the lint target only covered `hack/` and `.hooks/`. - Two comments described a `[patch.crates-io]` table that the 0.5.4 bump had already removed. - The prerequisites told you to install forge from a feature branch. praxis-proxy/forge#16 has merged, so `extraPortMappings` is in main. Observability of the trace pipeline: - A `memory_limiter` now runs first in the collector pipeline. The container is capped at 512Mi and nothing shed load before it, so under exporter backpressure the queue grew until the kernel killed the collector and every buffered span went with it. Refusals land in `otelcol_processor_refused_spans`, charted beside the export failures. - The Span Export Failures panel filters to `exporter="otlp/tempo"`. The pipeline also has a `debug` exporter, so the unfiltered sum counted every span twice and a debug-exporter hiccup read as a Tempo failure. Supply chain and build: - praxis-ai is pinned to the commit tagged v0.3.0 rather than to the tag, which is what deny.toml's comment already claimed. A tag can be force-moved upstream; a commit cannot. - `make container FEATURES=otel` plumbs the build-arg through, so the repo's own tooling can build the image the demo READMEs ask for instead of the hand-rolled `docker build` lines they carry today. - A cargo git cache mount alongside the registry one: the ai dependency is a git source and was re-cloned on every image build. Signed-off-by: Ladislav Smola <lsmola@redhat.com>
Ladas
force-pushed
the
feat/otel-benchmark-followup
branch
from
September 7, 2026 10:35
0ec8139 to
a2ac800
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
#13 merged at the state that was pushed; a review pass afterwards found the
issues below. Each is small, each is verified on a live cluster, and none of
them changes what the benchmark measures — they change whether what it
reports can be trusted, and whether the demo still works when a second KIND
cluster is running beside it.
Correctness
forgeexec steps now name the kubectl context. forge scopesmanifest,helmandwaitsteps to the cluster but runsexecwith theambient context (
execute_exectakes nokube_context, unlike itssiblings). With a second KIND cluster up,
kubectl create namespace oteland the dashboard ConfigMaps land on whichever cluster was created last,
while the properly scoped steps that follow target the right one.
0.1.ai-otel-full.yamlhad it;otel-full.yamldid not, and the core scenario runs at 2000 RPS — thelarger of the two firehoses. Unsampled is not the shape the overhead
question is about.
memory_limiterruns first in the collector pipeline. The containeris capped at 512Mi and nothing shed load before it: under exporter
backpressure the queue grew until the kernel killed the collector and every
buffered span went with it. Refusals now land in
otelcol_processor_refused_spans, charted beside the export failures.Verified: Memory limiter configured in the collector log, pod stable at 0
restarts, counter present in Prometheus.
exporter="otlp/tempo". Thepipeline also has a
debugexporter, so the unfilteredsum()countedevery span twice and a debug-exporter hiccup read as a Tempo failure. Both
exporters confirmed reporting on the live cluster.
praxis-aiis pinned to the commit taggedv0.3.0, not to the tag. Atag can be force-moved upstream; a commit cannot. The exactness lived only
in
Cargo.lockbefore this.Removals
ai-extendedscenario referencedai-extended-baseline.yamlandai-extended-otel-full.yaml, which do not exist, so selecting it aborted onthe first
kubectl createunderset -euo pipefail.report.shhad nobranch for its result prefix either.
kubectl top podsnapshot. No stack installsmetrics-server and KIND does not ship it, so the call always failed into
|| trueand wrote nine empty*-resources.txtfiles across a 3×3 run.The dashboards already plot CPU and memory from cAdvisor.
[patch.crates-io]table that the 0.5.4 bumpremoved — one in
Cargo.toml, one indeny.toml, both still explaining apatch to praxis
main"for otel features not yet in v0.5.3".Tooling
make container FEATURES=otelplumbs the build-arg through, so therepo's own tooling can build the image the demos ask for. Both demo READMEs
hand-rolled
docker build --build-arglines because it could not.is a git source, so it was re-cloned on every build.
shellcheckcoversdemos/*/scripts/*.shinmake lintand CI, whichit never did — the glob was
hack/*.sh .hooks/pre-commit. The first runfound a masked exit status in
report.sh(SC2155), fixed here.Test plan
make lint,actionlint,shellcheck,taplo,cargo check --locked -p praxis-experimental-server --features otelall cleanmarkdownlintclean against a clean checkout of this branchpraxis-forge config validateaccepts the editedforge.yamlconfigured,
otel-collectorPrometheus targetup