scenarios: full_host_otlp_cpu (full-host CPU via OTLP, standalone) - #178
scenarios: full_host_otlp_cpu (full-host CPU via OTLP, standalone)#178r1viollet wants to merge 1 commit into
Conversation
Full-host CPU profiling exercised through the analyzer's OTLP input path (the CPU counterpart to the pprof `full_host` scenario). Generic, reusable building blocks (kept out of the scenario): - tools/otlp-dump: a dependency-free OTLP/HTTP receiver that persists a profiler's OTLP export as .otlp files the analyzer reads natively. Reusable by any OTLP-emitting scenario. (Whether a sidecar dump is the capture mechanism we want long-term is an open project question.) - base_images/Dockerfile.fullhost-otlp -> prof-fullhost-otlp: the standalone full-host profiler image (registry.datadoghq.com/ddot-ebpf-dev:devtest-latest, the host-profiler as an OTel Collector distribution) with otlp_dump baked in. Self-contained: no agent, no hand-built binary. Moving devtest-latest tag is intentional, to surface drift. The scenario itself is thin (workload + config + start.sh): - Assertion: a load-independent CPU rate — the workload pins one core, so at the ~20 Hz sampler it accounts for ~20 samples/sec regardless of host load (value-matching-sum: 20, scale_by_duration: true), mirroring full_host's 1e9 ns/s. profile-type is "samples" (count), not "cpu-time". - Timing: the workload runs longer than the profiling window and the profiler is stopped while it is still busy, so no idle cool-down tail report is produced; only the first (start-up) report is partial, tolerated via allow_first_profile_failure. Config note: the image's Datadog-flavored otlp_http exporter requires a dd-api-key header even when pointed at the local sink (value unused). Validated end-to-end against the real image (base image + scenario built as the harness would): warm-up report tolerated, steady reports 0-3% error vs the 30% margin.
3a1b0c6 to
3ee20e7
Compare
| "profile-type": "samples", | ||
| "stack-content": [ | ||
| { | ||
| "regular_expression": ".*cpu_workload.*", |
There was a problem hiding this comment.
remote symbolication won't make this very interesting 😄
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3ee20e7633
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sleep 3 | ||
|
|
||
| echo "=== running cpu_workload for ${WORKLOAD_SECS}s (profiling for ${PROFILE_SECS}s) ===" | ||
| DD_SERVICE=cpu_workload_test timeout "${WORKLOAD_SECS}"s /app/cpu_workload & |
There was a problem hiding this comment.
The workload inherits EXECUTION_TIME_SEC, and workload.c uses that variable as its own duration, so it exits after PROFILE_SECS; the longer timeout "${WORKLOAD_SECS}"s is only an upper bound and cannot extend it. Because the script also sleeps exactly PROFILE_SECS before stopping the profiler, shutdown and the final flush race with—or follow—the workload's exit, potentially producing an idle tail that violates the 20-samples/s assertion. Pass WORKLOAD_SECS to the child or otherwise disable its shorter internal timer.
Useful? React with 👍 / 👎.
| { | ||
| "test_name": "full_host_otlp_cpu", | ||
| "note": "Full-host CPU profiling via the datadog-agent host-profiler, exported as OTLP and captured to .otlp by the otlp_dump sidecar. OTLP-vs-pprof difference: profile-type is 'samples' (a count, unit 'count'), NOT 'cpu-time' (nanoseconds). Like the pprof full_host scenario we assert a load-independent RATE: the workload pins one core, so at the eBPF sampler frequency (~20 Hz) it accounts for ~20 samples/sec regardless of what else runs on the host (value-matching-sum with scale_by_duration:true). Frames are mapping basenames (no symbol upload), so we match the workload binary by regex.", | ||
| "pprof-regex": ".*\\.otlp$", |
There was a problem hiding this comment.
not the best name 😄
| "scale_by_duration": true, | ||
| "stacks": [ | ||
| { | ||
| "profile-type": "samples", |
There was a problem hiding this comment.
samples is not a very convincing name
|
maybe I should create a CPU profile type with nanos, asserting on count is strange |
1 participant
What
Adds
full_host_otlp_cpu: full-host CPU profiling exercised through theanalyzer's OTLP input path (the CPU counterpart to the pprof
full_hostscenario). Depends on the OTLP support merged in #173.
Reusable building blocks (kept out of the scenario)
tools/otlp-dump— a generic, dependency-free OTLP/HTTP receiver thatpersists a profiler's OTLP export as
.otlpfiles the analyzer readsnatively. Reusable by any OTLP-emitting scenario. It exists because getting a
local-dump option upstream is hard; whether a sidecar dump is the capture
mechanism we want long-term is an open project question.
base_images/Dockerfile.fullhost-otlp(→prof-fullhost-otlp) — thestandalone full-host profiler image
(
registry.datadoghq.com/ddot-ebpf-dev:devtest-latest, thehost-profileras an OTel Collector distribution) with
otlp_dumpbaked in. Self-contained:no agent, no hand-built binary, no
binaries/prerequisite. Movingdevtest-latesttag is intentional, to surface drift.The scenario itself (thin: workload + config + start.sh)
so at the ~20 Hz sampler it accounts for ~20 samples/sec regardless of host
load (
value-matching-sum: 20,scale_by_duration: true), mirroringfull_host's1e9ns/s.profile-typeissamples(count), notcpu-time.profiler is stopped while it's still busy, so there's no idle cool-down tail
report; only the first (start-up) report is partial, tolerated via
allow_first_profile_failure.Validated
End-to-end against the real image, built the way the harness does (base image →
scenario,
--privileged --pid=host): warm-up report tolerated, steady-statereports 0–3% error vs the 30% margin. Schema + ruff clean.
Open questions (first cut)
otlp_dumpsidecar vs some other form. Projectquestion.
registry.datadoghq.com/ddot-ebpf-devand thefull_host.*job running--privileged --pid=host(+ debugfs/tracefs).from feature: OTLP format support #173 (
samplesvscpu-time, count vs ns, mapping-basename frames).Standalone mode is the intended direction (no agent dependency).
Config note
The image's Datadog-flavored
otlp_httpexporter requires add-api-keyheader even when pointed at the local sink; the value is unused.