Skip to content

Commit 3a1b0c6

Browse files
committed
scenarios: add full_host_otlp_cpu (full-host CPU via OTLP)
Full-host CPU profiling exercised through the analyzer's OTLP input path (the CPU counterpart to the pprof `full_host` scenario). - Profiler: the standalone full-host image (registry.datadoghq.com/ddot-ebpf-dev:devtest-latest — the host-profiler as an OpenTelemetry Collector distribution). Self-contained: no agent, no hand-built binary. The moving devtest-latest tag is intentional, to surface drift. - Capture: a tiny dependency-free otlp_dump sidecar persists the profiler's OTLP export as .otlp files the analyzer reads natively. (Whether this is the capture mechanism we want long-term is an open project question.) - Assertion: a load-independent CPU rate — the workload pins one core, so at the ~20 Hz sampler it accounts for ~20 samples/sec regardless of host load (value-matching-sum: 20, scale_by_duration: true), mirroring full_host's 1e9 ns/s. profile-type is "samples" (count), not "cpu-time". - Timing: the workload runs longer than the profiling window and the profiler is stopped while it is still busy, so no idle cool-down tail report is produced; only the first (start-up) report is partial, tolerated via allow_first_profile_failure. Config note: the image's Datadog-flavored otlp_http exporter requires a dd-api-key header even when pointed at the local sink (value unused). Validated end-to-end against the real image: warm-up report tolerated, steady reports 0-3% error vs the 30% margin.
1 parent 5586100 commit 3a1b0c6

8 files changed

Lines changed: 340 additions & 0 deletions

File tree

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# full_host_otlp_cpu scenario
2+
#
3+
# Full-host CPU profiling with the standalone Datadog full-host profiler,
4+
# exported over OTLP and captured to .otlp files (read natively by the
5+
# analyzer). CPU counterpart to the pprof `full_host` scenario.
6+
#
7+
# The profiler is the official standalone image (the agent host-profiler as an
8+
# OpenTelemetry Collector distribution) — no agent and no hand-built binary
9+
# needed. We use the moving devtest-latest tag on purpose, to surface drift.
10+
11+
# ----- build the CPU workload -----
12+
FROM ubuntu:22.04 AS workload-build
13+
RUN apt-get update && apt-get install -y gcc libc6-dev && rm -rf /var/lib/apt/lists/*
14+
WORKDIR /src
15+
ADD ./scenarios/full_host_otlp_cpu/workload.c .
16+
# Binary name "cpu_workload" is what the assertion matches (frames are mapping
17+
# basenames without symbol upload).
18+
RUN gcc -O2 -fno-omit-frame-pointer -o cpu_workload workload.c
19+
20+
# ----- build the OTLP dump sidecar (dependency-free) -----
21+
FROM golang:1.25 AS sink-build
22+
WORKDIR /src
23+
ADD ./scenarios/full_host_otlp_cpu/otlp_dump/ .
24+
RUN CGO_ENABLED=0 go build -o /out/otlp_dump .
25+
26+
# ----- final image: the standalone full-host profiler + our workload/sink -----
27+
FROM registry.datadoghq.com/ddot-ebpf-dev:devtest-latest AS final
28+
USER root
29+
RUN mkdir -p /app/data
30+
31+
COPY --from=workload-build /src/cpu_workload /app/cpu_workload
32+
COPY --from=sink-build /out/otlp_dump /usr/local/bin/otlp_dump
33+
34+
ADD ./scenarios/full_host_otlp_cpu/host-profiler-config.yaml /app/host-profiler-config.yaml
35+
ADD ./scenarios/full_host_otlp_cpu/start.sh /app/start.sh
36+
RUN chmod 755 /app/start.sh
37+
38+
ENV EXECUTION_TIME_SEC="20"
39+
40+
# Override the image's default host-profiler entrypoint with our orchestration.
41+
# Requires privileges (provided by the harness for full_host* scenarios).
42+
ENTRYPOINT []
43+
CMD ["/app/start.sh"]
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# full_host_otlp_cpu
2+
3+
Full-host **CPU** profiling with the datadog-agent `host-profiler`, exercising
4+
the analyzer's **OTLP** input path end to end. It is the CPU counterpart to the
5+
`full_host` (pprof, standalone dd-otel-host-profiler) scenario.
6+
7+
```
8+
cpu_workload -> host-profiler (eBPF CPU) -> OTLP export -> otlp_dump (.otlp) -> analyzer
9+
```
10+
11+
The profiler is the **standalone** full-host image
12+
(`registry.datadoghq.com/ddot-ebpf-dev:devtest-latest` — the agent
13+
`host-profiler` packaged as an OpenTelemetry Collector distribution), so the
14+
scenario is self-contained: **no agent, no hand-built binary, no `binaries/`
15+
prerequisite**. We use the moving `devtest-latest` tag on purpose, to surface
16+
drift. Run:
17+
18+
```sh
19+
TEST_SCENARIOS="full_host_otlp_cpu" go test -v -run TestScenarios
20+
```
21+
22+
The harness runs any `*full_host*` scenario `--privileged --pid=host` with the
23+
debugfs/tracefs mounts the eBPF profiler needs.
24+
25+
## What this scenario shows about OTLP vs pprof expectations
26+
27+
This is deliberately a "what does it look like" scenario. Compared with the
28+
pprof `full_host` scenario, the `expected_profile.json` differs in ways that are
29+
inherent to the OTLP host-profiler output, not to the harness:
30+
31+
- **`profile-type` is `samples`** (unit `count`), not `cpu-time` (nanoseconds).
32+
The value is a sample count.
33+
- **`scale_by_duration: false`** — the per-report sample counts are small, so
34+
rate-scaling would truncate them toward 0. Assert on raw counts / percent.
35+
- **Frames are mapping basenames** (`cpu_workload`, `libc.so.6`, `linux-vdso.1.so`)
36+
because symbols aren't uploaded, so one logical stack fragments into several
37+
entries (`cpu_workload;libc.so.6;libc.so.6;cpu_workload`, `…;linux-vdso.1.so`,
38+
…). We therefore assert a **regex-contains** on the workload binary
39+
(`.*cpu_workload.*`) with a `percent` band rather than an exact stack+value.
40+
41+
The assertion is a **load-independent rate**: the workload pins one core, so at
42+
the eBPF sampler frequency (~20 Hz) it accounts for ~20 samples/sec regardless
43+
of what else runs on the host (`value-matching-sum: 20`, `scale_by_duration:
44+
true`). Measured ~100 samples per ~5s steady-state report (0–3% error vs the
45+
30% margin).
46+
47+
**Timing / warm state:** a rate assertion only holds for reports where the
48+
workload is fully on-CPU. `start.sh` therefore runs the workload *longer* than
49+
the profiling window and stops the profiler while the workload is still busy, so
50+
there is no idle "cool-down" tail report. The first report still straddles
51+
profiler start-up (partial), which is why `allow_first_profile_failure` is set.
52+
53+
Config note: the image's Datadog-flavored `otlp_http` exporter requires a
54+
`dd-api-key` header even when pointed locally; the value is unused (otlp_dump
55+
ignores it).
56+
57+
## Open question this raises
58+
59+
The differences above are exactly what the label/expectations design note
60+
(`docs/label-expectations-design.md`) is about: do we converge these onto shared
61+
semantic fields (so one expected file works for pprof and OTLP), or accept some
62+
format-specific expectations? This scenario is a concrete data point for that
63+
discussion.
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"test_name": "full_host_otlp_cpu",
3+
"note": "Full-host CPU profiling via the datadog-agent host-profiler, exported as OTLP and captured to .otlp by the otlp_dump sidecar. OTLP-vs-pprof difference: profile-type is 'samples' (a count, unit 'count'), NOT 'cpu-time' (nanoseconds). Like the pprof full_host scenario we assert a load-independent RATE: the workload pins one core, so at the eBPF sampler frequency (~20 Hz) it accounts for ~20 samples/sec regardless of what else runs on the host (value-matching-sum with scale_by_duration:true). Frames are mapping basenames (no symbol upload), so we match the workload binary by regex.",
4+
"pprof-regex": ".*\\.otlp$",
5+
"allow_first_profile_failure": true,
6+
"scale_by_duration": true,
7+
"stacks": [
8+
{
9+
"profile-type": "samples",
10+
"stack-content": [
11+
{
12+
"regular_expression": ".*cpu_workload.*",
13+
"error_margin": 10
14+
}
15+
],
16+
"value-matching-sum": 20,
17+
"error-margin": 30
18+
}
19+
]
20+
}
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# host-profiler config for the full_host_otlp_cpu scenario (CPU only).
2+
#
3+
# Runs against the standalone ddot-ebpf-dev image. Profiles are exported over
4+
# OTLP to the local otlp_dump sidecar, which writes them as .otlp files the
5+
# analyzer reads natively. No backend, no ebpf-profiler PR #96 (CPU only).
6+
receivers:
7+
profiling:
8+
# Short reporting interval so a short run yields several reports.
9+
reporter_interval: 5s
10+
reporter_jitter: 0.05
11+
# No backend: don't upload symbols (frames will be mapping basenames).
12+
symbol_uploader:
13+
enabled: false
14+
15+
exporters:
16+
# The image's Datadog-flavored otlp_http exporter requires a dd-api-key
17+
# header even when pointed at a local endpoint; the value is unused because
18+
# otlp_dump ignores it and just persists the request body.
19+
otlp_http/local:
20+
profiles_endpoint: http://127.0.0.1:4318/v1development/profiles
21+
headers:
22+
dd-api-key: "local-capture-no-backend"
23+
compression: none
24+
tls:
25+
insecure: true
26+
27+
service:
28+
telemetry:
29+
logs:
30+
level: info
31+
pipelines:
32+
profiles:
33+
receivers: [profiling]
34+
exporters: [otlp_http/local]
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module otlp_dump
2+
3+
go 1.25.1
Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
// Command otlp_dump is a tiny, dependency-free OTLP/HTTP profiles receiver that
2+
// writes each received export request verbatim to a ".otlp" file.
3+
//
4+
// The datadog-agent host-profiler only exports profiles over OTLP (no local
5+
// file/pprof exporter). The prof-correctness analyzer reads the OTLP format
6+
// natively (analysis/otlp.go), so this sidecar does not parse the payload - it
7+
// just persists the raw protobuf bytes so the analyzer can read them from
8+
// /app/data.
9+
//
10+
// otlphttpexporter POSTs the (unstable) profiles signal to
11+
// "<endpoint>/v1development/profiles" as protobuf, optionally gzip-encoded.
12+
package main
13+
14+
import (
15+
"compress/gzip"
16+
"fmt"
17+
"io"
18+
"net/http"
19+
"os"
20+
"path/filepath"
21+
"sync/atomic"
22+
)
23+
24+
var seq atomic.Uint64
25+
26+
func writeDump(outDir string, w http.ResponseWriter, r *http.Request) {
27+
if r.Method != http.MethodPost {
28+
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
29+
return
30+
}
31+
32+
var body io.Reader = r.Body
33+
if r.Header.Get("Content-Encoding") == "gzip" {
34+
gz, err := gzip.NewReader(r.Body)
35+
if err != nil {
36+
http.Error(w, "bad gzip", http.StatusBadRequest)
37+
return
38+
}
39+
defer gz.Close()
40+
body = gz
41+
}
42+
43+
data, err := io.ReadAll(body)
44+
if err != nil {
45+
http.Error(w, "read error", http.StatusBadRequest)
46+
return
47+
}
48+
49+
n := seq.Add(1)
50+
name := fmt.Sprintf("profiles_%03d.otlp", n)
51+
if err := os.WriteFile(filepath.Join(outDir, name), data, 0o644); err != nil {
52+
fmt.Fprintf(os.Stderr, "otlp_dump: write %s: %v\n", name, err)
53+
http.Error(w, "write error", http.StatusInternalServerError)
54+
return
55+
}
56+
fmt.Printf("otlp_dump: wrote %s (%d bytes)\n", name, len(data))
57+
58+
// An empty body is a valid empty ExportProfilesServiceResponse.
59+
w.Header().Set("Content-Type", "application/x-protobuf")
60+
w.WriteHeader(http.StatusOK)
61+
}
62+
63+
func main() {
64+
addr := os.Getenv("SINK_ADDR")
65+
if addr == "" {
66+
addr = "0.0.0.0:4318"
67+
}
68+
outDir := os.Getenv("SINK_OUT_DIR")
69+
if outDir == "" {
70+
outDir = "/app/data"
71+
}
72+
if err := os.MkdirAll(outDir, 0o755); err != nil {
73+
fmt.Fprintf(os.Stderr, "otlp_dump: mkdir %s: %v\n", outDir, err)
74+
os.Exit(1)
75+
}
76+
77+
h := func(w http.ResponseWriter, r *http.Request) { writeDump(outDir, w, r) }
78+
mux := http.NewServeMux()
79+
mux.HandleFunc("/v1development/profiles", h)
80+
mux.HandleFunc("/v1/profiles", h) // tolerate a signal-version bump
81+
82+
fmt.Printf("otlp_dump listening on %s, writing .otlp to %s\n", addr, outDir)
83+
if err := http.ListenAndServe(addr, mux); err != nil {
84+
fmt.Fprintf(os.Stderr, "otlp_dump: serve: %v\n", err)
85+
os.Exit(1)
86+
}
87+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
# full_host_otlp_cpu: run the standalone host-profiler (CPU), capture its OTLP
3+
# export to .otlp files via the otlp_dump sidecar, and run a CPU workload.
4+
#
5+
# Timing matters for a rate assertion: every report we assert on must see the
6+
# workload fully on-CPU. So we run the workload LONGER than the profiling window
7+
# and stop the profiler while the workload is still busy — this avoids an
8+
# idle "cool-down" tail report. The first report still straddles profiler
9+
# start-up (partial), which is why the scenario sets allow_first_profile_failure.
10+
set -u
11+
12+
DATA_DIR=/app/data
13+
mkdir -p "${DATA_DIR}"
14+
15+
PROFILE_SECS="${EXECUTION_TIME_SEC}" # how long we profile
16+
WORKLOAD_SECS=$(( PROFILE_SECS + 10 )) # workload outlives the profiler
17+
18+
echo "=== starting otlp_dump (captures OTLP export to .otlp) ==="
19+
SINK_OUT_DIR="${DATA_DIR}" SINK_ADDR="127.0.0.1:4318" /usr/local/bin/otlp_dump &
20+
SINK_PID=$!
21+
sleep 1
22+
23+
echo "=== starting host-profiler (standalone) ==="
24+
# Needs privileges (the harness runs full_host* scenarios with --privileged
25+
# --pid=host and debugfs/tracefs mounts).
26+
/opt/datadog-agent/embedded/bin/host-profiler run --config /app/host-profiler-config.yaml &
27+
PROFILER_PID=$!
28+
sleep 3
29+
30+
echo "=== running cpu_workload for ${WORKLOAD_SECS}s (profiling for ${PROFILE_SECS}s) ==="
31+
DD_SERVICE=cpu_workload_test timeout "${WORKLOAD_SECS}"s /app/cpu_workload &
32+
APP_PID=$!
33+
34+
# Collect reports while the workload is busy, then stop the profiler *before*
35+
# the workload ends so the final flushed report is still fully on-CPU.
36+
sleep "${PROFILE_SECS}"
37+
38+
echo "=== stopping profiler (workload still running) ==="
39+
kill "${PROFILER_PID}" 2>/dev/null || true
40+
wait "${PROFILER_PID}" 2>/dev/null || true
41+
42+
kill "${APP_PID}" 2>/dev/null || true
43+
wait "${APP_PID}" 2>/dev/null || true
44+
sleep 1
45+
kill "${SINK_PID}" 2>/dev/null || true
46+
wait "${SINK_PID}" 2>/dev/null || true
47+
48+
echo "=== output files ==="
49+
ls -la "${DATA_DIR}"
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#include <time.h>
2+
#include <stdint.h>
3+
#include <stdlib.h>
4+
#include <stdio.h>
5+
6+
void a() {
7+
int64_t x = 0;
8+
int64_t i = 0;
9+
while (i < 500000000) {
10+
x += i * i; // More CPU intensive
11+
i += 1;
12+
}
13+
}
14+
15+
void b() {
16+
int64_t x = 0;
17+
int64_t i = 0;
18+
while (i < 1000000000) {
19+
x += i * i * i; // Even more CPU intensive
20+
i += 1;
21+
}
22+
}
23+
24+
int main(int argc, char *argv[]) {
25+
int test_duration = 60;
26+
const char *exec_time_env = getenv("EXECUTION_TIME_SEC");
27+
if (exec_time_env) {
28+
test_duration = atoi(exec_time_env);
29+
if (test_duration == 0) {
30+
exit(1);
31+
}
32+
}
33+
printf("Executable %s starting for %d seconds\n", argv[0], test_duration);
34+
time_t end = time(NULL) + test_duration;
35+
while (time(NULL) < end) {
36+
a();
37+
b();
38+
}
39+
printf("Executable %s finished successfully\n", argv[0]);
40+
return 0;
41+
}

0 commit comments

Comments
 (0)