scenarios: full_host_otlp_cpu (full-host CPU via OTLP, standalone)

base_images/Dockerfile.fullhost-otlp

@@ -0,0 +1,18 @@
+# Base image for full-host profiling scenarios that capture OTLP locally.
+#
+# It is the standalone Datadog full-host profiler (the host-profiler as an
+# OpenTelemetry Collector distribution) with the generic otlp_dump capture
+# sidecar (tools/otlp-dump) baked in. Scenarios add only their workload, config
+# and start script.
+#
+# Built by the test harness as image `prof-fullhost-otlp` (see
+# base_images / docker_helpers_test.go), context = repo root.
+#
+# The moving devtest-latest tag is intentional, to surface upstream drift.
+
+FROM golang:1.25 AS sink
+ADD ./tools/otlp-dump/ /src
+RUN cd /src && CGO_ENABLED=0 go build -o /out/otlp_dump .
+
+FROM registry.datadoghq.com/ddot-ebpf-dev:devtest-latest
+COPY --from=sink /out/otlp_dump /usr/local/bin/otlp_dump

scenarios/full_host_otlp_cpu/Dockerfile

@@ -0,0 +1,36 @@
+# full_host_otlp_cpu scenario
+#
+# Full-host CPU profiling with the standalone Datadog full-host profiler,
+# exported over OTLP and captured to .otlp files (read natively by the
+# analyzer). CPU counterpart to the pprof `full_host` scenario.
+#
+# The profiler + the generic otlp_dump capture sidecar come from the
+# prof-fullhost-otlp base image (base_images/Dockerfile.fullhost-otlp); this
+# scenario only adds the workload, config and start script.
+ARG BASE_IMAGE="prof-fullhost-otlp"
+
+# ----- build the CPU workload -----
+FROM ubuntu:22.04 AS workload-build
+RUN apt-get update && apt-get install -y gcc libc6-dev && rm -rf /var/lib/apt/lists/*
+WORKDIR /src
+ADD ./scenarios/full_host_otlp_cpu/workload.c .
+# Binary name "cpu_workload" is what the assertion matches (frames are mapping
+# basenames without symbol upload).
+RUN gcc -O2 -fno-omit-frame-pointer -o cpu_workload workload.c
+
+# ----- final image: base (profiler + otlp_dump) + our workload -----
+FROM ${BASE_IMAGE} AS final
+USER root
+RUN mkdir -p /app/data
+
+COPY --from=workload-build /src/cpu_workload /app/cpu_workload
+ADD ./scenarios/full_host_otlp_cpu/host-profiler-config.yaml /app/host-profiler-config.yaml
+ADD ./scenarios/full_host_otlp_cpu/start.sh /app/start.sh
+RUN chmod 755 /app/start.sh
+
+ENV EXECUTION_TIME_SEC="20"
+
+# Override the base image's default host-profiler entrypoint with our
+# orchestration. Requires privileges (provided by the harness for full_host*).
+ENTRYPOINT []
+CMD ["/app/start.sh"]

scenarios/full_host_otlp_cpu/README.md

@@ -0,0 +1,69 @@
+# full_host_otlp_cpu
+
+Full-host **CPU** profiling with the datadog-agent `host-profiler`, exercising
+the analyzer's **OTLP** input path end to end. It is the CPU counterpart to the
+`full_host` (pprof, standalone dd-otel-host-profiler) scenario.
+
+```
+cpu_workload  ->  host-profiler (eBPF CPU)  ->  OTLP export  ->  otlp_dump (.otlp)  ->  analyzer
+```
+
+The profiler and the OTLP capture come from the **`prof-fullhost-otlp` base
+image** (`base_images/Dockerfile.fullhost-otlp`): the **standalone** full-host
+profiler image (`registry.datadoghq.com/ddot-ebpf-dev:devtest-latest` — the
+agent `host-profiler` as an OpenTelemetry Collector distribution) with the
+generic `otlp_dump` capture sidecar (`tools/otlp-dump`) baked in. So the
+scenario is self-contained and thin — just workload + config + start script:
+**no agent, no hand-built binary, no `binaries/` prerequisite**. The moving
+`devtest-latest` tag is intentional, to surface drift. Run:
+
+```sh
+TEST_SCENARIOS="full_host_otlp_cpu" go test -v -run TestScenarios
+```
+
+The harness runs any `*full_host*` scenario `--privileged --pid=host` with the
+debugfs/tracefs mounts the eBPF profiler needs.
+
+`otlp_dump` (in `tools/otlp-dump`) is a generic, dependency-free OTLP/HTTP
+receiver that persists the profiler's export as `.otlp` files; it is reusable
+by any OTLP-emitting scenario, not specific to this one.
+
+## What this scenario shows about OTLP vs pprof expectations
+
+This is deliberately a "what does it look like" scenario. Compared with the
+pprof `full_host` scenario, the `expected_profile.json` differs in ways that are
+inherent to the OTLP host-profiler output, not to the harness:
+
+- **`profile-type` is `samples`** (unit `count`), not `cpu-time` (nanoseconds).
+  The value is a sample count.
+- **`scale_by_duration: false`** — the per-report sample counts are small, so
+  rate-scaling would truncate them toward 0. Assert on raw counts / percent.
+- **Frames are mapping basenames** (`cpu_workload`, `libc.so.6`, `linux-vdso.1.so`)
+  because symbols aren't uploaded, so one logical stack fragments into several
+  entries (`cpu_workload;libc.so.6;libc.so.6;cpu_workload`, `…;linux-vdso.1.so`,
+  …). We therefore assert a **regex-contains** on the workload binary
+  (`.*cpu_workload.*`) with a `percent` band rather than an exact stack+value.
+
+The assertion is a **load-independent rate**: the workload pins one core, so at
+the eBPF sampler frequency (~20 Hz) it accounts for ~20 samples/sec regardless
+of what else runs on the host (`value-matching-sum: 20`, `scale_by_duration:
+true`). Measured ~100 samples per ~5s steady-state report (0–3% error vs the
+30% margin).
+
+**Timing / warm state:** a rate assertion only holds for reports where the
+workload is fully on-CPU. `start.sh` therefore runs the workload *longer* than
+the profiling window and stops the profiler while the workload is still busy, so
+there is no idle "cool-down" tail report. The first report still straddles
+profiler start-up (partial), which is why `allow_first_profile_failure` is set.
+
+Config note: the image's Datadog-flavored `otlp_http` exporter requires a
+`dd-api-key` header even when pointed locally; the value is unused (otlp_dump
+ignores it).
+
+## Open question this raises
+
+The differences above are exactly what the label/expectations design note
+(`docs/label-expectations-design.md`) is about: do we converge these onto shared
+semantic fields (so one expected file works for pprof and OTLP), or accept some
+format-specific expectations? This scenario is a concrete data point for that
+discussion.

scenarios/full_host_otlp_cpu/expected_profile.json

@@ -0,0 +1,20 @@
+{
+  "test_name": "full_host_otlp_cpu",
+  "note": "Full-host CPU profiling via the datadog-agent host-profiler, exported as OTLP and captured to .otlp by the otlp_dump sidecar. OTLP-vs-pprof difference: profile-type is 'samples' (a count, unit 'count'), NOT 'cpu-time' (nanoseconds). Like the pprof full_host scenario we assert a load-independent RATE: the workload pins one core, so at the eBPF sampler frequency (~20 Hz) it accounts for ~20 samples/sec regardless of what else runs on the host (value-matching-sum with scale_by_duration:true). Frames are mapping basenames (no symbol upload), so we match the workload binary by regex.",
+  "pprof-regex": ".*\\.otlp$",
+  "allow_first_profile_failure": true,
+  "scale_by_duration": true,
+  "stacks": [
+    {
+      "profile-type": "samples",
+      "stack-content": [
+        {
+          "regular_expression": ".*cpu_workload.*",
+          "error_margin": 10
+        }
+      ],
+      "value-matching-sum": 20,
+      "error-margin": 30
+    }
+  ]
+}

scenarios/full_host_otlp_cpu/host-profiler-config.yaml

@@ -0,0 +1,34 @@
+# host-profiler config for the full_host_otlp_cpu scenario (CPU only).
+#
+# Runs against the standalone ddot-ebpf-dev image. Profiles are exported over
+# OTLP to the local otlp_dump sidecar, which writes them as .otlp files the
+# analyzer reads natively. No backend, no ebpf-profiler PR #96 (CPU only).
+receivers:
+  profiling:
+    # Short reporting interval so a short run yields several reports.
+    reporter_interval: 5s
+    reporter_jitter: 0.05
+    # No backend: don't upload symbols (frames will be mapping basenames).
+    symbol_uploader:
+      enabled: false
+
+exporters:
+  # The image's Datadog-flavored otlp_http exporter requires a dd-api-key
+  # header even when pointed at a local endpoint; the value is unused because
+  # otlp_dump ignores it and just persists the request body.
+  otlp_http/local:
+    profiles_endpoint: http://127.0.0.1:4318/v1development/profiles
+    headers:
+      dd-api-key: "local-capture-no-backend"
+    compression: none
+    tls:
+      insecure: true
+
+service:
+  telemetry:
+    logs:
+      level: info
+  pipelines:
+    profiles:
+      receivers: [profiling]
+      exporters: [otlp_http/local]

scenarios/full_host_otlp_cpu/start.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# full_host_otlp_cpu: run the standalone host-profiler (CPU), capture its OTLP
+# export to .otlp files via the otlp_dump sidecar, and run a CPU workload.
+#
+# Timing matters for a rate assertion: every report we assert on must see the
+# workload fully on-CPU. So we run the workload LONGER than the profiling window
+# and stop the profiler while the workload is still busy — this avoids an
+# idle "cool-down" tail report. The first report still straddles profiler
+# start-up (partial), which is why the scenario sets allow_first_profile_failure.
+set -u
+
+DATA_DIR=/app/data
+mkdir -p "${DATA_DIR}"
+
+PROFILE_SECS="${EXECUTION_TIME_SEC}"          # how long we profile
+WORKLOAD_SECS=$(( PROFILE_SECS + 10 ))        # workload outlives the profiler
+
+echo "=== starting otlp_dump (captures OTLP export to .otlp) ==="
+SINK_OUT_DIR="${DATA_DIR}" SINK_ADDR="127.0.0.1:4318" /usr/local/bin/otlp_dump &
+SINK_PID=$!
+sleep 1
+
+echo "=== starting host-profiler (standalone) ==="
+# Needs privileges (the harness runs full_host* scenarios with --privileged
+# --pid=host and debugfs/tracefs mounts).
+/opt/datadog-agent/embedded/bin/host-profiler run --config /app/host-profiler-config.yaml &
+PROFILER_PID=$!
+sleep 3
+
+echo "=== running cpu_workload for ${WORKLOAD_SECS}s (profiling for ${PROFILE_SECS}s) ==="
+DD_SERVICE=cpu_workload_test timeout "${WORKLOAD_SECS}"s /app/cpu_workload &
+APP_PID=$!
+
+# Collect reports while the workload is busy, then stop the profiler *before*
+# the workload ends so the final flushed report is still fully on-CPU.
+sleep "${PROFILE_SECS}"
+
+echo "=== stopping profiler (workload still running) ==="
+kill "${PROFILER_PID}" 2>/dev/null || true
+wait "${PROFILER_PID}" 2>/dev/null || true
+
+kill "${APP_PID}" 2>/dev/null || true
+wait "${APP_PID}" 2>/dev/null || true
+sleep 1
+kill "${SINK_PID}" 2>/dev/null || true
+wait "${SINK_PID}" 2>/dev/null || true
+
+echo "=== output files ==="
+ls -la "${DATA_DIR}"

scenarios/full_host_otlp_cpu/workload.c

@@ -0,0 +1,41 @@
+#include <time.h>
+#include <stdint.h>
+#include <stdlib.h>
+#include <stdio.h>
+
+void a() {
+    int64_t x = 0;
+    int64_t i = 0;
+    while (i < 500000000) {
+        x += i * i;  // More CPU intensive
+        i += 1;
+    }
+}
+
+void b() {
+    int64_t x = 0;
+    int64_t i = 0;
+    while (i < 1000000000) {
+        x += i * i * i;  // Even more CPU intensive
+        i += 1;
+    }
+}
+
+int main(int argc, char *argv[]) {
+    int test_duration = 60;
+    const char *exec_time_env = getenv("EXECUTION_TIME_SEC");
+    if (exec_time_env) {
+        test_duration = atoi(exec_time_env);
+        if (test_duration == 0) {
+            exit(1);
+        }
+    }
+    printf("Executable %s starting for %d seconds\n", argv[0], test_duration);
+    time_t end = time(NULL) + test_duration;
+    while (time(NULL) < end) {
+        a();
+        b();
+    }
+    printf("Executable %s finished successfully\n", argv[0]);
+    return 0;
+}

tools/otlp-dump/go.mod

@@ -0,0 +1,3 @@
+module otlp-dump
+
+go 1.25.1

tools/otlp-dump/main.go

@@ -0,0 +1,87 @@
+// Command otlp_dump is a tiny, dependency-free OTLP/HTTP profiles receiver that
+// writes each received export request verbatim to a ".otlp" file.
+//
+// The datadog-agent host-profiler only exports profiles over OTLP (no local
+// file/pprof exporter). The prof-correctness analyzer reads the OTLP format
+// natively (analysis/otlp.go), so this sidecar does not parse the payload - it
+// just persists the raw protobuf bytes so the analyzer can read them from
+// /app/data.
+//
+// otlphttpexporter POSTs the (unstable) profiles signal to
+// "<endpoint>/v1development/profiles" as protobuf, optionally gzip-encoded.
+package main
+
+import (
+	"compress/gzip"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"path/filepath"
+	"sync/atomic"
+)
+
+var seq atomic.Uint64
+
+func writeDump(outDir string, w http.ResponseWriter, r *http.Request) {
+	if r.Method != http.MethodPost {
+		http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+		return
+	}
+
+	var body io.Reader = r.Body
+	if r.Header.Get("Content-Encoding") == "gzip" {
+		gz, err := gzip.NewReader(r.Body)
+		if err != nil {
+			http.Error(w, "bad gzip", http.StatusBadRequest)
+			return
+		}
+		defer gz.Close()
+		body = gz
+	}
+
+	data, err := io.ReadAll(body)
+	if err != nil {
+		http.Error(w, "read error", http.StatusBadRequest)
+		return
+	}
+
+	n := seq.Add(1)
+	name := fmt.Sprintf("profiles_%03d.otlp", n)
+	if err := os.WriteFile(filepath.Join(outDir, name), data, 0o644); err != nil {
+		fmt.Fprintf(os.Stderr, "otlp_dump: write %s: %v\n", name, err)
+		http.Error(w, "write error", http.StatusInternalServerError)
+		return
+	}
+	fmt.Printf("otlp_dump: wrote %s (%d bytes)\n", name, len(data))
+
+	// An empty body is a valid empty ExportProfilesServiceResponse.
+	w.Header().Set("Content-Type", "application/x-protobuf")
+	w.WriteHeader(http.StatusOK)
+}
+
+func main() {
+	addr := os.Getenv("SINK_ADDR")
+	if addr == "" {
+		addr = "0.0.0.0:4318"
+	}
+	outDir := os.Getenv("SINK_OUT_DIR")
+	if outDir == "" {
+		outDir = "/app/data"
+	}
+	if err := os.MkdirAll(outDir, 0o755); err != nil {
+		fmt.Fprintf(os.Stderr, "otlp_dump: mkdir %s: %v\n", outDir, err)
+		os.Exit(1)
+	}
+
+	h := func(w http.ResponseWriter, r *http.Request) { writeDump(outDir, w, r) }
+	mux := http.NewServeMux()
+	mux.HandleFunc("/v1development/profiles", h)
+	mux.HandleFunc("/v1/profiles", h) // tolerate a signal-version bump
+
+	fmt.Printf("otlp_dump listening on %s, writing .otlp to %s\n", addr, outDir)
+	if err := http.ListenAndServe(addr, mux); err != nil {
+		fmt.Fprintf(os.Stderr, "otlp_dump: serve: %v\n", err)
+		os.Exit(1)
+	}
+}