Skip to content

Commit bc500ec

Browse files
bltblt
committed
chore(antithesis): Rework rig containers, setup
This commit modifies the rig so that we are always using Datadog Agent + ADP as the SUT. In the general case this is ADP-on and in the differential case this is between ADP-off and ADP-on. We make one small modification to the Datadog Agent container in that we do not restart ADP if it faults, giving antithesis a cleaner exit signal. I have tried to make all our containers sync'ed properly with the rest of the project, as they had drifted. If I understand the auto-update proecess correctly this should all stay square from now on. From Vector I have ported the launch script and, hopefully, this makes it a little more clear how to launch these things. Of interest I have changed the endpoint and turned OFF node faults for both scenarios as we cannot ensure that, especially in the differential case, that faults happen at the same instant and are fair.
1 parent e0473cd commit bc500ec

16 files changed

Lines changed: 291 additions & 150 deletions

‎Makefile‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -790,7 +790,7 @@ ifeq ($(shell command -v snouty >/dev/null || echo not-found), not-found)
790790
endif
791791

792792
.PHONY: antithesis-build
793-
antithesis-build: ## Builds the Antithesis harness container images
793+
antithesis-build: build-datadog-agent-image-release ## Builds the Antithesis harness container images
794794
@echo "[*] Building Antithesis harness images..."
795795
@docker compose -f $(ANTITHESIS_COMPOSE_FILE) build
796796

@@ -799,10 +799,10 @@ antithesis-build-differential: build-datadog-agent-image-release ## Builds the d
799799
@echo "[*] Building differential Antithesis harness images..."
800800
@docker compose -f $(ANTITHESIS_DIFFERENTIAL_COMPOSE_FILE) build
801801

802-
.PHONY: antithesis-validate
803-
antithesis-validate: check-antithesis-tools antithesis-build
804-
antithesis-validate: ## Validates the Antithesis harness: builds images, runs 'snouty validate'
805-
@echo "[*] Validating Antithesis harness with snouty..."
802+
.PHONY: antithesis-validate-general
803+
antithesis-validate-general: check-antithesis-tools antithesis-build
804+
antithesis-validate-general: ## Validates the general Antithesis harness: builds images, runs 'snouty validate'
805+
@echo "[*] Validating general Antithesis harness with snouty..."
806806
@snouty validate $(ANTITHESIS_CONFIG_DIR)
807807

808808
.PHONY: antithesis-validate-differential
@@ -811,6 +811,9 @@ antithesis-validate-differential: ## Validates the differential Antithesis harne
811811
@echo "[*] Validating differential Antithesis harness with snouty..."
812812
@snouty validate $(ANTITHESIS_DIFFERENTIAL_CONFIG_DIR)
813813

814+
.PHONY: antithesis-validate
815+
antithesis-validate: antithesis-validate-general antithesis-validate-differential ## Validates both Antithesis harnesses (general + differential)
816+
814817
##@ Profiling
815818

816819
.PHONY: profile-run-blackhole

‎renovate.json5‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,14 +108,18 @@
108108
"customType": "regex",
109109
"fileMatch": [
110110
"(^|/)Makefile$",
111-
"^\\.gitlab/windows\\.yml$"
111+
"^\\.gitlab/windows\\.yml$",
112+
// Antithesis harness images fetch the matching `-dbg` symbols package by version.
113+
"^test/antithesis/scenarios/(differential|general)/Dockerfile$"
112114
],
113115
"matchStrings": [
114116
// Bare version, e.g. `MACOS_TEST_AGENT_VERSION ?= 7.80.3` (Makefile).
115117
"MACOS_TEST_AGENT_VERSION \\?= (?<currentValue>[\\d.]+)",
116118
// Image pin, e.g. `registry.datadoghq.com/agent:7.80.3-ltsc2022` (.gitlab/windows.yml).
117119
// The `-suffix` is matched but not captured, so only the number is rewritten.
118-
"registry\\.datadoghq\\.com/agent:(?<currentValue>[\\d.]+)(?:-[a-z0-9]+)?"
120+
"registry\\.datadoghq\\.com/agent:(?<currentValue>[\\d.]+)(?:-[a-z0-9]+)?",
121+
// Agent debug-symbols version, e.g. `ARG DD_AGENT_VERSION=7.81.0` (antithesis Dockerfiles).
122+
"ARG DD_AGENT_VERSION=(?<currentValue>[\\d.]+)"
119123
],
120124
"datasourceTemplate": "docker",
121125
"depNameTemplate": "registry.datadoghq.com/agent",

‎test/antithesis/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Rendered compose snouty launches from (concrete image tags); regenerated per launch.
2+
scenarios/*/.launch/

‎test/antithesis/bin/launch.sh‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
#!/usr/bin/env bash
2+
# Generic Antithesis launcher shared by every scenario.
3+
#
4+
# ./launch.sh <scenario> [extra snouty flags]
5+
#
6+
# <scenario> names a directory under `test/antithesis/scenarios/` holding a
7+
# docker-compose.yaml and a launch.env. launch.env supplies the per-scenario
8+
# bits; everything else -- image tagging, property-history key, the fault profile
9+
# shape, build-before-submit -- is common and lives here so every shot is
10+
# identical and comparable and no fault flag is ever fumbled or forgotten. Change
11+
# a shot's faults by editing launch.env's node list, not by passing one-off flags.
12+
#
13+
# launch.env, sourced from the scenario directory, sets:
14+
# SCENARIO_TEST_NAME test name reported to Antithesis
15+
# SCENARIO_DESCRIPTION human description; the git commit is appended
16+
# SCENARIO_FAULT_NODES space-separated SUT container names to node-fault;
17+
# empty means no node termination/hang/throttle at all
18+
# SCENARIO_WEBHOOK optional; tenant webhook, default persistent_storage
19+
#
20+
# Required environment, read by snouty:
21+
# ANTITHESIS_TENANT tenant name
22+
# ANTITHESIS_API_KEY api key, or ANTITHESIS_USERNAME + ANTITHESIS_PASSWORD
23+
# ANTITHESIS_REPOSITORY registry to push the built config + service images to
24+
#
25+
# Optional overrides win over launch.env / defaults:
26+
# DURATION=<minutes> default 30
27+
# TEST_NAME=<name> default SCENARIO_TEST_NAME
28+
# DESCRIPTION=<text> default SCENARIO_DESCRIPTION; commit is appended
29+
# FAULT_NODES=<names> default SCENARIO_FAULT_NODES
30+
# WEBHOOK=<name> default SCENARIO_WEBHOOK or persistent_storage
31+
# SOURCE=<identifier> property-history key; default is the git branch
32+
# DRY_RUN=1 print the exact command and exit without submitting
33+
set -euo pipefail
34+
35+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
36+
ANTITHESIS_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
37+
REPO_ROOT="$(cd "$ANTITHESIS_DIR/../.." && pwd)"
38+
39+
SCENARIO="${1:?usage: launch.sh <scenario> [extra snouty flags]}"
40+
shift
41+
SCENARIO_DIR="$ANTITHESIS_DIR/scenarios/$SCENARIO"
42+
[ -d "$SCENARIO_DIR" ] || { echo "error: no scenario directory $SCENARIO_DIR" >&2; exit 1; }
43+
[ -f "$SCENARIO_DIR/docker-compose.yaml" ] || { echo "error: $SCENARIO_DIR/docker-compose.yaml not found" >&2; exit 1; }
44+
[ -f "$SCENARIO_DIR/launch.env" ] || { echo "error: $SCENARIO_DIR/launch.env not found" >&2; exit 1; }
45+
46+
# Per-scenario settings. Declared here so a missing one is caught, not silently empty.
47+
SCENARIO_TEST_NAME=""
48+
SCENARIO_DESCRIPTION=""
49+
SCENARIO_FAULT_NODES=""
50+
SCENARIO_WEBHOOK=""
51+
# shellcheck source=/dev/null
52+
. "$SCENARIO_DIR/launch.env"
53+
54+
# Immutable per-build revision: the short commit, marked -dirty when the working
55+
# tree has uncommitted changes so the tag never claims to be a clean commit it is
56+
# not. Images are tagged by this, never :latest, so a shot can never reuse a stale
57+
# mutable tag and every pushed image traces back to the source it was built from.
58+
GIT_SHA="$(git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)"
59+
if [[ -n "$(git -C "$REPO_ROOT" status --porcelain 2>/dev/null)" ]]; then
60+
GIT_SHA="${GIT_SHA}-dirty"
61+
fi
62+
export ANTITHESIS_IMAGE_TAG="$GIT_SHA"
63+
64+
WEBHOOK="${WEBHOOK:-${SCENARIO_WEBHOOK:-persistent_storage}}"
65+
DURATION="${DURATION:-30}"
66+
TEST_NAME="${TEST_NAME:-${SCENARIO_TEST_NAME:?launch.env must set SCENARIO_TEST_NAME}}"
67+
DESCRIPTION="${DESCRIPTION:-$SCENARIO_DESCRIPTION} (commit ${GIT_SHA})"
68+
# May be empty: an empty node list means no node faults for this scenario.
69+
FAULT_NODES="${FAULT_NODES:-$SCENARIO_FAULT_NODES}"
70+
71+
# Property-history key. Passing --source makes the run tracked, not ephemeral,
72+
# so findings are produced and each property's history is grouped by this key.
73+
# Default to the branch so history follows the branch; without it snouty runs
74+
# ephemeral and no findings are available to triage.
75+
SOURCE="${SOURCE:-$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)}"
76+
77+
# Pinned fault profile, submitted to the persistent_storage endpoint. cpu_mod and
78+
# clock_jitter are global and symmetric, so every scenario gets them; clock_jitter
79+
# is what exercises the AWS-LC CPU-jitter entropy path. Network faults stay on
80+
# everywhere and heal before judging.
81+
#
82+
# Node termination, hang, and throttle apply only to the containers in FAULT_NODES,
83+
# so a scenario gets none by leaving it empty. The differential A/B does: node
84+
# termination, hang, or throttle on one SUT but not the other -- or on both at
85+
# different times -- drops data unevenly and manufactures a divergence that is a
86+
# fault artifact, not an ADP defect. All three are unfair for the same reason, so
87+
# all three are gated together on FAULT_NODES.
88+
FAULTS=(
89+
--param custom.cpu_mod=true
90+
--param custom.clock_jitter=true
91+
)
92+
if [[ -n "$FAULT_NODES" ]]; then
93+
FAULTS+=(
94+
--param custom.include_for_node_termination="$FAULT_NODES"
95+
--param custom.include_for_node_hang="$FAULT_NODES"
96+
--param custom.include_for_node_throttle="$FAULT_NODES"
97+
)
98+
fi
99+
100+
for v in ANTITHESIS_TENANT ANTITHESIS_REPOSITORY; do
101+
if [[ -z "${!v:-}" ]]; then
102+
echo "error: $v is not set (required to build and submit the run)" >&2
103+
exit 1
104+
fi
105+
done
106+
107+
# Both scenarios embed our instrumented ADP into the project-pinned converged
108+
# Datadog Agent image, so build that first, then the scenario images. snouty
109+
# reuses a matching tag instead of rebuilding, so without this a shot can ship
110+
# stale code; layer caching keeps it near-instant when nothing changed.
111+
prebuild=(make -C "$REPO_ROOT" build-datadog-agent-image-release)
112+
build=(docker compose -f "$SCENARIO_DIR/docker-compose.yaml" build)
113+
114+
# Launch from a rendered copy so the image tag is concrete. snouty ships the compose
115+
# uninterpolated, so an `${ANTITHESIS_IMAGE_TAG:-latest}` tag reaches the platform as
116+
# the never-pushed `:latest`; `docker compose config` bakes in the tag snouty pushed.
117+
LAUNCH_DIR="$SCENARIO_DIR/.launch"
118+
render=(docker compose -f "$SCENARIO_DIR/docker-compose.yaml" config)
119+
120+
cmd=(snouty launch
121+
--webhook "$WEBHOOK"
122+
--config "$LAUNCH_DIR"
123+
--test-name "$TEST_NAME"
124+
--description "$DESCRIPTION"
125+
--source "$SOURCE"
126+
--duration "$DURATION"
127+
"${FAULTS[@]}"
128+
"$@")
129+
130+
printf 'prebuild:'; printf ' %q' "${prebuild[@]}"; printf '\n'
131+
printf 'build:'; printf ' %q' "${build[@]}"; printf '\n'
132+
printf 'render:'; printf ' %q' "${render[@]}"; printf ' > %q\n' "$LAUNCH_DIR/docker-compose.yaml"
133+
printf 'launch:'; printf ' %q' "${cmd[@]}"; printf '\n'
134+
if [[ "${DRY_RUN:-0}" == "1" ]]; then
135+
echo "(dry run; not building or submitting)"
136+
exit 0
137+
fi
138+
"${prebuild[@]}"
139+
"${build[@]}"
140+
mkdir -p "$LAUNCH_DIR"
141+
"${render[@]}" >"$LAUNCH_DIR/docker-compose.yaml"
142+
exec "${cmd[@]}"

‎test/antithesis/data-plane-finish‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/usr/bin/execlineb -S1
2+
3+
# Antithesis override of the converged image's data-plane s6 `finish`.
4+
#
5+
# A clean exit code 0 means the embedded data plane was disabled, the ADP-off
6+
# lane or ADP not enabled, so stop just this service as the stock script does.
7+
# Any non-zero exit is a crash: the antithesis build runs ADP with
8+
# panic="abort", so a panic becomes SIGABRT. Rather than respawn on a 2s loop and
9+
# mask it, bring the whole container down so Antithesis records the fault, exactly
10+
# as a standalone-ADP container death would.
11+
ifthenelse
12+
{ s6-test ${1} -eq 0 }
13+
{
14+
foreground { /initlog.sh "agent-data-plane exited with code ${1}, disabling" }
15+
foreground { /bin/s6-svc -d /var/run/s6/services/data-plane/ }
16+
}
17+
{
18+
foreground { /initlog.sh "agent-data-plane crashed (code ${1}, signal ${2}), bringing container down" }
19+
s6-svscanctl -t /var/run/s6/services
20+
}

‎test/antithesis/scenarios/differential/Dockerfile‎

Lines changed: 31 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
# The Datadog Agent lane wraps the project-wide Agent image built by Make from
66
# `.datadog-agent-version`. Do not pin an upstream Agent tag in this file.
77

8-
ARG BUILD_IMAGE=ubuntu:24.04@sha256:c4a8d5503dfb2a3eb8ab5f807da5bc69a85730fb49b5cfca2330194ebcc41c7b
9-
ARG APP_IMAGE=ubuntu:24.04@sha256:c4a8d5503dfb2a3eb8ab5f807da5bc69a85730fb49b5cfca2330194ebcc41c7b
8+
ARG BUILD_IMAGE=ubuntu:26.04@sha256:b7f48194d4d8b763a478a621cdc81c27be222ba2206ca3ca6bc42b49685f3d9e
9+
ARG APP_IMAGE=ubuntu:26.04@sha256:b7f48194d4d8b763a478a621cdc81c27be222ba2206ca3ca6bc42b49685f3d9e
1010
ARG DD_AGENT_IMAGE=saluki-images/datadog-agent:testing-release
1111

1212
# ---------------------------------------------------------------------------
@@ -68,32 +68,20 @@ RUN --mount=type=cache,target=/tools/target,id=antithesis-differential-tools-tar
6868
cp /tools/target/release/first_sample_config /usr/local/bin/first_sample_config
6969

7070
# ---------------------------------------------------------------------------
71-
# Runtime: Agent Data Plane (SUT).
72-
# ---------------------------------------------------------------------------
73-
FROM ${APP_IMAGE} AS adp
74-
ENV NO_COLOR=1
75-
RUN apt-get update && \
76-
apt-get install --no-install-recommends -y ca-certificates openssl && \
77-
rm -rf /var/lib/apt/lists/*
78-
COPY --from=adp-builder /usr/local/bin/agent-data-plane /usr/local/bin/agent-data-plane
79-
RUN mkdir -p /symbols && ln -s /usr/local/bin/agent-data-plane /symbols/agent-data-plane
80-
COPY --chmod=755 test/antithesis/scenarios/differential/adp/entrypoint.sh /entrypoint.sh
81-
RUN mkdir -p /etc/datadog-agent
82-
RUN openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
83-
-subj "/CN=agent-data-plane" \
84-
-keyout /tmp/ipc_key.pem -out /tmp/ipc_cert.pem && \
85-
cat /tmp/ipc_cert.pem /tmp/ipc_key.pem > /etc/datadog-agent/ipc_cert.pem && \
86-
rm -f /tmp/ipc_cert.pem /tmp/ipc_key.pem && \
87-
touch /etc/datadog-agent/auth_token
88-
ENTRYPOINT ["/entrypoint.sh"]
89-
CMD ["run"]
90-
91-
# ---------------------------------------------------------------------------
92-
# Runtime: Datadog Agent normative lane.
71+
# Runtime: converged Datadog Agent with our instrumented ADP embedded. Both
72+
# comparison lanes run this image. ADP is toggled on or off per lane via
73+
# DD_DATA_PLANE_ENABLED in the compose.
9374
# ---------------------------------------------------------------------------
9475
FROM ${DD_AGENT_IMAGE} AS datadog-agent
9576
ENV NO_COLOR=1
9677
COPY --chmod=755 test/antithesis/scenarios/differential/agent-config-init.sh /etc/cont-init.d/00-differential-agent-config.sh
78+
# Overlay our instrumented ADP over the release binary the converged image ships,
79+
# so the embedded data plane carries Antithesis coverage. Expose its symbols.
80+
COPY --from=adp-builder /usr/local/bin/agent-data-plane /opt/datadog-agent/embedded/bin/agent-data-plane
81+
RUN mkdir -p /symbols && ln -s /opt/datadog-agent/embedded/bin/agent-data-plane /symbols/agent-data-plane
82+
# Bring the container down on an ADP crash instead of respawning, so a SIGABRT is
83+
# recorded as a fault rather than masked by an s6 restart loop.
84+
COPY --chmod=755 test/antithesis/data-plane-finish /etc/services.d/data-plane/finish
9785

9886
# ---------------------------------------------------------------------------
9987
# Runtime: antithesis-intake.
@@ -104,6 +92,21 @@ ENV DD_HOSTNAME=antithesis-differential
10492
COPY --from=tools-builder /usr/local/bin/antithesis-intake /usr/local/bin/antithesis-intake
10593
ENTRYPOINT ["/usr/local/bin/antithesis-intake"]
10694

95+
# ---------------------------------------------------------------------------
96+
# Fetch the Datadog Agent debug symbols. The release binary is stripped; the
97+
# DWARF is published as a separate build-id-matched `-dbg` package, so Antithesis
98+
# can symbolize the off-the-shelf Core Agent without touching the agent image.
99+
# ---------------------------------------------------------------------------
100+
FROM build-base AS agent-symbols
101+
# renovate: datasource=docker depName=registry.datadoghq.com/agent versioning=docker
102+
ARG DD_AGENT_VERSION=7.81.0
103+
ARG TARGETARCH=amd64
104+
RUN curl -fsSL -o /tmp/agent-dbg.deb \
105+
"https://s3.amazonaws.com/apt.datadoghq.com/pool/d/da/datadog-agent-dbg_${DD_AGENT_VERSION}-1_${TARGETARCH}.deb" && \
106+
dpkg-deb --raw-extract /tmp/agent-dbg.deb /tmp/dbg && \
107+
mkdir -p /agent-symbols && \
108+
cp /tmp/dbg/opt/datadog-agent/.debug/opt/datadog-agent/bin/agent/agent.dbg /agent-symbols/agent.dbg
109+
107110
# ---------------------------------------------------------------------------
108111
# Runtime: differential workload client.
109112
# ---------------------------------------------------------------------------
@@ -120,5 +123,9 @@ COPY --from=tools-builder --chmod=755 /usr/local/bin/parallel_driver_send_dogsta
120123
COPY --from=tools-builder --chmod=755 /usr/local/bin/eventually_differential_contexts /opt/antithesis/test/v1/main/eventually_differential_contexts
121124
COPY --from=tools-builder --chmod=755 /usr/local/bin/finally_differential_contexts /opt/antithesis/test/v1/main/finally_differential_contexts
122125
COPY --from=tools-builder --chmod=755 /usr/local/bin/eventually_adp_alive /opt/antithesis/test/v1/main/eventually_adp_alive
126+
# Datadog Agent debug symbols. Antithesis scans /symbols in every pushed image and
127+
# matches by build-id, so this symbolizes the Core Agent running in the SUT lanes.
128+
RUN mkdir -p /symbols
129+
COPY --from=agent-symbols /agent-symbols/agent.dbg /symbols/agent.dbg
123130
COPY --chmod=755 test/antithesis/scenarios/differential/workload/entrypoint.sh /entrypoint.sh
124131
ENTRYPOINT ["/entrypoint.sh"]

‎test/antithesis/scenarios/differential/README.md‎

Lines changed: 11 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,24 @@
11
# Differential scenario
22

3-
This scenario tests that ADP and the Datadog Agent emit the same metric contexts
4-
for the same DogStatsD input. Values for contexts are not compared.
3+
This scenario tests that the Datadog Agent emits the same metric contexts for the
4+
same DogStatsD input whether its embedded data plane (ADP) is off or on. Values
5+
for contexts are not compared.
56

67
## How it works
78

89
This scenario comprises the following components:
910

10-
* Datadog Agent
11-
* ADP
11+
* Datadog Agent, ADP-off
12+
* Datadog Agent, ADP-on
1213
* `intake`
1314
* parallel drivers
1415

15-
The Datadog Agent and ADP are the systems under test. They are driven from the
16-
same, equivalently applicable configuration. That is, a configuration option
17-
that only affects ADP will not be present, as an example.
16+
Both lanes are the same converged Datadog Agent image, differing only in whether
17+
`DD_DATA_PLANE_ENABLED` hands DogStatsD to ADP. They are the systems under test,
18+
driven from the same, equivalently applicable configuration. That is, a
19+
configuration option that only affects one lane will not be present, as an
20+
example. References below to "ADP" mean the data-plane-on lane and "Datadog
21+
Agent" the data-plane-off lane.
1822

1923
The drivers emit into both SUTs. We take as 'transmitted' that the send has
2024
reached kernel buffers and is 'durable'. This is the pattern we advertise to

‎test/antithesis/scenarios/differential/adp/entrypoint.sh‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎test/antithesis/scenarios/differential/agent-config-init.sh‎

Lines changed: 0 additions & 11 deletions
This file was deleted.

0 commit comments

Comments
 (0)