Skip to content

Commit 726b570

Browse files
committed
chore(antithesis): Rework rig containers, setup
This commit modifies the rig so that we are always using Datadog Agent + ADP as the SUT. In the general case this is ADP-on and in the differential case this is between ADP-off and ADP-on. We make one small modification to the Datadog Agent container in that we do not restart ADP if it faults, giving antithesis a cleaner exit signal. I have tried to make all our containers sync'ed properly with the rest of the project, as they had drifted. If I understand the auto-update proecess correctly this should all stay square from now on. From Vector I have ported the launch script and, hopefully, this makes it a little more clear how to launch these things. Of interest I have changed the endpoint and turned OFF node faults for both scenarios as we cannot ensure that, especially in the differential case, that faults happen at the same instant and are fair.
1 parent 676845d commit 726b570

19 files changed

Lines changed: 409 additions & 306 deletions

File tree

Makefile

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -790,7 +790,7 @@ ifeq ($(shell command -v snouty >/dev/null || echo not-found), not-found)
790790
endif
791791

792792
.PHONY: antithesis-build
793-
antithesis-build: ## Builds the Antithesis harness container images
793+
antithesis-build: build-datadog-agent-image-release ## Builds the Antithesis harness container images
794794
@echo "[*] Building Antithesis harness images..."
795795
@docker compose -f $(ANTITHESIS_COMPOSE_FILE) build
796796

@@ -799,10 +799,10 @@ antithesis-build-differential: build-datadog-agent-image-release ## Builds the d
799799
@echo "[*] Building differential Antithesis harness images..."
800800
@docker compose -f $(ANTITHESIS_DIFFERENTIAL_COMPOSE_FILE) build
801801

802-
.PHONY: antithesis-validate
803-
antithesis-validate: check-antithesis-tools antithesis-build
804-
antithesis-validate: ## Validates the Antithesis harness: builds images, runs 'snouty validate'
805-
@echo "[*] Validating Antithesis harness with snouty..."
802+
.PHONY: antithesis-validate-general
803+
antithesis-validate-general: check-antithesis-tools antithesis-build
804+
antithesis-validate-general: ## Validates the general Antithesis harness: builds images, runs 'snouty validate'
805+
@echo "[*] Validating general Antithesis harness with snouty..."
806806
@snouty validate $(ANTITHESIS_CONFIG_DIR)
807807

808808
.PHONY: antithesis-validate-differential
@@ -811,6 +811,9 @@ antithesis-validate-differential: ## Validates the differential Antithesis harne
811811
@echo "[*] Validating differential Antithesis harness with snouty..."
812812
@snouty validate $(ANTITHESIS_DIFFERENTIAL_CONFIG_DIR)
813813

814+
.PHONY: antithesis-validate
815+
antithesis-validate: antithesis-validate-general antithesis-validate-differential ## Validates both Antithesis harnesses (general + differential)
816+
814817
##@ Profiling
815818

816819
.PHONY: profile-run-blackhole

test/antithesis/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Rendered compose snouty launches from (concrete image tags); regenerated per launch.
2+
scenarios/*/.launch/

test/antithesis/AGENTS.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,15 @@ This directory contains files relevant to running tests in Antithesis.
55
Use the `antithesis-setup` skill to scaffold and manage this directory. Use the
66
`antithesis-research` skill to analyze the system and build a property
77
catalog. Use the `antithesis-workload` skill to implement assertions and test
8-
commands. Use the `antithesis-launch` skill to build, validate, and submit
9-
Antithesis runs — do not run `snouty launch` directly.
8+
commands. Use the `antithesis-launch` skill or the `test/antithesis/bin/launch.sh` wrapper
9+
to build, validate, and submit Antithesis runs. Do not hand-type `snouty launch`.
1010

11-
**snouty launch**
11+
**launch.sh**
1212

13-
Use `snouty launch --json --webhook basic_test --config test/antithesis/scenarios/general`
14-
to start an Antithesis run. Always run `compose build` first to ensure images
15-
are up to date.
13+
`test/antithesis/bin/launch.sh <scenario>` builds the images, renders the compose
14+
with concrete tags, and submits to the `persistent_storage` webhook with the node,
15+
cpu, and clock fault profile the scenario's `launch.env` sets. See the script
16+
header for env overrides such as DURATION and WEBHOOK.
1617

1718
**snouty validate**
1819

@@ -34,7 +35,7 @@ testing. Antithesis will not run any test commands until it receives this event.
3435
`make check-all` / `make test`.
3536
- `scenarios/general/` — all Antithesis/Docker infrastructure: the `Dockerfile`,
3637
`docker-compose.yaml`, and per-container build inputs grouped by service
37-
(`scenarios/general/adp/`, `scenarios/general/workload/`). This is the directory snouty consumes as
38+
(`scenarios/general/workload/`). This is the directory snouty consumes as
3839
`--config`; it contains `docker-compose.yaml` at its top. Its Cargo package
3940
(`antithesis-scenario-general`) owns general-only test commands under
4041
`src/bin/`. Snouty will push tagged images, consume this directory, and
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
# Converged Datadog Agent boot config, shared by every scenario. first_sample_config
5+
# writes this timeline's datadog.yaml and a `ready` sentinel to the shared
6+
# agent-config volume. Block on the sentinel, then copy the config into place so
7+
# both the Core Agent and the embedded ADP boot under the sampled config.
8+
CONFIG_DIR="${AGENT_CONFIG_DIR:-/agent-config}"
9+
10+
echo "datadog-agent: waiting for ${CONFIG_DIR}/ready" >&2
11+
while [ ! -f "${CONFIG_DIR}/ready" ]; do
12+
sleep 1
13+
done
14+
15+
cp "${CONFIG_DIR}/datadog.yaml" /etc/datadog-agent/datadog.yaml

test/antithesis/bin/launch.sh

Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
#!/usr/bin/env bash
2+
# Generic Antithesis launcher shared by every scenario.
3+
#
4+
# ./launch.sh <scenario> [extra snouty flags]
5+
#
6+
# <scenario> names a directory under `test/antithesis/scenarios/` holding a
7+
# docker-compose.yaml and a launch.env. launch.env supplies the per-scenario
8+
# bits; everything else -- image tagging, property-history key, the fault profile
9+
# shape, build-before-submit -- is common and lives here so every shot is
10+
# identical and comparable and no fault flag is ever fumbled or forgotten. Change
11+
# a shot's faults by editing launch.env's node list, not by passing one-off flags.
12+
#
13+
# launch.env, sourced from the scenario directory, sets:
14+
# SCENARIO_TEST_NAME test name reported to Antithesis
15+
# SCENARIO_DESCRIPTION human description; the git commit is appended
16+
# SCENARIO_FAULT_NODES space-separated SUT container names to node-fault;
17+
# empty means no node termination/hang/throttle at all
18+
# SCENARIO_WEBHOOK optional; tenant webhook, default persistent_storage
19+
#
20+
# Required environment, read by snouty:
21+
# ANTITHESIS_TENANT tenant name
22+
# ANTITHESIS_API_KEY api key, or ANTITHESIS_USERNAME + ANTITHESIS_PASSWORD
23+
# ANTITHESIS_REPOSITORY registry to push the built config + service images to
24+
#
25+
# Optional overrides win over launch.env / defaults:
26+
# DURATION=<minutes> default 30
27+
# TEST_NAME=<name> default SCENARIO_TEST_NAME
28+
# DESCRIPTION=<text> default SCENARIO_DESCRIPTION; commit is appended
29+
# FAULT_NODES=<names> default SCENARIO_FAULT_NODES
30+
# WEBHOOK=<name> default SCENARIO_WEBHOOK or persistent_storage
31+
# SOURCE=<identifier> property-history key; default is the git branch
32+
# DRY_RUN=1 print the exact command and exit without submitting
33+
set -euo pipefail
34+
35+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
36+
ANTITHESIS_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
37+
REPO_ROOT="$(cd "$ANTITHESIS_DIR/../.." && pwd)"
38+
39+
SCENARIO="${1:?usage: launch.sh <scenario> [extra snouty flags]}"
40+
shift
41+
SCENARIO_DIR="$ANTITHESIS_DIR/scenarios/$SCENARIO"
42+
[ -d "$SCENARIO_DIR" ] || { echo "error: no scenario directory $SCENARIO_DIR" >&2; exit 1; }
43+
[ -f "$SCENARIO_DIR/docker-compose.yaml" ] || { echo "error: $SCENARIO_DIR/docker-compose.yaml not found" >&2; exit 1; }
44+
[ -f "$SCENARIO_DIR/launch.env" ] || { echo "error: $SCENARIO_DIR/launch.env not found" >&2; exit 1; }
45+
46+
# Per-scenario settings. Declared here so a missing one is caught, not silently empty.
47+
SCENARIO_TEST_NAME=""
48+
SCENARIO_DESCRIPTION=""
49+
SCENARIO_FAULT_NODES=""
50+
SCENARIO_WEBHOOK=""
51+
# shellcheck source=/dev/null
52+
. "$SCENARIO_DIR/launch.env"
53+
54+
# Immutable per-build revision: the short commit, marked -dirty when the working
55+
# tree has uncommitted changes so the tag never claims to be a clean commit it is
56+
# not. Images are tagged by this, never :latest, so a shot can never reuse a stale
57+
# mutable tag and every pushed image traces back to the source it was built from.
58+
GIT_SHA="$(git -C "$REPO_ROOT" rev-parse --short HEAD 2>/dev/null || echo unknown)"
59+
if [[ -n "$(git -C "$REPO_ROOT" status --porcelain 2>/dev/null)" ]]; then
60+
GIT_SHA="${GIT_SHA}-dirty"
61+
fi
62+
export ANTITHESIS_IMAGE_TAG="$GIT_SHA"
63+
64+
WEBHOOK="${WEBHOOK:-${SCENARIO_WEBHOOK:-persistent_storage}}"
65+
DURATION="${DURATION:-30}"
66+
TEST_NAME="${TEST_NAME:-${SCENARIO_TEST_NAME:?launch.env must set SCENARIO_TEST_NAME}}"
67+
DESCRIPTION="${DESCRIPTION:-$SCENARIO_DESCRIPTION} (commit ${GIT_SHA})"
68+
# May be empty: an empty node list means no node faults for this scenario.
69+
FAULT_NODES="${FAULT_NODES-$SCENARIO_FAULT_NODES}"
70+
71+
# Property-history key. Passing --source makes the run tracked, not ephemeral,
72+
# so findings are produced and each property's history is grouped by this key.
73+
# Default to the branch so history follows the branch; without it snouty runs
74+
# ephemeral and no findings are available to triage.
75+
SOURCE="${SOURCE:-$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)}"
76+
77+
# Pinned fault profile, submitted to the persistent_storage endpoint. cpu_mod and
78+
# clock_jitter are global and symmetric, so every scenario gets them; clock_jitter
79+
# is what exercises the AWS-LC CPU-jitter entropy path. Network faults stay on
80+
# everywhere and heal before judging.
81+
#
82+
# Node termination, hang, and throttle apply only to the containers in FAULT_NODES,
83+
# so a scenario gets none by leaving it empty. The differential A/B does: node
84+
# termination, hang, or throttle on one SUT but not the other -- or on both at
85+
# different times -- drops data unevenly and manufactures a divergence that is a
86+
# fault artifact, not an ADP defect. All three are unfair for the same reason, so
87+
# all three are gated together on FAULT_NODES.
88+
FAULTS=(
89+
--param custom.cpu_mod=true
90+
--param custom.clock_jitter=true
91+
)
92+
if [[ -n "$FAULT_NODES" ]]; then
93+
FAULTS+=(
94+
--param custom.include_for_node_termination="$FAULT_NODES"
95+
--param custom.include_for_node_hang="$FAULT_NODES"
96+
--param custom.include_for_node_throttle="$FAULT_NODES"
97+
)
98+
fi
99+
100+
for v in ANTITHESIS_TENANT ANTITHESIS_REPOSITORY; do
101+
if [[ -z "${!v:-}" ]]; then
102+
echo "error: $v is not set (required to build and submit the run)" >&2
103+
exit 1
104+
fi
105+
done
106+
107+
# Build via the Makefile's per-scenario target so the recipe -- converged base
108+
# image then scenario compose build -- lives in one place. snouty reuses a
109+
# matching tag instead of rebuilding, so without this a shot can ship stale code.
110+
# Layer caching keeps it near-instant when nothing changed.
111+
case "$SCENARIO" in
112+
general) build=(make -C "$REPO_ROOT" antithesis-build) ;;
113+
differential) build=(make -C "$REPO_ROOT" antithesis-build-differential) ;;
114+
*) echo "error: no antithesis-build make target for scenario $SCENARIO" >&2; exit 1 ;;
115+
esac
116+
117+
# Launch from a rendered copy so the image tag is concrete. snouty ships the compose
118+
# uninterpolated, so an `${ANTITHESIS_IMAGE_TAG:-latest}` tag reaches the platform as
119+
# the never-pushed `:latest`; `docker compose config` bakes in the tag snouty pushed.
120+
LAUNCH_DIR="$SCENARIO_DIR/.launch"
121+
render=(docker compose -f "$SCENARIO_DIR/docker-compose.yaml" config)
122+
123+
cmd=(snouty launch
124+
--webhook "$WEBHOOK"
125+
--config "$LAUNCH_DIR"
126+
--test-name "$TEST_NAME"
127+
--description "$DESCRIPTION"
128+
--source "$SOURCE"
129+
--duration "$DURATION"
130+
"${FAULTS[@]}"
131+
"$@")
132+
133+
printf 'build:'; printf ' %q' "${build[@]}"; printf '\n'
134+
printf 'render:'; printf ' %q' "${render[@]}"; printf ' > %q\n' "$LAUNCH_DIR/docker-compose.yaml"
135+
printf 'launch:'; printf ' %q' "${cmd[@]}"; printf '\n'
136+
if [[ "${DRY_RUN:-0}" == "1" ]]; then
137+
echo "(dry run; not building or submitting)"
138+
exit 0
139+
fi
140+
"${build[@]}"
141+
mkdir -p "$LAUNCH_DIR"
142+
"${render[@]}" >"$LAUNCH_DIR/docker-compose.yaml"
143+
exec "${cmd[@]}"

test/antithesis/data-plane-finish

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/execlineb -S1
2+
3+
# Antithesis override of the converged image's data-plane s6 `finish`.
4+
#
5+
# A clean exit code 0 means the embedded data plane was disabled, the ADP-off
6+
# lane or ADP not enabled, so stop just this service as the stock script does.
7+
# Any non-zero exit is a crash: the antithesis build runs ADP with
8+
# panic="abort", so a panic becomes SIGABRT. Bring the whole container down so
9+
# Antithesis records the fault, exactly as a standalone-ADP container death
10+
# would -- but only when the data plane is enabled. cont-init creates
11+
# /run/adp/env only in that case, so in the ADP-off reference lane a stray
12+
# non-zero ADP exit disables the service and leaves the Core Agent running
13+
# rather than tearing down a lane whose SUT is the Core Agent.
14+
ifthenelse
15+
{ s6-test ${1} -eq 0 }
16+
{
17+
foreground { /initlog.sh "agent-data-plane exited with code ${1}, disabling" }
18+
foreground { /bin/s6-svc -d /var/run/s6/services/data-plane/ }
19+
}
20+
{
21+
ifthenelse
22+
{ s6-test -d /run/adp/env }
23+
{
24+
foreground { /initlog.sh "agent-data-plane crashed, code ${1} signal ${2}, bringing container down" }
25+
s6-svscanctl -t /var/run/s6/services
26+
}
27+
{
28+
foreground { /initlog.sh "agent-data-plane exited with code ${1} while disabled, disabling" }
29+
foreground { /bin/s6-svc -d /var/run/s6/services/data-plane/ }
30+
}
31+
}

test/antithesis/harness/src/bin/first_sample_config/main.rs

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ use antithesis_sdk::prelude::*;
1414
use antithesis_sdk::random::AntithesisRng;
1515
use anyhow::Context;
1616
use clap::Parser;
17-
use harness::config::{ConfigProfile, DatadogConfig};
17+
use harness::config::DatadogConfig;
1818
use rand::rand_core::UnwrapErr;
1919
use serde_json::json;
2020

@@ -26,11 +26,6 @@ struct Cli {
2626
/// `agent-config` volume; blocked target containers read it).
2727
#[arg(long, env = "CONFIG_DIR", default_value = "/agent-config")]
2828
config_dir: PathBuf,
29-
/// Which `datadog.yaml` variation to sample. The differential scenario sets
30-
/// `CONFIG_PROFILE=differential` so both targets share a config they can both
31-
/// honor; ADP-only scenarios keep the default full feral surface.
32-
#[arg(long, env = "CONFIG_PROFILE", value_enum, default_value_t = ConfigProfile::General)]
33-
profile: ConfigProfile,
3429
/// Agent hostname written into the config. (`DD_HOSTNAME`, not the ambient
3530
/// `HOSTNAME`, so a container's own hostname does not leak in.)
3631
#[arg(long, env = "DD_HOSTNAME", default_value = "antithesis-adp")]
@@ -60,7 +55,6 @@ fn main() -> anyhow::Result<()> {
6055
&cli.api_key,
6156
&cli.dd_url,
6257
&cli.dogstatsd_socket,
63-
cli.profile,
6458
);
6559

6660
let yaml_path = cli.config_dir.join("datadog.yaml");

0 commit comments

Comments
 (0)