DiffEq -- Only context

This draft PR introduces a new differential equivalence scenario focused
only on context totals for now. There's enough change in here that I will
pull alterations down-stack, more to come later.

Author: blt

Cargo.lock

@@ -35,6 +35,7 @@ members = [
   "lib/stringtheory",
   "test/antithesis/harness",
   "test/antithesis/intake",
+  "test/antithesis/scenarios/differential",
   "test/antithesis/scenarios/general",
 ]
 resolver = "2"

Cargo.toml

@@ -19,6 +19,9 @@ export ADP_APP_VERSION_AUTO := $(shell cat bin/agent-data-plane/Cargo.toml | gre
 export ADP_APP_VERSION := $(or $(ADP_APP_VERSION),$(ADP_APP_VERSION_AUTO))
 export ADP_APP_BUILD_TIME := $(APP_BUILD_TIME)
 
+# Datadog Agent version used by the project-wide comparison image.
+export DATADOG_AGENT_VERSION := $(shell cat .datadog-agent-version)
+
 # SPDX license-list-data tag used by package-adp-host to harvest THIRD-PARTY-* license texts.
 # Pinned to match docker/Dockerfile.agent-data-plane so the host-built tarball ships the same
 # set of license texts as the linux Docker artifact; bump in lockstep with the Dockerfile.
@@ -201,6 +204,7 @@ build-datadog-agent-image: build-adp-image ## Builds the converged Datadog Agent
 	@docker build \
 		--tag saluki-images/datadog-agent:testing-devel \
 		--tag local.dev/saluki-images/datadog-agent:testing-devel \
+		--build-arg "DD_AGENT_VERSION=$(DATADOG_AGENT_VERSION)-jmx" \
 		--build-arg ADP_IMAGE=saluki-images/agent-data-plane:testing-devel \
 		--file ./docker/Dockerfile.datadog-agent \
 		.
@@ -211,6 +215,7 @@ build-datadog-agent-image-release: build-adp-image-release ## Builds the converg
 	@docker build \
 		--tag saluki-images/datadog-agent:testing-release \
 		--tag local.dev/saluki-images/datadog-agent:testing-release \
+		--build-arg "DD_AGENT_VERSION=$(DATADOG_AGENT_VERSION)-jmx" \
 		--build-arg ADP_IMAGE=saluki-images/agent-data-plane:testing-release \
 		--file ./docker/Dockerfile.datadog-agent \
 		.
@@ -753,6 +758,8 @@ endif
 
 ANTITHESIS_CONFIG_DIR := test/antithesis/scenarios/general
 ANTITHESIS_COMPOSE_FILE := $(ANTITHESIS_CONFIG_DIR)/docker-compose.yaml
+ANTITHESIS_DIFFERENTIAL_CONFIG_DIR := test/antithesis/scenarios/differential
+ANTITHESIS_DIFFERENTIAL_COMPOSE_FILE := $(ANTITHESIS_DIFFERENTIAL_CONFIG_DIR)/docker-compose.yaml
 
 .PHONY: check-antithesis-tools
 check-antithesis-tools:
@@ -765,12 +772,23 @@ antithesis-build: ## Builds the Antithesis harness container images
 	@echo "[*] Building Antithesis harness images..."
 	@docker compose -f $(ANTITHESIS_COMPOSE_FILE) build
 
+.PHONY: antithesis-build-differential
+antithesis-build-differential: build-datadog-agent-image-release ## Builds the differential Antithesis harness images
+	@echo "[*] Building differential Antithesis harness images..."
+	@docker compose -f $(ANTITHESIS_DIFFERENTIAL_COMPOSE_FILE) build
+
 .PHONY: antithesis-validate
 antithesis-validate: check-antithesis-tools antithesis-build
 antithesis-validate: ## Validates the Antithesis harness: builds images, runs 'snouty validate'
 	@echo "[*] Validating Antithesis harness with snouty..."
 	@snouty validate $(ANTITHESIS_CONFIG_DIR)
 
+.PHONY: antithesis-validate-differential
+antithesis-validate-differential: check-antithesis-tools antithesis-build-differential
+antithesis-validate-differential: ## Validates the differential Antithesis harness
+	@echo "[*] Validating differential Antithesis harness with snouty..."
+	@snouty validate $(ANTITHESIS_DIFFERENTIAL_CONFIG_DIR)
+
 ##@ Profiling
 
 .PHONY: profile-run-blackhole

Makefile

@@ -39,6 +39,15 @@ testing. Antithesis will not run any test commands until it receives this event.
   (`antithesis-scenario-general`) owns general-only test commands under
   `src/bin/`. Snouty will push tagged images, consume this directory, and
   launch the run.
+- `scenarios/differential/` — the A/B scenario that runs ADP and the Datadog
+  Agent side by side against one shared sampled config and asserts they emit the
+  same metric context inventory for an identical DogStatsD stream. The Agent is
+  normative. Its layout mirrors `general/`: `Dockerfile`, `docker-compose.yaml`,
+  per-service build inputs, and a `README.md` describing the oracle and the
+  private control API. Its Cargo package (`antithesis-scenario-differential`)
+  owns the differential test commands under `src/bin/`. Build and validate it
+  with `make antithesis-build-differential` and `make
+  antithesis-validate-differential`.
 
 **scratchbook**
 

test/antithesis/AGENTS.md

@@ -1,18 +1,18 @@
-//! Antithesis `first_` command: sample this timeline's `datadog.yaml` and release ADP.
+//! Antithesis `first_` command: sample this timeline's `datadog.yaml` and release blocked targets.
 //!
 //! Runs once per execution path after `setup_complete`, so the sample (see
 //! [`harness::config`], Antithesis SDK randomness) is a post-snapshot, per-timeline
 //! decision Antithesis branches. Writes the config to the shared `agent-config`
-//! volume then a `ready` sentinel the blocked ADP entrypoint waits on; running
-//! upstream of ADP's boot is what makes each timeline boot under its own config.
+//! volume then a `ready` sentinel the blocked target entrypoints wait on; running
+//! upstream of target boot is what makes each timeline boot under its own config.
 //! Deployment fields come from the environment (see [`Cli`]).
 
 use std::fs;
 use std::path::PathBuf;
 
 use antithesis_sdk::prelude::*;
 use antithesis_sdk::random::AntithesisRng;
-use anyhow::Context as _;
+use anyhow::Context;
 use clap::Parser;
 use harness::config::{ConfigProfile, DatadogConfig};
 use rand::rand_core::UnwrapErr;
@@ -23,7 +23,7 @@ use serde_json::json;
 #[command(name = "first_sample_config")]
 struct Cli {
     /// Directory to write `datadog.yaml` and the `ready` sentinel into (shared
-    /// `agent-config` volume; the ADP container reads it).
+    /// `agent-config` volume; blocked target containers read it).
     #[arg(long, env = "CONFIG_DIR", default_value = "/agent-config")]
     config_dir: PathBuf,
     /// Which `datadog.yaml` variation to sample. The differential scenario sets
@@ -72,7 +72,7 @@ fn main() -> anyhow::Result<()> {
     let details = serde_json::to_value(&config).unwrap_or_else(|e| json!({ "serialize_error": e.to_string() }));
     assert_reachable!("first_sample_config.config_sampled", &details);
 
-    // Release ADP: it blocks on this sentinel, then boots under the config above.
+    // Release blocked targets: they wait on this sentinel, then boot under the config above.
     let ready_path = cli.config_dir.join("ready");
     fs::write(&ready_path, b"ready\n").with_context(|| format!("write sentinel {}", ready_path.display()))?;
     Ok(())

test/antithesis/harness/src/bin/first_sample_config/main.rs

@@ -239,7 +239,7 @@ const MAX_STRING_INTERNER_ENTRIES: u64 = 8_388_608;
 /// usually realistic but rarely tiny or wild to probe the truncation edge. A
 /// sampled `0` leaves ADP no room past the 4-byte length prefix, so it drops
 /// every packet before decode — useful when ADP runs alone, but it would make
-/// the differential targets diverge for a reason the oracle is not testing.
+/// the differential targets diverge for a reason the scenario is not testing.
 fn sample_buffer_size<R: Rng + ?Sized>(rng: &mut R, profile: ConfigProfile) -> u64 {
     if profile.is_general() && rng.random_ratio(1, 16) {
         Probe::new(0, 536_870_912).sample(rng)

test/antithesis/harness/src/config.rs

@@ -1,7 +1,13 @@
 //! Shared helpers for Antithesis test commands.
 
+use std::time::Duration;
+
 pub mod config;
 #[cfg(unix)]
 pub mod driver;
 pub mod payload;
 pub mod rand;
+
+/// How long a context may take to appear on both lanes before it counts as a
+/// divergence.
+pub const ACCEPTABLE_FLUSH_DELAY: Duration = Duration::from_secs(30);

test/antithesis/harness/src/lib.rs

@@ -102,6 +102,7 @@ fn choose_message<R: Rng + ?Sized>(rng: &mut R) -> Message {
 }
 
 /// Write one `DogStatsD` message of a sampled type to `buf` at the given vibe.
+///
 /// Returns the packed value count when a multi-value metric was emitted, else
 /// `None`.
 pub fn send<R: Rng + ?Sized>(rng: &mut R, buf: &mut Vec<u8>, vibe: Vibe) -> Option<usize> {

test/antithesis/harness/src/payload/dogstatsd.rs

@@ -24,6 +24,7 @@ mime = { workspace = true }
 protobuf = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true }
+stele = { workspace = true }
 tokio = { workspace = true, features = [
   "macros",
   "net",
@@ -48,3 +49,6 @@ tracing-subscriber = { workspace = true, features = [
   "std",
   "tracing-log",
 ] }
+
+[dev-dependencies]
+proptest = { workspace = true }

test/antithesis/intake/Cargo.toml

@@ -1,54 +1,76 @@
 # A property asserting intake for Datadog Agent-like artifacts
 
-This intake asserts structural and aggregation properties on payloads from
-Datadog Agent-like programs and is intended to mimic the constraints set by
-Datadog Intake API. This implementation is forked from the other intake in this
-project and may be later merged back up, although the goals to make aggregation
-assertions here is invasive.
+This intake asserts structural properties on payloads from Datadog Agent-like programs and is intended
+to mimic the constraints set by Datadog Intake API. This implementation is forked from the other intake
+in this project and may be later merged back up, although the goals to make aggregation assertions here
+are invasive.
 
-### How This Project Works
+## How this project works
 
-This document is the 'specification' for an abstract DogStatsD Agent. We assert
-that for any given input stream ADP emits to the intake data that is correctly
-shaped and, in a future update, that the aggregation model of ADP is accurate to
-the reference implementation of Datadog Agent DogStatsD.
+This document is the specification for an abstract DogStatsD Agent. We assert that for any given input
+stream ADP emits to the intake data that is correctly shaped and, in a future update, that the
+aggregation model of ADP is accurate to the reference implementation of Datadog Agent DogStatsD.
 
-# Properties
+The differential scenario adds one narrower oracle. For the same generated configuration and workload,
+ADP and the Datadog Agent must eventually report the same metric contexts.
 
-## Payloads
+## Properties
 
-The Agent emits outputs to Datadog intake endpoints as payloads. The current
-specification covers `/api/v2/series` only. The Agent also emits to
-`/api/v3/series`, which a future revision will add.
+### Payloads
 
-In this section we define properties that hold for `/api/v2/series` payloads
-irrespective of load generation profile. Precisely, a 'payload' is an HTTP
-envelope wrap around the compressed bytes of a
+The Agent emits outputs to Datadog intake endpoints as payloads. The current specification covers
+`/api/v2/series` only. The Agent also emits to `/api/v3/series`, which a future revision will add.
+
+In this section we define properties that hold for `/api/v2/series` payloads irrespective of load
+generation profile. Precisely, a payload is an HTTP envelope around the compressed bytes of a
 [`MetricPayload`](https://github.com/DataDog/agent-payload/blob/0a5f9ebbbe9c2a1f1e671467511f6189d3a3b443/proto/metrics/agent_payload.proto#L30-L72).
 
-Some properties reference rig-controlled parameters. `MaxTags(orgID)` and
-`MaxResources(orgID)` are per-org caps with defaults 100 and 500 respectively.
-
-| Number | Category      | Name                   | Description                                                    |
-|--------|---------------|------------------------|----------------------------------------------------------------|
-| Pyld01     | Envelope      | Content-Type           | `Content-Type` in `{application/x-protobuf, application/json}` |
-| Pyld02     | Envelope      | Content-Encoding       | `Content-Encoding` in `{deflate, gzip, zstd, identity}`        |
-| Pyld03     | Envelope      | API Key                | `DD-Api-Key` header present and non-empty                      |
-| Pyld05     | Bytes         | Compressed Size        | body < 500 KiB compressed                                      |
-| Pyld06     | Bytes         | Uncompressed Size      | body <= 5 MiB uncompressed                                     |
-| Pyld07     | MetricPayload | Decode                 | body decodes as v2 `MetricPayload` via `rust-protobuf`.        |
-| Pyld08     | MetricPayload | Point Count            | total points <= configured `serializer_max_series_points_per_payload` |
-| Pyld09     | MetricSeries  | Metric Non-Empty       | `MetricSeries.metric` is non-empty                             |
-| Pyld10    | MetricSeries  | Metric Length          | `len(metric) <= 350` bytes                                     |
-| Pyld11    | MetricSeries  | Metric Alphabetic      | `metric` contains at least one ASCII alphabetic char           |
-| Pyld12    | MetricSeries  | Type Enum              | `type` in `{COUNT, RATE, GAUGE}`                               |
-| Pyld13    | MetricSeries  | Tag Count              | `len(tags) <= MaxTags(orgID)`                                  |
-| Pyld14    | MetricSeries  | Tag Prefix Reserved    | no tag starts with `device:` or `dd.internal.resource:`        |
-| Pyld15    | MetricSeries  | Per-Series Point Count | `len(points) <=` configured `serializer_max_series_points_per_payload` |
-| Pyld16    | MetricSeries  | Origin Populated       | `origin.{product, category, service}` enum-valid               |
-| Pyld17    | Resource      | Host Resource Resolved | every series resolves a non-empty `(type="host")` resource and all series in a payload share one host |
-| Pyld18    | Resource      | Resource Count         | `len(resources) <= MaxResources(orgID)`                        |
-| Pyld19    | Resource      | Host Name Length       | host `name <= 255` bytes                                       |
-| Pyld20    | MetricPoint   | Value Not-NaN          | `value` is not NaN                                             |
-| Pyld21    | MetricPoint   | Timestamp Future Bound | `timestamp <= intake_now + 600s`                               |
-| Pyld22    | Bytes         | Content-Length         | `Content-Length` absent or value equals body byte count        |
+Some properties reference rig-controlled parameters. `MaxTags(orgID)` and `MaxResources(orgID)` are
+per-org caps with defaults 100 and 500 respectively.
+
+| Number | Category | Name | Description |
+| --- | --- | --- | --- |
+| Pyld01 | Envelope | Content-Type | `Content-Type` in `{application/x-protobuf, application/json}` |
+| Pyld02 | Envelope | Content-Encoding | `Content-Encoding` in `{deflate, gzip, zstd, identity}` |
+| Pyld03 | Envelope | API Key | `DD-Api-Key` header present and non-empty |
+| Pyld05 | Bytes | Compressed Size | body < 500 KiB compressed |
+| Pyld06 | Bytes | Uncompressed Size | body <= 5 MiB uncompressed |
+| Pyld07 | MetricPayload | Decode | body decodes as v2 `MetricPayload` via `rust-protobuf` |
+| Pyld08 | MetricPayload | Point Count | total points <= configured `serializer_max_series_points_per_payload` |
+| Pyld09 | MetricSeries | Metric Non-Empty | `MetricSeries.metric` is non-empty |
+| Pyld10 | MetricSeries | Metric Length | `len(metric) <= 350` bytes |
+| Pyld11 | MetricSeries | Metric Alphabetic | `metric` contains at least one ASCII alphabetic char |
+| Pyld12 | MetricSeries | Type Enum | `type` in `{COUNT, RATE, GAUGE}` |
+| Pyld13 | MetricSeries | Tag Count | `len(tags) <= MaxTags(orgID)` |
+| Pyld14 | MetricSeries | Tag Prefix Reserved | no tag starts with `device:` or `dd.internal.resource:` |
+| Pyld15 | MetricSeries | Per-Series Point Count | `len(points) <=` configured `serializer_max_series_points_per_payload` |
+| Pyld16 | MetricSeries | Origin Populated | `origin.{product, category, service}` enum-valid |
+| Pyld17 | Resource | Host Resource Resolved | every series resolves a non-empty `(type="host")` resource and all series in a payload share one host |
+| Pyld18 | Resource | Resource Count | `len(resources) <= MaxResources(orgID)` |
+| Pyld19 | Resource | Host Name Length | host `name <= 255` bytes |
+| Pyld20 | MetricPoint | Value Not-NaN | `value` is not NaN |
+| Pyld21 | MetricPoint | Timestamp Future Bound | `timestamp <= intake_now + 600s` |
+| Pyld22 | Bytes | Content-Length | `Content-Length` absent or value equals body byte count |
+
+### Differential context capture
+
+The differential scenario uses the same intake binary for both lanes:
+
+- Datadog Agent lane: `POST /api/v2/series`
+- ADP lane: `POST /api/v2/series`
+- Private control API: `GET /antithesis/metrics/agent`
+- Private control API: `GET /antithesis/metrics/adp`
+
+For context equivalence, a metric context is a `stele::MetricContext`:
+
+- metric name
+- canonical tag list
+
+The intake folds each captured metric down to its canonical `stele::MetricContext` and stores the
+deduplicated set per lane, but it does not compare them. The control API returns those context
+sets. The differential workload command fetches both sets and owns the Antithesis assertion for
+eventual equivalence.
+
+The context oracle intentionally does not assert aggregate values, sketch values, event payloads, or
+service-check payloads. Those remain covered by the normal workload generation and payload structural
+assertions rather than by the context-equivalence check.