Skip to content

Commit 82f962d

Browse files
committed
Vary lading.yaml configs under Antithesis
This commit updates the Antithesis rig to vary lading configs based on the work done in Datadog/saluki. We keep the variation simpler than what we do in saluki, consistent with the the goal of only adding complexity as we need it. For instance, we do not have variation in the sink destination yet. That will come in a later line of work.
1 parent a40ac9d commit 82f962d

11 files changed

Lines changed: 367 additions & 15 deletions

File tree

Cargo.lock

Lines changed: 15 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ members = [
1010
"lading_fuzz",
1111
"lading_payload",
1212
"lading_throttle",
13+
"test/antithesis/harness",
1314
"test/antithesis/sink",
1415
]
1516

docs/adr/009-antithesis-test-harness.md

Lines changed: 48 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -46,13 +46,15 @@ Datadog/saluki's layout, with a "general" MVP scenario of three containers:
4646
sancov coverage instrumentation.
4747
- **lading** (system under test): the real lading binary, built
4848
`--features antithesis` with sancov coverage instrumentation and
49-
`panic="abort"`, run with `--no-target --experiment-duration-infinite` against
50-
a hard-wired `lading.yaml`, a tcp generator pointed at the sink, with telemetry
51-
exposed via `--prometheus-addr` (lading requires telemetry to be configured).
52-
Faulted.
53-
- **workload** (driver): emits the Antithesis `setup_complete` signal, then
54-
idles. lading drives the load, so this container is the seam where
55-
config-variation test commands will land later.
49+
`panic="abort"`. lading reads its config once at startup and cannot be
50+
reconfigured, so its entrypoint blocks on a `ready` sentinel and then boots
51+
under the per-timeline config the harness sampled to the shared volume, run
52+
`--no-target --experiment-duration-infinite --prometheus-addr <addr>`
53+
(`--prometheus-addr` satisfies lading's telemetry requirement). A tcp generator
54+
points at the sink. Faulted.
55+
- **workload** (driver): emits the Antithesis `setup_complete` signal, then idles
56+
to host the test commands. lading drives the load itself, so the only command
57+
today is `first_sample_config`, which samples this timeline's lading config.
5658

5759
In a like manner to saluki we introduce a generic
5860
`test/antithesis/bin/launch.sh`, driven by per-scenario `launch.env`, tags
@@ -67,6 +69,25 @@ is off, is the single path both the sink and lading's bootstrap use to reach
6769
`lading/src/antithesis_hooks.rs`, referenced from `lading/src/bin/lading.rs`. It
6870
does SDK init plus a panic-reporting hook.
6971

72+
Config variation is a shared mechanism, not per-scenario code. The `harness`
73+
crate (`test/antithesis/harness/`) samples a config by building lading's own
74+
`generator::tcp::Config` from a value menu and serializing it, so the menu cannot
75+
drift from the real schema. Its `first_sample_config` command draws the
76+
structured choices from `AntithesisRng` (the SDK RNG, so each draw is a branch
77+
point Antithesis explores; `thread_rng` under Antithesis is seeded once and does
78+
not branch richly), writes the config plus a `ready` sentinel to a volume shared
79+
with the lading container, and tags the sample with `reachable!` so triage can
80+
count distinct variants. Scenarios reuse `harness` and differ only in wiring. The
81+
MVP menu varies the free axes the TCP sink already catches -- payload variant,
82+
throughput, and parallel connections -- over a fixed TCP transport; transport
83+
variation waits on a multi-protocol sink. Block size is derived from the sampled
84+
rate and connection count (not varied independently) so it stays within the
85+
divided per-connection throttle capacity. The payload `seed` is drawn from system
86+
entropy, not `AntithesisRng`: lading seeds its own PRNG from it and the docs
87+
forbid seeding a userspace RNG from SDK randomness, so payload content is opaque
88+
and effectively fixed across timelines -- acceptable because the sink asserts on
89+
bytes received, not content.
90+
7091
Key sub-decisions:
7192

7293
- **Standalone sink, not lading's blackhole, as the oracle, per constraint 3.**
@@ -82,8 +103,13 @@ Key sub-decisions:
82103
- **Three containers.** `setup_complete` and future config-variation
83104
live in a dedicated workload container rather than being owned by the faulted
84105
system under test.
85-
- **Config hard-wired for the MVP.** Config variation and test commands are
86-
deferred to the workload seam.
106+
- **Config varied per timeline, sampled from lading's own types.** The shared
107+
`harness` builds `tcp::Config` and serializes it, drawing the structured
108+
choices from `AntithesisRng`. Sampling is a post-`setup_complete` `first_`
109+
command so Antithesis branches each choice per timeline. Unit tests assert
110+
every sampled config re-deserializes as a valid lading config and that the
111+
block size never exceeds the divided per-connection throttle capacity, so the
112+
menu cannot silently drift from the schema or produce a discard-spin config.
87113

88114
## Alternatives Considered
89115

@@ -113,10 +139,22 @@ Rejected: the faulted system under test would own the setup signal, and
113139
config-variation test commands would have no home. A dedicated workload
114140
container keeps those concerns separate.
115141

142+
### Sample the config in lading's entrypoint at boot
143+
144+
Rejected. `snouty validate` does not execute `first_` commands, so with the
145+
sentinel approach lading stays blocked and does not boot under validate --
146+
validate still passes on `setup_complete`, exactly as saluki behaves. Sampling in
147+
lading's own entrypoint would make validate boot lading and push, but a boot-time
148+
draw is pre-`setup_complete`, which Antithesis branches less richly than a
149+
post-setup `first_` command. We chose the richer exploration; validate passing
150+
without booting the SUT is acceptable and is what saluki lives with.
151+
116152
## References
117153

118154
- `lading_antithesis/` - SDK facade over `antithesis_sdk`
119-
- `test/antithesis/` - harness (to be created)
155+
- `test/antithesis/sink/` - the sink oracle crate
156+
- `test/antithesis/harness/` - shared config-variation crate (`first_sample_config`)
157+
- `test/antithesis/scenarios/general/` - the general scenario (Dockerfile, compose, launcher inputs)
120158
- `integration/sheepdog/`, `integration/ducks/` - the mechanism this replaces
121159
- saluki `test/antithesis/` - pattern source
122160
- ADR-001: Generator-Target-Blackhole Architecture (the sink is an

test/antithesis/harness/Cargo.toml

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
[package]
2+
name = "harness"
3+
version = "0.1.0"
4+
edition = "2024"
5+
license = "MIT"
6+
publish = false
7+
description = "Shared Antithesis harness for lading scenarios: per-timeline config sampling."
8+
9+
[lib]
10+
doctest = false
11+
12+
[[bin]]
13+
name = "first_sample_config"
14+
path = "src/bin/first_sample_config.rs"
15+
16+
[lints]
17+
workspace = true
18+
19+
[dependencies]
20+
lading = { path = "../../../lading" }
21+
lading-payload = { path = "../../../lading_payload" }
22+
lading-antithesis = { workspace = true, features = ["antithesis"] }
23+
antithesis_sdk = { workspace = true, features = ["full", "rand_v0_10"] }
24+
byte-unit = { workspace = true, features = ["std"] }
25+
rand = { workspace = true, features = ["thread_rng", "std_rng"] }
26+
serde_yaml = { workspace = true }
27+
anyhow = { workspace = true }
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
//! Antithesis `first_` command: sample this timeline's lading config and release
2+
//! the blocked system-under-test.
3+
//!
4+
//! Runs once per timeline after `setup_complete`, so the `AntithesisRng` draws
5+
//! are post-snapshot decisions Antithesis branches: each timeline boots lading
6+
//! under its own sampled config. Writes the config to the shared volume, tags
7+
//! the sample for triage, then writes the `ready` sentinel last so the config is
8+
//! always present before the SUT unblocks.
9+
10+
use std::path::PathBuf;
11+
12+
use anyhow::Context as _;
13+
14+
fn main() -> anyhow::Result<()> {
15+
lading_antithesis::init();
16+
17+
let dir: PathBuf = std::env::var_os("CONFIG_DIR")
18+
.map_or_else(|| PathBuf::from("/shared"), PathBuf::from);
19+
std::fs::create_dir_all(&dir)
20+
.with_context(|| format!("create config dir {}", dir.display()))?;
21+
22+
// Draw structured choices from AntithesisRng so Antithesis branches each pick
23+
// and explores the config menu across timelines. UnwrapErr adapts the SDK's
24+
// fallible RNG to rand's infallible RngCore.
25+
let mut rng = rand::rand_core::UnwrapErr(antithesis_sdk::random::AntithesisRng);
26+
let cfg = harness::config::sample(&mut rng);
27+
let variant = harness::config::variant_label(&cfg.variant);
28+
let yaml = harness::config::to_yaml(&cfg).context("serialize sampled config")?;
29+
30+
let config_path = dir.join("lading.yaml");
31+
std::fs::write(&config_path, yaml.as_bytes())
32+
.with_context(|| format!("write {}", config_path.display()))?;
33+
34+
// Per-timeline anchor: counting these in triage shows how many distinct
35+
// variants the run explored.
36+
lading_antithesis::reachable!("first_sample_config sampled a config", { "variant": variant });
37+
38+
let ready = dir.join("ready");
39+
std::fs::write(&ready, b"ready\n")
40+
.with_context(|| format!("write sentinel {}", ready.display()))?;
41+
42+
Ok(())
43+
}
Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
//! Per-timeline lading config sampling.
2+
//!
3+
//! The sampler builds lading's real [`tcp::Config`] so the value menu is exactly
4+
//! the config schema, then serializes it under the `generator: [{ tcp: … }]`
5+
//! shape lading parses. The transport is fixed to TCP against the sink oracle,
6+
//! which counts bytes and so catches any payload variant; only the free axes
7+
//! (payload variant, rate, connections, block sizes, seed) vary. Transport
8+
//! variation waits on a multi-protocol sink.
9+
10+
use lading::generator::tcp;
11+
use lading_payload as payload;
12+
use rand::seq::IndexedRandom as _;
13+
use rand::{Rng, RngExt as _};
14+
15+
/// Fixed address of the sink oracle every sampled config targets.
16+
const SINK_ADDR: &str = "sink:9000";
17+
18+
/// Payload variants the TCP sink catches with no decoding. All are fieldless, so
19+
/// they need no extra configuration to sample.
20+
fn variant_menu() -> [payload::Config; 6] {
21+
[
22+
payload::Config::Ascii,
23+
payload::Config::Syslog5424,
24+
payload::Config::Json,
25+
payload::Config::Fluent,
26+
payload::Config::ApacheCommon,
27+
payload::Config::DatadogLog,
28+
]
29+
}
30+
31+
/// Sample a lading TCP generator config for one timeline.
32+
///
33+
/// Draws the payload variant, throughput, parallel connections, and seed from
34+
/// `rng`; the transport and target stay fixed to the sink.
35+
#[must_use]
36+
pub fn sample<R: Rng>(rng: &mut R) -> tcp::Config {
37+
// Structured choices come from the caller's rng -- AntithesisRng in
38+
// production -- so Antithesis branches each pick and sweeps the menu.
39+
let variant = variant_menu()
40+
.choose(rng)
41+
.cloned()
42+
.unwrap_or(payload::Config::Ascii);
43+
44+
let bps_mib = [1_u64, 5, 10, 50, 100].choose(rng).copied().unwrap_or(10);
45+
let bytes_per_second_bytes = bps_mib * 1024 * 1024;
46+
let bytes_per_second = Some(byte_unit::Byte::from_u64(bytes_per_second_bytes));
47+
48+
let parallel_connections = rng.random_range(1..=8_u16);
49+
50+
// Cap the block size at the smallest per-connection throttle capacity. lading
51+
// divides `bytes_per_second` evenly across `parallel_connections`; a block
52+
// larger than a worker's divided capacity is rejected by the throttle, and
53+
// the TCP worker then busy-spins discarding it with no backoff. Keeping
54+
// `maximum_block_size <= bytes_per_second / parallel_connections` guarantees
55+
// every block fits, so no timeline degrades into a discard spin.
56+
let per_connection_capacity = bytes_per_second_bytes / u64::from(parallel_connections);
57+
let maximum_block_size =
58+
byte_unit::Byte::from_u64(per_connection_capacity.clamp(1, 1024 * 1024));
59+
60+
// lading seeds its own payload PRNG from `seed`. The Antithesis docs warn
61+
// against seeding your own RNG from SDK randomness, so draw the seed from
62+
// system entropy rather than the (Antithesis) `rng`. This makes payload byte
63+
// content opaque to Antithesis and effectively fixed across timelines, which
64+
// is fine: the sink asserts on bytes received, not on content.
65+
let mut seed = [0u8; 32];
66+
rand::rng().fill_bytes(&mut seed);
67+
68+
tcp::Config {
69+
seed,
70+
addr: SINK_ADDR.to_string(),
71+
variant,
72+
bytes_per_second,
73+
maximum_block_size,
74+
maximum_prebuild_cache_size_bytes: byte_unit::Byte::from_u64(8 * 1024 * 1024),
75+
parallel_connections,
76+
throttle: None,
77+
}
78+
}
79+
80+
/// Serialize a sampled `tcp::Config` into the top-level `generator: [{ tcp: … }]`
81+
/// YAML that lading consumes.
82+
///
83+
/// # Errors
84+
///
85+
/// Returns an error if serialization fails.
86+
pub fn to_yaml(cfg: &tcp::Config) -> Result<String, serde_yaml::Error> {
87+
let mut tcp_item = serde_yaml::Mapping::new();
88+
tcp_item.insert(serde_yaml::Value::from("tcp"), serde_yaml::to_value(cfg)?);
89+
let generators = serde_yaml::Value::Sequence(vec![serde_yaml::Value::Mapping(tcp_item)]);
90+
let mut top = serde_yaml::Mapping::new();
91+
top.insert(serde_yaml::Value::from("generator"), generators);
92+
serde_yaml::to_string(&serde_yaml::Value::Mapping(top))
93+
}
94+
95+
/// Short, stable label for a payload variant, for tagging the Antithesis sample.
96+
#[must_use]
97+
pub fn variant_label(variant: &payload::Config) -> &'static str {
98+
match variant {
99+
payload::Config::Ascii => "ascii",
100+
payload::Config::Syslog5424 => "syslog5424",
101+
payload::Config::Json => "json",
102+
payload::Config::Fluent => "fluent",
103+
payload::Config::ApacheCommon => "apache_common",
104+
payload::Config::DatadogLog => "datadog_log",
105+
_ => "other",
106+
}
107+
}
108+
109+
#[cfg(test)]
110+
mod tests {
111+
use super::{sample, to_yaml};
112+
use rand::SeedableRng as _;
113+
use rand::rngs::StdRng;
114+
115+
#[test]
116+
fn sampled_config_deserializes_as_valid_lading_config() {
117+
// The load-bearing invariant: whatever we sample must be a config lading
118+
// actually accepts. Sweep many seeds so the whole menu is exercised.
119+
for s in 0..256_u64 {
120+
let mut rng = StdRng::seed_from_u64(s);
121+
let cfg = sample(&mut rng);
122+
let yaml = to_yaml(&cfg).expect("serialize sampled config");
123+
let parsed: Result<lading::config::Config, _> = serde_yaml::from_str(&yaml);
124+
assert!(
125+
parsed.is_ok(),
126+
"seed {s} produced a config lading rejects: {err:?}\n{yaml}",
127+
err = parsed.err()
128+
);
129+
}
130+
}
131+
132+
#[test]
133+
fn sampled_config_holds_invariants() {
134+
for s in 0..256_u64 {
135+
let mut rng = StdRng::seed_from_u64(s);
136+
let cfg = sample(&mut rng);
137+
assert_eq!(cfg.addr, "sink:9000");
138+
assert!((1..=8).contains(&cfg.parallel_connections));
139+
assert!(cfg.bytes_per_second.is_some());
140+
}
141+
}
142+
143+
#[test]
144+
fn block_size_never_exceeds_per_connection_capacity() {
145+
// Regression: lading divides bytes_per_second across parallel_connections,
146+
// and a block larger than a worker's divided capacity busy-spins on
147+
// discard. maximum_block_size must fit the smallest per-connection share.
148+
for s in 0..256_u64 {
149+
let mut rng = StdRng::seed_from_u64(s);
150+
let cfg = sample(&mut rng);
151+
let bps = cfg.bytes_per_second.expect("bytes_per_second set").as_u64();
152+
let per_connection = bps / u64::from(cfg.parallel_connections);
153+
assert!(
154+
cfg.maximum_block_size.as_u64() <= per_connection,
155+
"seed {s}: block {block} exceeds per-connection capacity {per_connection}",
156+
block = cfg.maximum_block_size.as_u64()
157+
);
158+
}
159+
}
160+
}

test/antithesis/harness/src/lib.rs

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
//! Shared Antithesis harness for lading scenarios.
2+
//!
3+
//! Holds the per-timeline config-variation mechanism every scenario reuses:
4+
//! [`config::sample`] draws a lading generator config from a value menu, and the
5+
//! `first_sample_config` command serializes it to the shared volume the
6+
//! system-under-test boots from. The menu is built from lading's own
7+
//! `tcp::Config`, so it cannot drift from the real config schema.
8+
9+
pub mod config;

0 commit comments

Comments
 (0)