|
| 1 | +# Improve duck: backport smp-test-target features |
| 2 | + |
| 3 | +## Background |
| 4 | + |
| 5 | +Comparison of `smp-test-target` (in the SMP repo at `testing/target/`) with lading's |
| 6 | +duck + sheepdog integration testing (`integration/ducks/` + `integration/sheepdog/`). |
| 7 | + |
| 8 | +--- |
| 9 | + |
| 10 | +## Systems at a Glance |
| 11 | + |
| 12 | +| Aspect | smp-test-target | duck + sheepdog | |
| 13 | +|---|---|---| |
| 14 | +| Control plane | CLI flags only | gRPC over Unix socket (StartTest / GetMetrics / Shutdown RPCs) | |
| 15 | +| Metrics exposure | Prometheus HTTP endpoint | gRPC `GetMetrics` RPC → structured types | |
| 16 | +| Port allocation | Fixed/configured addresses | Random ports (`127.0.0.1:0`), returned via RPC | |
| 17 | +| Protocols | HTTP/1.1, TCP, UDS (simultaneously) | HTTP, TCP, UDP (one at a time via enum) | |
| 18 | +| Failure simulation | Normal / Crash / Hang behavior modes | None | |
| 19 | +| Data quality | Not measured | Shannon entropy + DDSketch quantiles | |
| 20 | +| Tests | `proptest` timing + `insta` snapshots | None inside duck | |
| 21 | +| Orchestration | External (SMP system) | Sheepdog builds both binaries, runs load test, asserts | |
| 22 | + |
| 23 | +--- |
| 24 | + |
| 25 | +## Where duck/sheepdog is already better |
| 26 | + |
| 27 | +- **gRPC control plane** – runtime config, structured metric collection, dynamic port |
| 28 | + allocation; far more composable than CLI flags. |
| 29 | +- **Sheepdog orchestration** – true end-to-end harness; SMP's target is driven |
| 30 | + externally. |
| 31 | +- **Data quality** – Shannon entropy + DDSketch per protocol; SMP only counts bytes. |
| 32 | +- **Port isolation** – `127.0.0.1:0` is safe for parallel test runs; SMP uses fixed |
| 33 | + addresses. |
| 34 | + |
| 35 | +--- |
| 36 | + |
| 37 | +## Backports (priority order) |
| 38 | + |
| 39 | +### 1. Behavior modes — `Normal / Crash / Hang` ← most impactful |
| 40 | + |
| 41 | +smp-test-target's `behavior.rs` simulates target misbehavior so the *harness* can |
| 42 | +be tested. Duck has none of this; sheepdog never exercises lading's reaction to a |
| 43 | +failing target. |
| 44 | + |
| 45 | +**Design:** |
| 46 | + |
| 47 | +Add `BehaviorConfig` to `DucksConfig` in `integration/shared/src/lib.rs`: |
| 48 | + |
| 49 | +```rust |
| 50 | +#[derive(Debug, Clone, Serialize, Deserialize)] |
| 51 | +pub enum BehaviorConfig { |
| 52 | + Normal, |
| 53 | + Crash { delay_secs: u64 }, |
| 54 | + Hang { delay_secs: u64 }, |
| 55 | +} |
| 56 | +``` |
| 57 | + |
| 58 | +Duck's `start_test()` spawns a behavior task alongside the listener. On `Hang`, the |
| 59 | +behavior task drops its shutdown receiver so it must be explicitly aborted (mirror |
| 60 | +smp-test-target's approach of aborting the task handle after the 5-second timeout). |
| 61 | + |
| 62 | +**Files:** |
| 63 | +- `integration/shared/src/lib.rs` — add `BehaviorConfig` to `DucksConfig` |
| 64 | +- `integration/ducks/src/main.rs` — spawn behavior task; explicit abort on Hang |
| 65 | +- `integration/sheepdog/src/lib.rs` — new test cases exercising Crash/Hang |
| 66 | + |
| 67 | +**Reference:** `../single-machine-performance/testing/target/src/behavior.rs` |
| 68 | + |
| 69 | +--- |
| 70 | + |
| 71 | +### 2. Shutdown timeout |
| 72 | + |
| 73 | +smp-test-target enforces a 5-second timeout waiting for tasks to exit. Duck has no |
| 74 | +timeout guard; a stuck task will block shutdown indefinitely. |
| 75 | + |
| 76 | +**Design:** wrap duck's `JoinSet`/task-wait in |
| 77 | +`tokio::time::timeout(Duration::from_secs(5), ...)`. |
| 78 | + |
| 79 | +**Files:** |
| 80 | +- `integration/ducks/src/main.rs` — shutdown path |
| 81 | + |
| 82 | +--- |
| 83 | + |
| 84 | +### 3. Multiple simultaneous inputs |
| 85 | + |
| 86 | +smp-test-target runs HTTP + TCP + UDS together. Duck's `ListenConfig` is a |
| 87 | +single-variant enum so only one protocol can be active per test. |
| 88 | + |
| 89 | +**Design:** replace the enum with a struct: |
| 90 | + |
| 91 | +```rust |
| 92 | +pub struct ListenConfig { |
| 93 | + pub http: bool, |
| 94 | + pub tcp: bool, |
| 95 | + pub udp: bool, |
| 96 | + pub uds: Option<PathBuf>, // see backport 4 |
| 97 | +} |
| 98 | +``` |
| 99 | + |
| 100 | +**Files:** |
| 101 | +- `integration/shared/src/lib.rs` — restructure `ListenConfig` |
| 102 | +- `integration/ducks/src/main.rs` — spin up all enabled listeners |
| 103 | +- `integration/sheepdog/src/lib.rs` — update test configs |
| 104 | + |
| 105 | +--- |
| 106 | + |
| 107 | +### 4. UDS input |
| 108 | + |
| 109 | +smp-test-target has a Unix datagram socket input (`inputs/uds.rs`). Duck supports |
| 110 | +HTTP / TCP / UDP but not UDS, despite lading being able to generate UDS traffic. |
| 111 | + |
| 112 | +**Design:** add `uds_listen()` to duck; add `uds: Option<PathBuf>` to `ListenConfig` |
| 113 | +(naturally pairs with backport 3); add `SocketMetrics` for UDS to the protobuf. |
| 114 | + |
| 115 | +**Files:** |
| 116 | +- `integration/ducks/src/main.rs` — `uds_listen()` function |
| 117 | +- `integration/shared/src/lib.rs` — UDS in `ListenConfig` + `Metrics` |
| 118 | +- `integration/shared/proto/integration_api.proto` — UDS in `Metrics` message |
| 119 | + |
| 120 | +**Reference:** `../single-machine-performance/testing/target/src/inputs/uds.rs` |
| 121 | + |
| 122 | +--- |
| 123 | + |
| 124 | +### 5. Internal tests for duck |
| 125 | + |
| 126 | +smp-test-target uses `proptest` + Tokio time-pausing (`time::pause()` / |
| 127 | +`time::advance()`) to validate behavior delay timing without real sleeps. Duck has |
| 128 | +no internal tests at all. |
| 129 | + |
| 130 | +Add `proptest` tests for behavior timing once backport 1 is in place. |
| 131 | + |
| 132 | +**Files:** |
| 133 | +- `integration/ducks/src/main.rs` (or a new `behavior.rs` module) — `#[cfg(test)]` |
| 134 | + block |
| 135 | + |
| 136 | +--- |
| 137 | + |
| 138 | +## Verification |
| 139 | + |
| 140 | +``` |
| 141 | +cargo build -p ducks |
| 142 | +cargo build -p sheepdog |
| 143 | +cargo test -p sheepdog # existing tests still green |
| 144 | +cargo test -p sheepdog crash # new Crash behavior test |
| 145 | +``` |
| 146 | + |
| 147 | +New sheepdog test goal: lading exits 0 even when duck exits 1 (Crash mode). |
0 commit comments