Commit 0eda53e
authored
feat(cli): add offline block-building benchmark sub-command (lambdaclass#595)
## 🗒️ Description / Motivation
Adds `ethlambda benchmark synthetic` — an offline harness that measures
block building
**exactly as executed when the node proposes**, against a reproducible
synthetic
workload, with no devnet required.
Second of three (design doc → **this** → comparable reports). What lands
here is the
smallest thing that runs: the real proposer path, driven
deterministically, reporting one
row per measured iteration. Aggregate statistics, build provenance and
machine-readable
output follow in the next PR, so this one can be reviewed for *what it
measures* rather
than how it formats.
> **Stacked on lambdaclass#591**, which adds the `node`/`benchmark` token dispatch
this uses. Review
> that first; the base moves to `main` once it lands.
```
$ make bench
Block-building benchmark — synthetic workload (mock crypto)
validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42
enable_proposer_aggregation=false max_attestations_per_block=3
ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 os=macos arch=aarch64 threads=14
iter compact select_payloads stf_simulate overhead wall root
1 0.000ms 0.002ms 0.015ms 0.068ms 0.085ms 0x7282cc99
2 0.000ms 0.002ms 0.015ms 0.066ms 0.083ms 0xb9065af0
3 0.000ms 0.002ms 0.015ms 0.064ms 0.081ms 0x303f6b0f
```
## How to read this
In this order — each piece is understandable without the next:
1. **`corpus.rs`** — the workload. Deterministic validators, a genesis
store over
`InMemoryBackend`, and `seed_pool`, which fills the pending pool for one
slot and
reports how many entries the next build will see.
2. **`build_one_slot`** (`mod.rs`) — one slot end to end: seed, time the
build, import the
block. Warmup and measured slots run this same path; only whether the
sample is kept
differs, so there is no "am I warming up?" branching inside.
3. **`PhaseTimer`** (`mod.rs`) — `start()` before the build, `finish()`
after. Two
readings of the existing phase histogram; the difference between their
sample *sums* is
the build's phase time, so nothing is added to the hot path.
4. **`run_synthetic`** (`mod.rs`) — validate, set up, loop over slots,
report. 46 lines.
5. **`report.rs`** — the types and the per-iteration table.
## What Changed
| File | Change |
|------|--------|
| `bin/ethlambda/src/benchmark/mod.rs` | Harness driver: clap options +
validation, `run_synthetic`'s slot loop, `build_one_slot` for one slot's
work, and `PhaseTimer` for per-phase attribution |
| `bin/ethlambda/src/benchmark/corpus.rs` | Seeded synthetic corpus:
genesis store over `InMemoryBackend`, deterministic pubkeys via
splitmix64, and per-slot pool seeding in fixed insertion order, which
also rejects a batch the pool would evict whole |
| `bin/ethlambda/src/benchmark/report.rs` | Params/Environment/Sample
types and the human-readable per-iteration table |
| `bin/ethlambda/src/command.rs` | `benchmark` joins `node` as a second
clap sub-command, so clap lists it in `--help` and names it in its own
usage lines. The node payload becomes `Box<CliOptions>` now that a much
smaller variant sits beside it |
| `bin/ethlambda/src/main.rs` | `main` becomes synchronous and
dispatches; only the node path enters the tokio runtime (`run_node`
carries the `#[tokio::main]` attributes). Benchmark logs go to stderr at
WARN so the report owns stdout |
| `crates/storage/{lib,store}.rs` | Export `NEW_PAYLOAD_CAP` so the
harness rejects a `--proofs-per-data` batch the pending pool would evict
whole |
| `Makefile` | `make bench` (override `BENCH_ARGS` to customize) |
## Correctness / Behavior Guarantees
- **`cli.rs` is not touched by this PR and the node runtime is
unchanged.** The harness
arguments live in their own `Args` group; the node's stay plain
`PathBuf`/`String`, so
clap keeps emitting its own missing-argument errors.
- **It measures the production path**, not a copy: the harness enters
through
`produce_block_with_signatures`, the same function
`BlockChainServer::propose_block`
calls, and seeds the *pending* pool so the proposal tick promotes it
exactly as on a
live node.
- **Determinism:** same seed + params → identical per-iteration block
roots. Verified
across repeated runs; the roots are printed so a baseline-vs-optimized
diff proves an
optimization changed only speed, not attestation selection. The harness
never reads the
wall clock into results.
- **Exact phase attribution with zero hot-path changes:** per-iteration
`select_payloads`/`compact`/`stf_simulate` come from the sample sums of
the existing
`lean_block_proposal_attestation_build_phase_seconds` histogram, deltaed
between
iterations, with a per-phase assertion that the count advanced by
exactly one.
`overhead` is the clamped remainder of wall minus the phases.
- The benchmark never starts the tokio runtime, so it cannot park a
worker thread for the
duration of a CPU-bound run.
## Tests Added / Run
- `corpus.rs`: participant groups partition every validator; synthetic
pubkeys are
deterministic for a seed.
- `command.rs`: the benchmark token parses with no node argument,
rejects node flags, and
its usage line names the sub-command.
- Verified by hand: `make bench`; identical block-root sequences across
two runs at the
same seed; `ethlambda --genesis config.yaml` still failing with clap's
own
missing-argument list.
- `make fmt`, `make lint`, `make test` (576 tests, 30 suites) — all
clean.
## Related Issues / PRs
- Stacked on lambdaclass#591; design doc in the accompanying docs PR
- Followed by the comparable-reports PR
- Splits the now-closed lambdaclass#497 / lambdaclass#593
- Related to lambdaclass#465
## ✅ Verification Checklist
- [x] Ran `make fmt` — clean
- [x] Ran `make lint` (clippy with `-D warnings`) — clean
- [x] Ran `make test` (`cargo test --workspace --profile release-fast`)
— all passing1 parent d54044c commit 0eda53e
11 files changed
Lines changed: 705 additions & 22 deletions
File tree
- bin/ethlambda
- src
- benchmark
- crates/storage/src
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
17 | 22 | | |
18 | 23 | | |
19 | 24 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
0 commit comments