Skip to content

Commit 0eda53e

Browse files
authored
feat(cli): add offline block-building benchmark sub-command (lambdaclass#595)
## 🗒️ Description / Motivation Adds `ethlambda benchmark synthetic` — an offline harness that measures block building **exactly as executed when the node proposes**, against a reproducible synthetic workload, with no devnet required. Second of three (design doc → **this** → comparable reports). What lands here is the smallest thing that runs: the real proposer path, driven deterministically, reporting one row per measured iteration. Aggregate statistics, build provenance and machine-readable output follow in the next PR, so this one can be reviewed for *what it measures* rather than how it formats. > **Stacked on lambdaclass#591**, which adds the `node`/`benchmark` token dispatch this uses. Review > that first; the base moves to `main` once it lands. ``` $ make bench Block-building benchmark — synthetic workload (mock crypto) validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42 enable_proposer_aggregation=false max_attestations_per_block=3 ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 os=macos arch=aarch64 threads=14 iter compact select_payloads stf_simulate overhead wall root 1 0.000ms 0.002ms 0.015ms 0.068ms 0.085ms 0x7282cc99 2 0.000ms 0.002ms 0.015ms 0.066ms 0.083ms 0xb9065af0 3 0.000ms 0.002ms 0.015ms 0.064ms 0.081ms 0x303f6b0f ``` ## How to read this In this order — each piece is understandable without the next: 1. **`corpus.rs`** — the workload. Deterministic validators, a genesis store over `InMemoryBackend`, and `seed_pool`, which fills the pending pool for one slot and reports how many entries the next build will see. 2. **`build_one_slot`** (`mod.rs`) — one slot end to end: seed, time the build, import the block. Warmup and measured slots run this same path; only whether the sample is kept differs, so there is no "am I warming up?" branching inside. 3. **`PhaseTimer`** (`mod.rs`) — `start()` before the build, `finish()` after. Two readings of the existing phase histogram; the difference between their sample *sums* is the build's phase time, so nothing is added to the hot path. 4. **`run_synthetic`** (`mod.rs`) — validate, set up, loop over slots, report. 46 lines. 5. **`report.rs`** — the types and the per-iteration table. ## What Changed | File | Change | |------|--------| | `bin/ethlambda/src/benchmark/mod.rs` | Harness driver: clap options + validation, `run_synthetic`'s slot loop, `build_one_slot` for one slot's work, and `PhaseTimer` for per-phase attribution | | `bin/ethlambda/src/benchmark/corpus.rs` | Seeded synthetic corpus: genesis store over `InMemoryBackend`, deterministic pubkeys via splitmix64, and per-slot pool seeding in fixed insertion order, which also rejects a batch the pool would evict whole | | `bin/ethlambda/src/benchmark/report.rs` | Params/Environment/Sample types and the human-readable per-iteration table | | `bin/ethlambda/src/command.rs` | `benchmark` joins `node` as a second clap sub-command, so clap lists it in `--help` and names it in its own usage lines. The node payload becomes `Box<CliOptions>` now that a much smaller variant sits beside it | | `bin/ethlambda/src/main.rs` | `main` becomes synchronous and dispatches; only the node path enters the tokio runtime (`run_node` carries the `#[tokio::main]` attributes). Benchmark logs go to stderr at WARN so the report owns stdout | | `crates/storage/{lib,store}.rs` | Export `NEW_PAYLOAD_CAP` so the harness rejects a `--proofs-per-data` batch the pending pool would evict whole | | `Makefile` | `make bench` (override `BENCH_ARGS` to customize) | ## Correctness / Behavior Guarantees - **`cli.rs` is not touched by this PR and the node runtime is unchanged.** The harness arguments live in their own `Args` group; the node's stay plain `PathBuf`/`String`, so clap keeps emitting its own missing-argument errors. - **It measures the production path**, not a copy: the harness enters through `produce_block_with_signatures`, the same function `BlockChainServer::propose_block` calls, and seeds the *pending* pool so the proposal tick promotes it exactly as on a live node. - **Determinism:** same seed + params → identical per-iteration block roots. Verified across repeated runs; the roots are printed so a baseline-vs-optimized diff proves an optimization changed only speed, not attestation selection. The harness never reads the wall clock into results. - **Exact phase attribution with zero hot-path changes:** per-iteration `select_payloads`/`compact`/`stf_simulate` come from the sample sums of the existing `lean_block_proposal_attestation_build_phase_seconds` histogram, deltaed between iterations, with a per-phase assertion that the count advanced by exactly one. `overhead` is the clamped remainder of wall minus the phases. - The benchmark never starts the tokio runtime, so it cannot park a worker thread for the duration of a CPU-bound run. ## Tests Added / Run - `corpus.rs`: participant groups partition every validator; synthetic pubkeys are deterministic for a seed. - `command.rs`: the benchmark token parses with no node argument, rejects node flags, and its usage line names the sub-command. - Verified by hand: `make bench`; identical block-root sequences across two runs at the same seed; `ethlambda --genesis config.yaml` still failing with clap's own missing-argument list. - `make fmt`, `make lint`, `make test` (576 tests, 30 suites) — all clean. ## Related Issues / PRs - Stacked on lambdaclass#591; design doc in the accompanying docs PR - Followed by the comparable-reports PR - Splits the now-closed lambdaclass#497 / lambdaclass#593 - Related to lambdaclass#465 ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `make test` (`cargo test --workspace --profile release-fast`) — all passing
1 parent d54044c commit 0eda53e

11 files changed

Lines changed: 705 additions & 22 deletions

File tree

Cargo.lock

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Makefile

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
.PHONY: help fmt lint docker-build shadow-build shadow-docker-build run-devnet test docs docs-deps docs-serve
1+
.PHONY: help fmt lint bench docker-build shadow-build shadow-docker-build run-devnet test docs docs-deps docs-serve
22

33
help: ## 📚 Show help for each of the Makefile recipes
44
@grep -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'
@@ -14,6 +14,11 @@ test: leanSpec/fixtures ## 🧪 Run all tests
1414
# signature verification/aggregation, without paying for LTO on every rebuild
1515
cargo test --workspace --profile release-fast
1616

17+
BENCH_ARGS ?= synthetic --mock-crypto
18+
19+
bench: ## 🏁 Benchmark block building offline (override BENCH_ARGS to customize)
20+
cargo run --release --bin ethlambda -- benchmark $(BENCH_ARGS)
21+
1722
GIT_COMMIT=$(shell git rev-parse HEAD)
1823
GIT_BRANCH=$(shell git rev-parse --abbrev-ref HEAD)
1924
DOCKER_TAG?=local

bin/ethlambda/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ shadow-integration = ["ethlambda-crypto/shadow-integration"]
2121
[dependencies]
2222
ethlambda-blockchain.workspace = true
2323
ethlambda-crypto.workspace = true
24+
ethlambda-metrics.workspace = true
2425
ethlambda-network-api.workspace = true
2526
ethlambda-p2p.workspace = true
2627
ethlambda-types.workspace = true
Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
//! Synthetic benchmark corpus: deterministic validators, a genesis store, and
2+
//! per-slot attestation-pool seeding.
3+
4+
use std::sync::Arc;
5+
6+
use ethlambda_blockchain::store::produce_attestation_data;
7+
use ethlambda_storage::{Store, backend::InMemoryBackend};
8+
use ethlambda_types::{
9+
attestation::{AggregationBits, HashedAttestationData},
10+
block::SingleMessageAggregate,
11+
state::{State, Validator, ValidatorPubkeyBytes},
12+
};
13+
14+
/// Fixed genesis time for synthetic runs. The harness derives every tick
15+
/// timestamp from slot numbers relative to this value and never reads the wall
16+
/// clock, so runs are reproducible at any time of day.
17+
const GENESIS_TIME: u64 = 1_700_000_000;
18+
19+
pub(crate) struct SyntheticCorpus {
20+
num_validators: u64,
21+
proofs_per_data: u64,
22+
}
23+
24+
impl SyntheticCorpus {
25+
pub(crate) fn new(num_validators: u64, proofs_per_data: u64) -> Self {
26+
Self {
27+
num_validators,
28+
proofs_per_data,
29+
}
30+
}
31+
32+
/// Build a genesis store over an in-memory backend with `num_validators`
33+
/// seed-derived validators.
34+
///
35+
/// Pubkeys are deterministic placeholder bytes: in mock-crypto mode no code
36+
/// path decodes them (signature verification is skipped and best-proof
37+
/// compaction never resolves pubkeys).
38+
pub(crate) fn genesis_store(&self, seed: u64) -> Store {
39+
let mut rng_state = seed;
40+
let validators = (0..self.num_validators)
41+
.map(|index| Validator {
42+
attestation_pubkey: synthetic_pubkey(&mut rng_state),
43+
proposal_pubkey: synthetic_pubkey(&mut rng_state),
44+
index,
45+
})
46+
.collect();
47+
let genesis_state = State::from_genesis(GENESIS_TIME, validators);
48+
Store::from_anchor_state(Arc::new(InMemoryBackend::new()), genesis_state)
49+
}
50+
51+
/// Seed the pending ("new") pool with the full validator set's attestations
52+
/// for `attestation_slot`, split into `proofs_per_data` disjoint aggregates.
53+
///
54+
/// Mirrors what committee aggregators gossip during a slot: several
55+
/// aggregates for the same `AttestationData`, each covering a validator
56+
/// subset. The proposal tick then promotes them to the known pool, exactly
57+
/// as on a live node. Entries are inserted in a fixed order because pool
58+
/// insertion order pins within-entry proof choice during selection.
59+
///
60+
/// Returns the total number of pool entries the next build will see, across
61+
/// both pools.
62+
pub(crate) fn seed_pool(
63+
&self,
64+
store: &mut Store,
65+
attestation_slot: u64,
66+
) -> eyre::Result<usize> {
67+
let data = produce_attestation_data(store, attestation_slot);
68+
let entries = participant_groups(self.num_validators, self.proofs_per_data)
69+
.into_iter()
70+
.map(|participants| {
71+
(
72+
HashedAttestationData::new(data.clone()),
73+
SingleMessageAggregate::empty(participants),
74+
)
75+
})
76+
.collect();
77+
store.insert_new_aggregated_payloads_batch(entries);
78+
79+
// The pending pool evicts whole data-root entries FIFO once its proof
80+
// cap is exceeded, so an over-cap batch seeds nothing and every
81+
// measured block would come out empty.
82+
let pending = store.new_aggregated_payloads_count();
83+
eyre::ensure!(
84+
pending > 0,
85+
"attestations seeded for slot {attestation_slot} were evicted from the pending pool; \
86+
the measured workload would not match the requested parameters"
87+
);
88+
Ok(pending + store.known_aggregated_payloads_count())
89+
}
90+
}
91+
92+
/// Partition validators 0..num_validators into `groups` disjoint bitfields,
93+
/// assigning validator `i` to group `i % groups`. Every group is non-empty
94+
/// (groups is capped at the validator count) and the union covers every
95+
/// validator exactly once.
96+
fn participant_groups(num_validators: u64, groups: u64) -> Vec<AggregationBits> {
97+
let groups = groups.clamp(1, num_validators);
98+
(0..groups)
99+
.map(|group| {
100+
let mut bits = AggregationBits::with_length(num_validators as usize)
101+
.expect("validator count is within the bitlist limit");
102+
for index in (group..num_validators).step_by(groups as usize) {
103+
bits.set(index as usize, true)
104+
.expect("index is within the bitlist length");
105+
}
106+
bits
107+
})
108+
.collect()
109+
}
110+
111+
/// splitmix64: tiny deterministic generator for placeholder pubkey bytes,
112+
/// avoiding a rand dependency.
113+
fn splitmix64(state: &mut u64) -> u64 {
114+
*state = state.wrapping_add(0x9e37_79b9_7f4a_7c15);
115+
let mut z = *state;
116+
z = (z ^ (z >> 30)).wrapping_mul(0xbf58_476d_1ce4_e5b9);
117+
z = (z ^ (z >> 27)).wrapping_mul(0x94d0_49bb_1331_11eb);
118+
z ^ (z >> 31)
119+
}
120+
121+
fn synthetic_pubkey(rng_state: &mut u64) -> ValidatorPubkeyBytes {
122+
let mut bytes = [0u8; 52];
123+
for chunk in bytes.chunks_mut(8) {
124+
let word = splitmix64(rng_state).to_le_bytes();
125+
chunk.copy_from_slice(&word[..chunk.len()]);
126+
}
127+
bytes
128+
}
129+
130+
#[cfg(test)]
131+
mod tests {
132+
use super::*;
133+
use ethlambda_types::attestation::validator_indices;
134+
135+
#[test]
136+
fn participant_groups_partition_all_validators() {
137+
for (validators, groups) in [(8u64, 2u64), (8, 3), (5, 8), (1, 1), (4096, 4)] {
138+
let partition = participant_groups(validators, groups);
139+
assert_eq!(partition.len() as u64, groups.min(validators));
140+
let mut seen = vec![0u32; validators as usize];
141+
for bits in &partition {
142+
let indices: Vec<u64> = validator_indices(bits).collect();
143+
assert!(!indices.is_empty(), "every group must be non-empty");
144+
for index in indices {
145+
seen[index as usize] += 1;
146+
}
147+
}
148+
assert!(
149+
seen.iter().all(|&count| count == 1),
150+
"every validator must appear in exactly one group: {seen:?}"
151+
);
152+
}
153+
}
154+
155+
#[test]
156+
fn synthetic_pubkeys_are_deterministic() {
157+
let mut a = 42u64;
158+
let mut b = 42u64;
159+
assert_eq!(synthetic_pubkey(&mut a), synthetic_pubkey(&mut b));
160+
let mut c = 43u64;
161+
assert_ne!(synthetic_pubkey(&mut a), synthetic_pubkey(&mut c));
162+
}
163+
}

0 commit comments

Comments
 (0)