Skip to content

Commit d3a2ad3

Browse files
pablodeymopablodeymo
committed
docs: document the block-building benchmark
Review feedback on the first version of this PR: a `docs/plans/` file written as "this is how the benchmark was originally designed" goes stale the moment the benchmark changes, and then actively misleads. This documents what the benchmark does instead, in the mdbook alongside the other operational pages. Covers what to run and with which flags, which phases are measured and what is deliberately outside the span, how phase times are derived from the existing histogram, how to read a report, and what the block-root column is for — a root sequence that survives an optimization is the evidence that only speed changed. Two things the plan file never said, both of which a reader needs: when two reports may not be compared at all (the leanSig and leanVM revisions, and the machine fields), and what is not supported yet — real crypto, the seal phase, and replay from a datadir. Those are stated as current limitations rather than as milestones, so the page describes the tool rather than a schedule for it. The module doc in benchmark/mod.rs points at the new page.
1 parent 3d98b58 commit d3a2ad3

3 files changed

Lines changed: 141 additions & 2 deletions

File tree

‎bin/ethlambda/src/benchmark/mod.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,8 @@
66
//! and the slot-alignment sleep are outside the measured span, matching the
77
//! node's own `lean_block_building_time_seconds` boundary.
88
//!
9-
//! See docs/plans/block-building-benchmark.md for the design and roadmap
10-
//! (real-crypto pools and replay-from-datadir land in later milestones).
9+
//! See docs/benchmarking.md for what is and is not measured, how to read a
10+
//! report, and the current limitations.
1111
1212
mod corpus;
1313
mod report;

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,4 +23,5 @@
2323

2424
# Development
2525

26+
- [Benchmarking](./benchmarking.md)
2627
- [Spec Deviations](./spec_deviations.md)

‎docs/benchmarking.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Benchmarking block building
2+
3+
`ethlambda benchmark` measures block building the way the node performs it when
4+
it proposes, against a reproducible synthetic workload, with no devnet running.
5+
6+
Block building is otherwise only observable through the Prometheus histograms a
7+
live node exports. Those are noisy, depend on whatever the network happened to
8+
be doing, and cannot be diffed against a baseline — which makes them a poor
9+
instrument for the work tracked in
10+
[#465](https://github.com/lambdaclass/ethlambda/issues/465). The benchmark
11+
trades network realism for repeatability: the same parameters produce the same
12+
blocks every run, so two reports differ only where the code differs.
13+
14+
## Running it
15+
16+
```bash
17+
make bench # defaults, mock crypto
18+
BENCH_ARGS="synthetic --iterations 50" make bench
19+
```
20+
21+
`make bench` is a thin wrapper. The binary takes the same arguments directly:
22+
23+
```bash
24+
ethlambda benchmark synthetic --mock-crypto --num-validators 8 --iterations 10
25+
```
26+
27+
A default mock run finishes in well under a second, which is why CI can afford
28+
to run one on every pull request.
29+
30+
| Flag | Default | Meaning |
31+
| --- | --- | --- |
32+
| `--num-validators` | `8` | Validators in the synthetic genesis |
33+
| `--warmup-slots` | `8` | Unmeasured slots built first, so measured builds run on a state with realistic historical roots and justifications |
34+
| `--iterations` | `10` | Measured builds, one block each |
35+
| `--proofs-per-data` | `1` | Aggregates seeded per `AttestationData`, mimicking committee aggregators over disjoint validator subsets |
36+
| `--seed` | `42` | Seed for the validator set; fixes the whole run |
37+
| `--mock-crypto` | off | Placeholder proofs instead of real XMSS/leanVM signatures. **Currently required** — see [Limitations](#limitations) |
38+
| `--enable-proposer-aggregation` | off | Mirrors the node flag: collapse same-data proofs via recursive leanVM aggregation |
39+
| `--max-attestations-per-block` | `3` | Mirrors the node flag: distinct `AttestationData` per block |
40+
| `--format` | `human` | `human` or `json` |
41+
| `--output <path>` | — | Also write the JSON report to a file |
42+
43+
Logs go to stderr and the report to stdout, so `--format json` pipes straight
44+
into `jq`.
45+
46+
## What it measures
47+
48+
Each iteration enters `produce_block_with_signatures` — the same function
49+
`BlockChainServer::propose_block` calls — and the harness reports the phases
50+
inside it:
51+
52+
| Phase | Work |
53+
| --- | --- |
54+
| `select_payloads` | Choosing which attestations go in the block |
55+
| `compact` | Collapsing or picking among proofs for the same data |
56+
| `stf_simulate` | The state transition that seals `state_root` |
57+
| `overhead` | The rest of the measured span: tick processing, attestation promotion, fork-choice head, pool clone |
58+
| `wall` | The whole span |
59+
60+
`overhead` is `wall` minus the sum of the phases, so the columns add up by
61+
construction.
62+
63+
Deliberately **outside** the measured span, matching the boundary of the node's
64+
own `lean_block_building_time_seconds` metric: gossip publish, the
65+
slot-alignment sleep, and importing the block that was just built. The import
66+
still happens between iterations — otherwise every iteration would build on the
67+
same head and `process_slots` would get more expensive as the run went on.
68+
69+
Phase times come from the sample sums of the existing
70+
`lean_block_proposal_attestation_build_phase_seconds` histogram, read before and
71+
after each build. Histogram sums accumulate raw f64 seconds, so the difference
72+
between two readings is the elapsed phase time and bucket boundaries play no
73+
part. Nothing is added to the hot path for the benchmark's benefit. The harness
74+
asserts each phase was observed exactly once per build and fails the run
75+
otherwise, because a mis-attributed report is worse than no report.
76+
77+
## Reading a report
78+
79+
```
80+
Block-building benchmark — synthetic workload (mock crypto)
81+
validators=8 warmup_slots=8 iterations=10 proofs_per_data=1 seed=42
82+
enable_proposer_aggregation=false max_attestations_per_block=3
83+
ethlambda/v0.1.0/aarch64-apple-darwin/rustc-v1.97.1 leansig=15cbdd43 leanvm=e2592df4 os=macos arch=aarch64 threads=14
84+
85+
iter compact select_payloads stf_simulate overhead wall root
86+
1 0.000ms 0.002ms 0.015ms 0.068ms 0.085ms 0x7282cc99
87+
...
88+
89+
phase count min mean p50 p90 max
90+
select_payloads 10 0.002ms 0.002ms 0.002ms 0.003ms 0.003ms
91+
...
92+
```
93+
94+
Every measured iteration gets its own row, and the summary follows below it.
95+
Outliers are never discarded: XMSS signing and OTS window advancement produce
96+
legitimate heavy tails, and hiding them would misrepresent the thing being
97+
measured. A coefficient of variation above 10% is flagged so a noisy run is not
98+
mistaken for a result.
99+
100+
Percentiles are nearest-rank, without interpolation. Sample counts here are
101+
small, so an actual observed value is more informative than a blend of two
102+
neighbours.
103+
104+
The `root` column is the block root of each built block. It is what makes a
105+
before/after comparison trustworthy: if an optimization leaves the root
106+
sequence unchanged, it changed only speed and not which attestations were
107+
selected. If the roots move, the change altered block contents and the timing
108+
comparison means something different than intended.
109+
110+
## Comparing two runs
111+
112+
Same seed and same parameters produce identical root sequences, so a baseline
113+
and a candidate can be diffed directly. The header line exists to tell you when
114+
they *cannot* be compared:
115+
116+
- `leansig` and `leanvm` are the resolved revisions the binary was built
117+
against, read from `Cargo.lock` at build time. leanSig tracks a moving branch
118+
and leanVM performs the signature aggregation, so either one moving changes
119+
the measured crypto.
120+
- `os`, `arch` and `threads` change results across machines.
121+
122+
Two reports that disagree on any of those are not measuring the same thing.
123+
124+
## Limitations
125+
126+
- **`--mock-crypto` is required.** Real XMSS/leanVM pools are not wired up yet,
127+
so the run measures selection, compaction and the state transition — not
128+
signing or aggregation.
129+
- **The seal phase is not measured.** Signing, type-1 wrapping and type-2
130+
merging happen after the measured span and are not reported.
131+
- **Synthetic workloads only.** Replaying a real datadir is not implemented, so
132+
results reflect a synthetic chain rather than a deep production state.
133+
134+
## In CI
135+
136+
The Test job runs a short mock benchmark and asserts the JSON report's shape
137+
(`schema_version`, one sample per iteration). It costs seconds, and it means a
138+
change to the report contract cannot land unnoticed.

0 commit comments

Comments
 (0)