perf(aggregation): aggregates cover ~1 slot in 3; justification lag grows with run length

[dimka90]

Root cause found — this is #371

Resolved after filing. The open question below ("why do 46% of sessions produce nothing?") has an answer, and it is not in the aggregation selection logic. Aggregation proving takes 3–10 seconds against a 4-second slot.

From gean_0 logs, every session overruns and each produces exactly one aggregate:

slot=16401  produced=1  duration=10.535s  budget=1.588s
slot=16438  produced=1  duration=8.591s   budget=2.083s
slot=16417  produced=1  duration=7.693s   budget=2.144s
slot=16415  produced=1  duration=6.265s   budget=1.056s
slot=16399  produced=1  duration=6.626s   budget=2.219s
slot=16396  produced=1  duration=2.833s   budget=2.355s

A 10.5-second session spans more than two and a half slots. That single fact accounts for every symptom recorded below.

The cascade

1. Proving takes ~3–10 s, so an aggregator produces roughly one aggregate per three slots (matching the measured 300/hour against 900 slots).
2. The safe target advances only on a 2/3 supermajority of new votes at interval 3. With aggregates arriving a third as often as slots, it advances in jumps and then sits still.
3. While it sits still, GetAttestationTarget walks down from head and finds nothing above the source, so produce.go:40 clamps target up to source. The vote is valid but targets the already-justified checkpoint.

Visible directly in the produce log — same root, same slot on both sides:

slot=16427  source=0x14f852…/16409   target=0x14f852…/16409   ← clamped
slot=16432  source=0x14f852…/16409   target=0x14f852…/16409   ← clamped
slot=16433  source=0x14f852…/16409   target=0x79c590…/16430   ← safe target advanced
slot=16448  source=0x79c590…/16430   target=0x79c590…/16430   ← clamped again
slot=16455  source=0x79c590…/16430   target=0x79c590…/16430

Target sits at 16409 for six slots, jumps to 16430, sits for eight more. Every slot it sits is a slot whose votes cannot advance justification.

4. orderedGroups correctly skips those as target_justified — hence 98% of group skips and the 46% empty sessions.
5. Fewer usable votes means justification lags further, which means the safe target advances more slowly, which produces more clamped votes.

So the empty sessions are gean behaving correctly in the face of votes that cannot help. The defect is upstream: proving is three times slower than the slot rate.

Relationship to #371

This is the same ceiling #371 recorded, now measured on a 15-node network rather than a single node, and worse: #371's worst case was 3.33 s, here it reaches 10.5 s.

Closing question in favour of #371. The numbers below stay as corroborating evidence from a 15-node, 18-hour run — in particular that the cascade is not a transient and that lag grows continuously with run length.

What was ruled out along the way

• Group selection / ordering — orderedGroups filters before the session loop and sorts current-slot groups first, so stale groups never consume the session cap. Not the cause.
• Attestation verification latency — measured at 21 ms per attestation, not the ~500 ms assumed. Not the cause.
• Vote delivery — gean receives ~10 attestations/slot. Votes are arriving.
• Pool bloat, storage pressure, tick-loop starvation — all rejected, see below.

---

Summary

On an 18-hour devnet run, gean aggregators produce an aggregate roughly one slot in three per subnet, and justification lag grows steadily with run length — 20 → 27 → 47 slots. Finalisation follows it out, 174 → 240 over six hours.

Nothing is stalled and all 15 nodes agree on head, justified and finalised throughout. But the lag has widened continuously for 18 hours rather than settling, and we do not know why 46% of aggregation sessions produce nothing.

Filed as an observation with evidence, not a diagnosis. Two hypotheses were tested and both failed (see below), so please don't treat any mechanism here as established.

Setup

15 nodes on one host — 4 gean, 6 ethlambda, 5 lantern. 4 committees. All four aggregators are gean, one per subnet. Fresh genesis, ~18 hours, ~slot 16,150 at time of writing. Zero restarts. Build 4c3c927 (PR #423).

Evidence

Aggregate production

increase(lean_pq_sig_aggregated_signatures_total[1h]) — "aggregated signature proofs produced":

node	per hour
gean_0	300
gean_1	299
gean_3	296
gean_2	294

900 slots/hour, so ~1 aggregate per 3 slots per subnet. Do not compare this figure across clients — ethlambda and lantern implement their own metrics and are not aggregating.

Session outcomes

increase(lean_proof_operations_total{operation="aggregation"}[1h]), gean_0 (other three within a few percent):

result	count	share of ~864 dispatches
empty — ran, produced nothing	398	46%
success — produced ≥1 aggregate	283	33%
expired — arrived past deadline, never started	96	11%
canceled — lost the prover	87	10%
truncated — hit the budget (overlaps success/empty)	247	36% of those that ran

canceled is the proving gate yielding to block proposals, which is the intended priority and not in question. The empty share is the open question.

Group skips

increase(lean_aggregation_groups_skipped_total[1h]), gean_0:

reason	count	share
target_justified	48,301	98%
budget	987	2%
too_few_signers	139	0.3%
session_cap	65	0.1%

target_justified is correct behaviour — no value in proving a vote for an already-settled target. But at ~54 groups per slot it also means the metric is 98% benign, so a real signal in the other 2% would be invisible in the total.

The two lag trends

max_over_time((lean_head_slot - on(job) lean_latest_justified_slot)[1h:5m]):

window	max justification lag
6h ago	24–28
last hour	46–48

max_over_time((lean_head_slot - on(job) lean_latest_finalized_slot)[1h:5m]):

window	max finality gap
6h ago	174 (uniform across all 15 nodes)
last hour	233–240

Both uniform across gean, ethlambda and lantern — which is expected rather than exculpatory, since every client consumes the same gean-produced aggregates.

240 is a legitimate justifiable distance (4×240+1 = 961 = 31²), so finalisation is landing on a valid rung; the concern is that it is reaching for ever more distant rungs.

Hypotheses tested and rejected

Pool bloat feedback loop — "wider finality gap → less pruning → bigger pools → more stale groups → fewer aggregates → wider gap". Rejected: lean_gossip_signatures is 53–57 on gean nodes against a cap of 8192. The pools are small and not accumulating.

Storage pressure — rejected: lean_table_bytes gives states at 16.2 MB and the whole database at ~1.26 GB, smaller than ethlambda's 1.71 GB at the same height. Pruning is verifiably running (non-zero states/blocks in every prune log since #423).

Also ruled out as a cause: the tick loop. It held at 0.800–0.8066 for the entire 18 hours, so this is not scheduling starvation.

The open question

Why do 46% of sessions run and produce nothing?

dispatchAggregationCycle already filters out "nothing to aggregate" before dispatching (AggregatorSkipOther), so these sessions had input. The skip reasons are 98% target_justified, which is legitimate — but that should not leave nearly half of all sessions empty-handed, and the pools are too small for staleness to be the whole story.

Worth investigating:

• Whether empty sessions correlate with subnet, slot parity, or proposal duty
• Whether the ~54 target_justified groups per slot are consuming session budget before a viable group is reached — group ordering is frontier-first by ascending target slot (orderedGroups), so stale groups may be examined first
• Whether dropping groups from the pool once their target justifies, rather than re-skipping them every session, changes the outcome
• Whether expired at 11% points to dispatch latency worth measuring directly

Suggested first step

The correlation query nobody has run yet: does empty cluster in time, or is it evenly spread? If it clusters, something periodic is causing it. If it is uniform, it is structural.

Notes

• Independent of fix(node,store): take the full-table scans off the dispatch loop and repair finalization pruning #423 — the tick loop held throughout, and the old build could never have run long enough to expose this.
• Separate from perf(memory): gean holds 35 glibc malloc arenas (~2.2 GB) where ethlambda holds 1 #424 (glibc arenas / memory), which came out of the same run.
• Devnet is still up at time of filing; the Prometheus data is available for anyone who wants to query it before it comes down.