fix(node,store): take the full-table scans off the dispatch loop and repair finalization pruning

[dimka90]

Description

The dispatch loop is a single goroutine and it keeps the slot clock. Four full-table Pebble scans were running on it, all O(chain length). At ~22,600 blocks the mean tick interval reached 38–288s against an 800ms target, and every duty the clock drives — propose, attest, aggregate, head update, prune — stopped. Silently: the code that would have reported a problem was never reached.

Four commits, each buildable on its own.

5bb11d1 — finalization pruning was inert (the root of the growth)

updateFinalizedFromHead called FC.Prune before PruneOnFinalization. FC.Prune removes the finalized-below ancestors and every losing branch from the ProtoArray; PruneOnFinalization then asks that same array which roots to delete via GetCanonicalAnalysis. After the prune the parent walk terminates at the new root, so canonical has length 1 and canonical[1:] is empty, and the siblings are already gone so nonCanonical is empty. Both delete lists came back empty every time.

So TableStates and TableBlockHeaders were never pruned at all — only pruneLiveChain worked, because it scans by slot rather than going through fork choice. That is why devnet-5 held 13,184 states and ~22,600 headers after thousands of finalizations, and why the scans below got expensive enough to matter.

2c6c0da — the scans

call site	before	after
EstimateTableBytes	iterated every entry of all six tables, copying keys and values, per imported block	Pebble range metadata, sampled off the loop
StatesCount	scanned 13,184 states once per slot, for one log line	removed
MaxStoredBlockSlot	scanned and SSZ-decoded every header, up to 3×/slot	atomic.Uint64 high-water mark
BlockRoots	materialised every root into a map, per proposal	membership predicate

EstimateTableBytes changes meaning: SST bytes for the key range, not logical live bytes, and it excludes the memtable. For a size gauge that is the more useful figure; the test asserts the new contract.

MaxStoredBlockSlot deliberately still covers pending headers, not just imported ones. A live-chain index would have been cheaper but would miss a persisted pending block — a node restarting with head 100 and a pending block at 109 would read its own import lag as a network stall and resume duties on a stale head, which is the dead-fork behaviour the gate's carve-out exists to prevent. Covered by TestMaxStoredBlockSlotSeedsFromDiskAfterRestart.

84e2ebf — aggregation is no longer behind the sync-lag duty gate

gean gated aggregation as well as block production and attestation, reasoning that aggregates built on a stale view get dropped anyway. That holds with several aggregators. It inverts with one: the sole aggregator withholds the aggregates the network is waiting on at exactly the moment it is furthest behind, so nothing justifies, nothing finalizes, pruning never runs, and the lag that closed the gate gets worse.

Observed on devnet-5 as not_synced climbing to ~4,100 per node with justification frozen; stopping two lagging nodes advanced justification 1,520 slots in five minutes.

Checked against the other implementations, on the branches they actually run:

	attestation	proposal	aggregation
leanSpec timeline.py:49	—	—	not gated (case 2 if is_aggregator)
ethlambda (build/leanvm-track-main)	gated	gated	not gated
lantern	no gate	no gate	not gated
ream	gated	gated	gated

ethlambda is the informative one: it has a duty gate and applies it to two of three duties. The exclusion is deliberate, not an omission.

Work stays bounded without the gate, but only between groups: a session has a slot-anchored deadline and MaxGroupsPerSession, and the deadline is checked between proofs. It cannot interrupt a native proof already running, so one overlong proof still overruns it. The bound is "at most MaxGroupsPerSession proofs per slot, the last of which may overrun" — not a hard time limit. Correcting an overstatement in the original description.

Drops the not_synced skip reason with it, since nothing can emit it now.

478f7e3 — signals that survive a stall

lean_tick_interval_duration_seconds is observed inside onTick, so it records nothing while the loop is blocked: the failure it should report makes it go quiet rather than spike. Its top bucket is 1.6s besides, so it could not have sized a 288s stall. Adds lean_tick_last_age_seconds (written from outside the loop) and lean_dispatch_event_duration_seconds{event} with buckets to 600s, so a slow handler is attributable rather than only visible as a late tick.

Evidence

30-second CPU profile of a stalled node:

49.12% cum   internal/storage.(*pebbleIterator).Next
26.89% flat  runtime/syscall.Syscall6
24.56% cum   os.(*File).pread
24.02% flat  github.com/golang/snappy.decode

The 24% in snappy.decode is the tell — only a scan that reads values decompresses that much, which is recordTableBytes and StatesCount. A goroutine dump put the dispatch goroutine inside pebbleIterator.Next → sstable.readBlock → vfs.Prefetch → syscall, not parked on a channel.

Degradation is monotonic in chain length:

slot	tick interval	justified lag	finalized lag
~600	~0.8s	−7	−16
~15,400	degraded	−77	−126
~22,600	38–288s	frozen	frozen

Associated Issue

None — this targets fix/aggregation-skip-visibility, not main or a devnet-N branch.

Related context: #371 (recursive aggregation exceeding the slot budget, stalling finalization) describes the same symptom from a different cause; #368 (ProtoArray.Prune() O(N²)) is adjacent to the pruning path touched here but is not addressed.

Review follow-ups (8bfad59)

Two lifecycle findings from review, both fixed:

• A failed seeding scan latched permanently. scanMaxStoredBlockSlot returned 0 for an unopenable read view, and a mid-iteration failure was indistinguishable from reaching the end of the table, since Next reports false either way. sync.Once then cached that understated mark forever. Not merely a bad gauge — an understated watermark inflates the duty gate's computed network lag, which can trip the network-stall carve-out and resume duties on a stale head, the behaviour the carve-out exists to prevent. storage.Iterator grows an Err() error so a truncated scan is distinguishable from a complete one; seeding reports errors, latches only on success, and now runs at startup before dispatch — which also removes the last full-table scan from the duty path.
• The storage sampler could outlive shutdown. waitForShutdown cancels, sleeps 500ms and returns; backend.Close then runs from a defer with nothing joining the workers. A sampler mid-round calls into a closed Pebble instance, which panics. Storage-reading goroutines are now tracked on a WaitGroup and joined in main before Close.

Known gap, not addressed here

Existing database bloat is not repaired. Pruning only knows roots still in fork choice, and restart restoration anchors at the justified block (internal/node/restore.go), excluding older headers from replay. States orphaned by the pre-fix pruning bug therefore stay on disk indefinitely on an upgraded node. That wants a one-off cleanup, not another recurring scan on the dispatch path — worth its own issue. A node started from fresh data is unaffected.

Test Plan

• go test ./internal/... — 25/25 packages pass; go vet ./internal/... and gofmt clean.
• Each of the four commits was checked out individually and built, so the branch is bisectable.
• TestFinalizationPrunesAncestorAndLosingBranch (new) — builds genesis → a(1) → b(2) → c(3) with a losing fork x(2), finalizes b, asserts x is removed entirely and a's state is dropped while b's and c's survive. Verified it fails on the pre-fix ordering with all three symptoms, which is the point of it.
• TestMaxStoredBlockSlotSeedsFromDiskAfterRestart (new) — head 100 with a persisted pending block at 109, new store over the same backend, asserts 109. This is the case a live-chain index would get wrong.
• TestMaxStoredBlockSlotTracksInserts (new) — the mark rises on insert and never falls.
• TestMaxStoredBlockSlotDoesNotLatchAFailedSeed (new) — a truncated header scan must report an error and must not cache its partial answer; a later store over the same data recovers the real mark.
• TestWaitForStorageWorkersBlocksUntilSamplerReturns (new) — the wait must not return while the sampler runs, and must return once the context is cancelled. Passes under -race.
• TestPebbleEstimateTableBytes updated to the new contract: zero before flush, non-zero after, each table attributed to its own prefix range.

Devnet validation

Validated on a 15-node devnet — 4 gean, 6 ethlambda, 5 lantern, 4 committees, gean the sole aggregator. Fresh genesis, 14 hours, 12,400 slots, zero restarts.

The tick loop never slipped. max_over_time of the 10m mean tick interval across the whole run:

node	worst over 14h	same node, old build
gean_0	0.8065 s	38.80 s
gean_1	0.8052 s	287.99 s
gean_2	0.8066 s	103.28 s
gean_3	0.8066 s	68.82 s

Worst case in fourteen hours is 6.6 ms above an 800 ms target. Sampling gean_0 every 30 minutes across the run gives min 0.79998, max 0.80549, and the last sample (0.79999) is indistinguishable from the first (0.80000) — no drift, no rising tail. That matters more than the absolute number: the old build's failure mode was degradation with chain length, and 12,400 slots of growth produced no trend at all.

For reference, on the same network: lantern 0.7999–0.8001, ethlambda 0.8330–0.9261.

Pruning is demonstrably running. Before the ordering fix these counts were always zero:

04:11:01  pruning: finalized_slot=11604  states=94  blocks=23  live_chain=94  non_canonical=23
04:16:24  pruning: finalized_slot=11688  states=38  blocks=9   live_chain=38  non_canonical=9
04:41:33  pruning: finalized_slot=12057  states=82  blocks=26  live_chain=82  non_canonical=26
05:04:55  pruning: finalized_slot=12403  states=11  blocks=4   live_chain=11  non_canonical=4

blocks tracks non_canonical exactly, which is the losing-branch deletion the ordering bug prevented.

The database is an order of magnitude smaller, and smaller than ethlambda's at the same height (lean_table_bytes):

table	gean_0	ethlambda_0
states	16.2 MB	2.1 MB + 1.9 MB state_diffs
signed_blocks / block_proof	1.243 GB	1.703 GB
block_headers	0.95 MB	1.66 MB
live_chain	0.97 MB	1.11 MB
total	~1.26 GB	~1.71 GB

16 MB of states, against roughly 4.6 GB retained by the old build at a comparable height (13,184 states at ~350 KB). That retention is what made the scans fatal.

Dispatch-event breakdown (lean_dispatch_event_duration_seconds, new in this PR), gean_0 at slot 12,349:

event	mean	share of an 800 ms interval
tick	17 ms	2%
block	210 ms	26%
proposal_result	135 ms	17%
early_aggregate	2 ms	0.3%

block is the largest consumer and is the synchronous signature-verification and STF path noted below — comfortably inside budget for one block, but drainPendingBlocks can process up to 256 back-to-back.

Two growth trends observed, neither blocking. Between slot 166 and slot 12,349, tick went 6 ms → 17 ms and RSS went 2.3 GB → 3.8 GB. The first is expected: the database grew from empty to 1.26 GB, so point reads cost marginally more, and 17 ms of an 800 ms budget leaves no pressure. The second is not storage — a 16 MB states table cannot account for 3.8 GB of RSS, and ~97% of gean's RSS sits outside the Go heap in XMSS/leanVM allocations. That is a prover-memory question, separate from this PR, and worth its own issue.

Behaviour changes worth knowing about

• lean_aggregator_skipped_total{reason="not_synced"} will stop appearing. Any dashboard panel keyed on it needs updating.
• gean will now aggregate while behind. Bounded, but expect lean_aggregation_worker_total_time_seconds to rise during catch-up.
• EstimateTableBytes reports a different quantity, so the storage-size gauge will step to new values on deploy.

Checklist

• I have read the CONTRIBUTING.md (if applicable)
• I have performed a self-review of my code
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[dimka90]

Devnet validation complete — 14 hours, 12,400 slots, 15 nodes (4 gean / 6 ethlambda / 5 lantern), gean sole aggregator, zero restarts. Full detail added to the description; the headline:

The tick loop never slipped. max_over_time of the 10m mean interval across the entire run, against the same nodes on the old build:

node	worst over 14h	old build
gean_0	0.8065 s	38.80 s
gean_1	0.8052 s	287.99 s
gean_2	0.8066 s	103.28 s
gean_3	0.8066 s	68.82 s

Worst case is 6.6 ms above an 800 ms target. Sampled every 30 minutes, gean_0 runs 0.79998–0.80549 with the final sample identical to the first — no drift. That is the part that matters, since the old failure mode was degradation with chain length.

Pruning is running — states and blocks are non-zero on every finalisation and blocks tracks non_canonical exactly, where before the ordering fix both were always zero.

States table is 16 MB, against roughly 4.6 GB retained by the old build at a comparable height. Total on-disk 1.26 GB vs ethlambda's 1.71 GB.

Two trends noted and neither blocking: tick event 6 ms → 17 ms (expected as the DB grew from empty to 1.26 GB; 2% of budget), and RSS 2.3 GB → 3.8 GB. The RSS growth is not storage — a 16 MB states table cannot explain it, and ~97% of gean's RSS is outside the Go heap in prover allocations. Separate issue.

The two caveats from review still stand as written: historical database bloat is not repaired for upgraded nodes, and the aggregation deadline cannot interrupt a proof already running.