engine_newPayload dispatched ~300ms after block receipt delays head votes on heavy blocks

[twoeths]

Summary

On heavy blocks (high gas + many blobs), Lodestar dispatches engine_newPayload to the EL only ~300 ms after receiving the gossip block — after gossip validation and part of the state transition, rather than early on receipt. Because validators cannot attest to an optimistically-imported head (notOnOptimisticBlockRoot), votability is gated on EL VALID, so this late dispatch pushes EL VALID — and the head vote — past the 4 s attestation deadline. The EL itself is fast; the addressable delay is CL-side.

This is not a one-off. We operate a large fleet of Lodestar nodes on mainnet, and the pattern is consistent across them: all of our Lodestar nodes miss the head vote on block slot 15051810 (misses are fully correlated per node — every validator on a node misses together)

Environment

• mainnet, Lodestar CL + Nethermind EL
• Slot 15051810 / EL block 25814338 / 0xc8f361…339247
• 13 blobs, 54.26M gas, 508 txs, proposer 209291 (Titan builder)
• Slot start ≈ 2026-08-23 00:42:23.0 UTC

Logs (same host, clocks aligned)

CL — Lodestar

00:42:26.566 [network] debug: Received gossip block     slot=15051810 delaySec=3.522 recvToValLatency=0.044
00:42:27.176 [chain]   debug: Verified execution payload slot=15051810 recvToValLatency=0.045 recvToValidation=0.654 validationTime=0.609
00:42:27.224 [chain] verbose: Block processed          slot=15051810 root=0xac4ec0… delaySec=4.224

EL — Nethermind

00:42:26.864 Received New Block:  25814338 (0xc8f361…)
00:42:26.869 Started pre-warming caches for block 25814338
00:42:27.141 Finished pre-warming caches for block 25814338
00:42:27.177 Valid. Result of New Block: 25814338
00:42:27.177 Processed 25814338 | 310.2 ms
00:42:27.273 Received ForkChoice: 25814338

Timeline (seconds into the slot; slot start ≈ 00:42:23.0 UTC)

into slot	where	event
3.52 s	network thread	block received — gossipsub stamps seenTimestampSec (delaySec=3.522)
3.56 s	main thread	Received gossip block — handler starts (+44 ms bridge/pickup, recvToValLatency=0.044)
3.86 s	main thread → EL	newPayload dispatched; Nethermind receives it — ~298 ms of CL work
4.18 s	EL	Nethermind Processed … 310.2 ms → VALID
4.22 s	main thread	fork-choice import → Block processed (delaySec=4.224)

The 4 s attestation deadline falls between newPayload dispatch (3.86 s) and EL VALID (4.18 s) — so the block is not votable at attestation time. Total receipt → import ≈ 0.70 s, of which only ~0.31 s is the EL; the block was already 3.52 s into the slot on arrival, leaving essentially no margin.

Analysis

1. The block is valid and was available in time for the majority. It became canonical with 60.8% of the network voting it as head; 39.2% voted the parent (a network-wide miss driven by the ~3.5 s arrival). This is not a bad/unusable block — it's a marginally-late block most of the network still handled. Lodestar's lateness is therefore a CL-side processing issue, not a block issue — and it's consistent enough that our entire Lodestar fleet lands on the losing side of these votes.
2. The EL is not the bottleneck. Nethermind executed 54.26M gas in 310 ms (~175 Mgas/s) — healthy.
3. The addressable delay is the ~298 ms CL-side window before newPayload. The block was picked up 44 ms after arrival (recvToValLatency=0.044), yet a further ~298 ms elapsed (3.56 s → 3.86 s) before the EL even received the newPayload call. Note validationTime=0.609 on the CL is not the EL time (310 ms) — roughly half of it is CL-side.
4. Votability is gated on EL VALID. Validators must not attest to an optimistically-imported head, so the block only becomes votable once the EL returns VALID (~4.18 s here). Dispatching newPayload ~300 ms late directly pushes that past the 4 s deadline.

Contributing factors to the ~298 ms (in order)

• newPayload is dispatched late in the pipeline. It fires after gossip validation + part of the state transition, instead of early on receipt so EL execution overlaps CL work. This is the primary lever. See Do early notifyNewPayload call to execution engine #6381 (Do early notifyNewPayload call to execution engine).
• Event-loop contention from concurrent data-column validation. A 13-blob block → up to 128 column sidecars validated on the main thread around the same time, stretching the block's path to newPayload. Gossip-path column KZG is verified one column at a time (validateGossipFuluDataColumnSidecar → per-column asyncVerifyCellKzgProofBatch), unlike the batched reqresp path (downloadByRange/downloadByRoot).
• JSON engine-API serialization scales with payload size. newPayload sends the full ExecutionPayload as JSON — hex-encoding 508 txs (hex doubles the bytes) + allocation/GC. jsonRpcHttpClient hard-codes Content-Type: application/json. This grows with block/blob size and is what the SSZ engine API targets — Lodestar client-side impl in feat: ssz engine API transport #9382 (open; spec ethereum/execution-apis#764 → Must match target to match head #793).

Ask

• review early newPayload (Do early notifyNewPayload call to execution engine #6381)
• SSZ transport (feat: ssz engine API transport #9382) will likely help