Gossip message received to validation latency

[twoeths]

Describe the bug

recvToValLatency is the root cause of majority of wrong head vote on mainnet (+ when gossip block received late into the slot)

• validate gossip block

2026-08-23 07:24:51.127	
Aug-23 00:24:51.127[network]         debug: Received gossip block slot=15051722, currentSlot=15051722, peerId=16Uiu2HAmDtzcwLDvCTyoZ7CAh1nMpC3BJUkjQRFkd4MsGSdpFK8R, delaySec=3.6410000324249268, recvToValLatency=0.48600006103515625

2026-08-23 07:33:02.940	
Aug-23 00:33:02.939[network]         debug: Received gossip block slot=15051763, currentSlot=15051763, peerId=16Uiu2HAkz8J7FpbMWAiy6Vq8S2UMNp7Sx62g3LsGLC6YGyNMjDX3, delaySec=3.2060000896453857, recvToValLatency=0.7329998016357422

Aug-23 00:54:27.149[network]         debug: Received gossip block slot=15051870, currentSlot=15051870, peerId=16Uiu2HAmNFy1RPmyskFNu1TNJ3sADdCGMjNKWkA6YSautThVVrdi, delaySec=3.434000015258789, recvToValLatency=0.7149999141693115

• verify payload

2026-08-23 08:59:27.400 | Aug-23 01:59:27.400[chain]           debug: Verified execution payload slot=15052195, recvToValLatency=0.7590000629425049, recvToValidation=1.1770000457763672, validationTime=0.4179999828338623

• validate gossip data column (fulu)

Aug-23 01:59:27.432[network]         debug: Received gossip dataColumn slot=15052195, blockRoot=0x0996…7d61, timeCreatedSec=1787450366.11, expectedColumns=128, receivedColumns=122, currentSlot=15052195, peerId=16Uiu2HAmA18CwM2hv6xDJXykbCDrnPob3p6rVx6UWKm4wTYLTyYq, delaySec=3.299999952316284, secFromSlot=4.107000112533569, gossipSubnet=92, columnIndex=92, recvToValLatency=0.8070001602172852, recvToValidation=1.1319999694824219, validationTime=0.3249998092651367

Expected behavior

• low latency

Steps to reproduce

No response

Additional context

this is not related to #9939 because it happens before notifyNewPayload() engine api

Operating system

Linux

Lodestar version or commit hash

unstable

[twoeths]

this profile was from grapebaba/forkchoice-bigint-weight, should be same to unstable

main_thread_2026-08-30T09:29:57.352Z.cpuprofile.zip

was able to catch this block

• 1st data column: sszDeserialize()=101ms, verifyDataColumnSidecarKzgProofs=150ms
• 2nd data column: sszDeserialize()=86ms, verifyDataColumnSidecarKzgProofs=150ms
• 3rd data column: sszDeserialize()=86ms, verifyDataColumnSidecarKzgProofs=151ms

last sszDeserialize() could be from the block (~143ms), state transition took ~100ms
totally it took ~1.4s on the main thread there

[twoeths]

deployed latest unstsable

main_thread_2026-08-31T09:41:05.043Z.cpuprofile.zip

uncompress() was gone, it's the gc time instead, and it decreased significantly could be due to the blst update in #8900

need to revisit #8628 to see if we can reuse memory bytes when deserializing gossip message