Skip to content

Data column gossip verification taking more than 4s when blob count is above 20 #7203

New issue
New issue
Open
Open
Data column gossip verification taking more than 4s when blob count is above 20#7203
Assignees
pawanjay176
Labels
dasData Availability SamplingoptimizationSomething to make Lighthouse run more efficiently.
@jimmygchen

Description

@jimmygchen
jimmygchen
opened on Mar 26, 2025 · edited by jimmygchen
Member

In the latest PeerDAS devnet testing report published by Sunnyside Lab:
https://testinprod.notion.site/Sunnyside-Devnet-Updates-03-25-Internal-1c08fc57f54680339b2fe2dbc8232b6d

Findings show that the network degraded when blobs per block is above 20, and Lighthouse's data column verification time increases quite significantly, taking more than 4s, causing the network to breakdown:

Image

Might be worth checking these metrics as well:

  • beacon_kzg_verification_data_column_batch_seconds
  • beacon_kzg_verification_data_column_single_seconds

Activity

jimmygchen
added
dasData Availability Sampling
on Mar 26, 2025
jimmygchen

jimmygchen commented on Mar 26, 2025

@jimmygchen
jimmygchen
on Mar 26, 2025
MemberAuthor

I looked at the following verification metric, and got quite different numbers as the report. I'll reach out to Sunnyside Lab to confirm the metric they're looking at.

  • histogram_quantile(0.50, sum(rate(beacon_data_column_sidecar_gossip_verification_seconds_bucket[5m])) by (le))
Image
jimmygchen

jimmygchen commented on Mar 26, 2025

@jimmygchen
jimmygchen
on Mar 26, 2025
MemberAuthor

In the test the 95th percentile was used (instead of 50th), and showing very different results:

I used histogram_quantile(0.95, rate(beacon_data_column_sidecar_gossip_verification_seconds_bucket{network=~'$network', execution_client=~'$execution', consensus_client=~'$consensus'}[$__rate_interval])) so that we could see the data from each series. If we look in this data individually, full node(have even numbers like lighthouse-geth-2) have higher runtime than supernodes (with odd numbers)

Image
pawanjay176
self-assigned this
on Apr 3, 2025
pawanjay176

pawanjay176 commented on Apr 3, 2025

@pawanjay176
pawanjay176
on Apr 3, 2025
Member

I can take a look

jimmygchen

jimmygchen commented on Apr 3, 2025

@jimmygchen
jimmygchen
on Apr 3, 2025
MemberAuthor

Thanks @pawanjay176!

jimmygchen

jimmygchen commented on Apr 7, 2025

@jimmygchen
jimmygchen
on Apr 7, 2025
MemberAuthor

@pawanjay176
I was thinking if the following issue may be related - multiple concurrent writes holding up the fork choice write lock
#6439

jimmygchen
mentioned this on Apr 10, 2025
mergify
added a commit that references this issue on May 1, 2025

Compute proposer shuffling only once in gossip verification (#7304)

93ec9df
jimmygchen
added
optimizationSomething to make Lighthouse run more efficiently.
on May 29, 2025
jimmygchen
mentioned this on Jun 6, 2025
jimmygchen

jimmygchen commented on Jul 7, 2025

@jimmygchen
jimmygchen
on Jul 7, 2025 · edited by jimmygchen
MemberAuthor

I've done some more testing and I'm seeing numbers between 10ms and 20ms (time.busy), so the metrics were actually not very accurate due to bad prometheus buckets:

2025-07-06T14:59:16.779858Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=1}: beacon_chain::data_column_verification: close time.busy=8.45ms time.idle=941ns
2025-07-06T14:59:16.815840Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=15}: beacon_chain::data_column_verification: close time.busy=10.4ms time.idle=1.38µs
2025-07-06T14:59:16.816173Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=9}: beacon_chain::data_column_verification: close time.busy=11.0ms time.idle=1.44µs
2025-07-06T14:59:16.816432Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=14}: beacon_chain::data_column_verification: close time.busy=11.1ms time.idle=1.44µs
2025-07-06T14:59:16.818649Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=24}: beacon_chain::data_column_verification: close time.busy=11.6ms time.idle=1.38µs
2025-07-06T14:59:16.819102Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=12}: beacon_chain::data_column_verification: close time.busy=13.9ms time.idle=1.81µs
2025-07-06T14:59:16.819959Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=13}: beacon_chain::data_column_verification: close time.busy=14.7ms time.idle=927ns
2025-07-06T14:59:16.822445Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=10}: beacon_chain::data_column_verification: close time.busy=17.2ms time.idle=1.71µs
2025-07-06T14:59:16.822823Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=8}: beacon_chain::data_column_verification: close time.busy=17.7ms time.idle=1.25µs
2025-07-06T14:59:16.823140Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=23}: beacon_chain::data_column_verification: close time.busy=16.1ms time.idle=1.22µs
2025-07-06T14:59:16.826797Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=27}: beacon_chain::data_column_verification: close time.busy=10.9ms time.idle=840ns
2025-07-06T14:59:16.826937Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=33}: beacon_chain::data_column_verification: close time.busy=10.4ms time.idle=892ns
2025-07-06T14:59:16.827017Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=28}: beacon_chain::data_column_verification: close time.busy=10.8ms time.idle=725ns
2025-07-06T14:59:16.831967Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=25}: beacon_chain::data_column_verification: close time.busy=15.7ms time.idle=961ns
2025-07-06T14:59:16.832092Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=41}: beacon_chain::data_column_verification: close time.busy=12.1ms time.idle=733ns
2025-07-06T14:59:16.832256Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=38}: beacon_chain::data_column_verification: close time.busy=13.5ms time.idle=983ns
2025-07-06T14:59:16.834612Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=46}: beacon_chain::data_column_verification: close time.busy=11.7ms time.idle=912ns
2025-07-06T14:59:16.835096Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=43}: beacon_chain::data_column_verification: close time.busy=12.6ms time.idle=791ns
2025-07-06T14:59:16.835584Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=40}: beacon_chain::data_column_verification: close time.busy=16.4ms time.idle=927ns
2025-07-06T14:59:16.845956Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=54}: beacon_chain::data_column_verification: close time.busy=8.95ms time.idle=1.02µs
2025-07-06T14:59:16.849790Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=53}: beacon_chain::data_column_verification: close time.busy=13.9ms time.idle=1.25µs
2025-07-06T14:59:16.850050Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=57}: beacon_chain::data_column_verification: close time.busy=11.2ms time.idle=724ns
2025-07-06T14:59:16.851004Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=59}: beacon_chain::data_column_verification: close time.busy=11.1ms time.idle=939ns
2025-07-06T14:59:16.851240Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=52}: beacon_chain::data_column_verification: close time.busy=16.4ms time.idle=1.16µs
2025-07-06T14:59:16.852728Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=61}: beacon_chain::data_column_verification: close time.busy=11.8ms time.idle=1.00µs
2025-07-06T14:59:16.852814Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=56}: beacon_chain::data_column_verification: close time.busy=14.9ms time.idle=1.13µs
2025-07-06T14:59:16.855261Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=63}: beacon_chain::data_column_verification: close time.busy=12.4ms time.idle=1.07µs
2025-07-06T14:59:16.858261Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=62}: beacon_chain::data_column_verification: close time.busy=16.3ms time.idle=1.23µs
2025-07-06T14:59:16.859613Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=65}: beacon_chain::data_column_verification: close time.busy=10.8ms time.idle=1.04µs
2025-07-06T14:59:16.859971Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=68}: beacon_chain::data_column_verification: close time.busy=10.1ms time.idle=711ns
2025-07-06T14:59:16.860254Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=70}: beacon_chain::data_column_verification: close time.busy=9.16ms time.idle=762ns
2025-07-06T14:59:16.863504Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=64}: beacon_chain::data_column_verification: close time.busy=17.4ms time.idle=713ns
2025-07-06T14:59:16.865082Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=69}: beacon_chain::data_column_verification: close time.busy=14.9ms time.idle=722ns
2025-07-06T14:59:16.866071Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=71}: beacon_chain::data_column_verification: close time.busy=14.7ms time.idle=827ns
2025-07-06T14:59:16.881093Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=73}: beacon_chain::data_column_verification: close time.busy=13.4ms time.idle=972ns
2025-07-06T14:59:16.882666Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=75}: beacon_chain::data_column_verification: close time.busy=13.7ms time.idle=967ns
2025-07-06T14:59:16.883891Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=76}: beacon_chain::data_column_verification: close time.busy=13.6ms time.idle=811ns
2025-07-06T14:59:16.889338Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=85}: beacon_chain::data_column_verification: close time.busy=12.9ms time.idle=898ns
2025-07-06T14:59:16.890130Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=77}: beacon_chain::data_column_verification: close time.busy=13.7ms time.idle=1.20µs
2025-07-06T14:59:16.892505Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=79}: beacon_chain::data_column_verification: close time.busy=16.1ms time.idle=1.83µs
2025-07-06T14:59:16.893451Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=84}: beacon_chain::data_column_verification: close time.busy=17.0ms time.idle=1.67µs
2025-07-06T14:59:16.894242Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=81}: beacon_chain::data_column_verification: close time.busy=17.8ms time.idle=1.85µs
2025-07-06T14:59:16.894441Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=87}: beacon_chain::data_column_verification: close time.busy=18.0ms time.idle=2.09µs
2025-07-06T14:59:16.894665Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}: beacon_chain::data_column_verification: close time.busy=18.3ms time.idle=729ns
2025-07-06T14:59:17.078836Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=91}: beacon_chain::data_column_verification: close time.busy=11.2ms time.idle=838ns

And a breakdown shows out of 18.3ms

  • 13.8ms was spent on kzg verification
  • 2.24ms was spent on proposer signature verification (we can optimise this via caching or batch verification)
2025-07-06T14:59:16.876406Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_data_column_sidecar: beacon_chain::data_column_verification: close time.busy=339ns time.idle=828ns
2025-07-06T14:59:16.876420Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_index_matches_subnet: beacon_chain::data_column_verification: close time.busy=401ns time.idle=574ns
2025-07-06T14:59:16.876426Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_sidecar_not_from_future_slot: beacon_chain::data_column_verification: close time.busy=823ns time.idle=641ns
2025-07-06T14:59:16.876463Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_slot_greater_than_latest_finalized_slot: beacon_chain::data_column_verification: close time.busy=923ns time.idle=1.54µs
2025-07-06T14:59:16.878366Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_is_first_sidecar: beacon_chain::data_column_verification: close time.busy=627ns time.idle=1.06µs
2025-07-06T14:59:16.878392Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_column_inclusion_proof: beacon_chain::data_column_verification: close time.busy=15.7µs time.idle=718ns
2025-07-06T14:59:16.878401Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_parent_block_and_finalized_descendant: beacon_chain::data_column_verification: close time.busy=1.23µs time.idle=758ns
2025-07-06T14:59:16.878415Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_slot_higher_than_parent: beacon_chain::data_column_verification: close time.busy=373ns time.idle=794ns
2025-07-06T14:59:16.880665Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_proposer_and_signature: beacon_chain::data_column_verification: close time.busy=2.24ms time.idle=1.16µs
2025-07-06T14:59:16.894646Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:verify_kzg_for_data_column: beacon_chain::data_column_verification: close time.busy=13.8ms time.idle=1.60µs
2025-07-06T14:59:16.894656Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:observe_slashable: beacon_chain::data_column_verification: close time.busy=3.01µs time.idle=737ns
2025-07-06T14:59:16.894662Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}:observe_gossip_data_column: beacon_chain::data_column_verification: close time.busy=629ns time.idle=632ns
2025-07-06T14:59:16.894665Z  INFO validate_data_column_sidecar_for_gossip{slot=Slot(203) block_root=0xb40a7bfc496062e4c62c0945d359319b3d1145de0bd92c88a928324f94a69fff index=80}: beacon_chain::data_column_verification: close time.busy=18.3ms time.idle=729ns
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

dasData Availability SamplingoptimizationSomething to make Lighthouse run more efficiently.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @jimmygchen@pawanjay176

      Issue actions
        Data column gossip verification taking more than 4s when blob count is above 20 · Issue #7203 · sigp/lighthouse