Skip to content

fix: pre-establish builder connections before requesting bids - #9947

Open
nflaig wants to merge 7 commits into
unstablefrom
nflaig/pre-establish-builder-connections
Open

fix: pre-establish builder connections before requesting bids#9947
nflaig wants to merge 7 commits into
unstablefrom
nflaig/pre-establish-builder-connections

Conversation

@nflaig

@nflaig nflaig commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Post-Gloas prepareNextSlot replaced the builder status check with the circuit breaker update, so the builders a proposal is about to request bids from are never contacted beforehand. Every bid request then pays a full TCP/TLS handshake inside BUILDER_BID_DEADLINE_MS, which a distant builder can exhaust on the handshake alone.

On glamsterdam-devnet-8 with 8 builders configured, 21 of 41 bid requests failed with Timeout getExecutionPayloadBid request. Node drops idle keep-alive sockets after ~4s, measured against a builder at 262ms RTT:

idle before request request
3s 292ms
6s 881ms
12s 920ms

prepareNextSlot runs at PREPARE_NEXT_SLOT_BPS, a slot before the proposal, so a check there is cold again by the time bids are requested. Scheduling it close to the slot boundary instead, same builder:

bid request
no preceding check 918ms
check 2s earlier 294ms

Running this on the same node took bid request errors from 51% (21/41) to 5.7% (8/140), and bids received from 49% to 83%.

  • add checkStatus() to BuilderApiClient, pinging every known builder concurrently, failures are logged and never propagate
  • call it from prepareNextSlot post-gloas, scheduled BUILDER_STATUS_CHECK_BEFORE_SLOT_BPS before the slot we propose
  • add lodestar_builder_api_status_checks_total

Raising the client keep-alive is not a substitute, a connection idle for minutes through NAT is frequently half-open and fails only once written to, inside the deadline where there is no recovery.

Only builders already in the clients map are pinged, which the preference submissions populate ~96s ahead of a proposal. That map is never evicted, so a builder dropped from the config keeps being pinged until restart, see #9951.

@nflaig
nflaig requested a review from a team as a code owner August 31, 2026 09:02
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-08-31T12:22:30.378648Z 8a94a25 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@nflaig
nflaig marked this pull request as draft August 31, 2026 09:05
@nflaig
nflaig force-pushed the nflaig/pre-establish-builder-connections branch from f47f8a9 to e1d6082 Compare August 31, 2026 09:21
@github-actions

github-actions Bot commented Aug 31, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 97a8cb0 Previous: 59d71c2 Ratio
getPubkeys - native cache - req 1000 vs - 250000 vc 419.43 us/op 435.30 us/op 0.96
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 37.230 us/op 40.692 us/op 0.91
BLS verify - blst 847.08 us/op 899.32 us/op 0.94
BLS verifyMultipleSignatures 3 - blst 1.2645 ms/op 1.3541 ms/op 0.93
BLS verifyMultipleSignatures 8 - blst 2.1336 ms/op 2.2602 ms/op 0.94
BLS verifyMultipleSignatures 32 - blst 6.9856 ms/op 7.4540 ms/op 0.94
BLS verifyMultipleSignatures 64 - blst 13.317 ms/op 14.310 ms/op 0.93
BLS verifyMultipleSignatures 128 - blst 25.457 ms/op 27.270 ms/op 0.93
BLS deserializing 10000 signatures 612.06 ms/op 641.13 ms/op 0.95
BLS deserializing 100000 signatures 6.1715 s/op 6.4440 s/op 0.96
BLS verifyMultipleSignatures - same message - 3 - blst 935.52 us/op 970.21 us/op 0.96
BLS verifyMultipleSignatures - same message - 8 - blst 1.0874 ms/op 1.1012 ms/op 0.99
BLS verifyMultipleSignatures - same message - 32 - blst 1.6840 ms/op 1.7266 ms/op 0.98
BLS verifyMultipleSignatures - same message - 64 - blst 2.4784 ms/op 2.5609 ms/op 0.97
BLS verifyMultipleSignatures - same message - 128 - blst 4.0898 ms/op 4.2510 ms/op 0.96
BLS aggregatePubkeys 32 - blst 18.947 us/op 19.666 us/op 0.96
BLS aggregatePubkeys 128 - blst 68.811 us/op 71.175 us/op 0.97
getSlashingsAndExits - default max 37.660 us/op 40.375 us/op 0.93
getSlashingsAndExits - 2k 363.69 us/op 357.69 us/op 1.02
proposeBlockBody type=full, size=empty 679.65 us/op 730.08 us/op 0.93
isKnown best case - 1 super set check 193.00 ns/op 167.00 ns/op 1.16
isKnown normal case - 2 super set checks 167.00 ns/op 166.00 ns/op 1.01
isKnown worse case - 16 super set checks 168.00 ns/op 162.00 ns/op 1.04
validate api signedAggregateAndProof - struct 1.4437 ms/op 1.5207 ms/op 0.95
validate gossip signedAggregateAndProof - struct 1.4406 ms/op 1.5190 ms/op 0.95
batch validate gossip attestation - vc 640000 - chunk 32 121.86 us/op 127.85 us/op 0.95
batch validate gossip attestation - vc 640000 - chunk 64 103.34 us/op 108.52 us/op 0.95
batch validate gossip attestation - vc 640000 - chunk 128 92.978 us/op 98.855 us/op 0.94
batch validate gossip attestation - vc 640000 - chunk 256 92.615 us/op 99.383 us/op 0.93
bytes32 toHexString 273.00 ns/op 286.00 ns/op 0.95
bytes32 Buffer.toString(hex) 163.00 ns/op 177.00 ns/op 0.92
bytes32 Buffer.toString(hex) from Uint8Array 235.00 ns/op 241.00 ns/op 0.98
bytes32 Buffer.toString(hex) + 0x 160.00 ns/op 176.00 ns/op 0.91
Return object 10000 times 0.20550 ns/op 0.21760 ns/op 0.94
Throw Error 10000 times 3.1684 us/op 3.3407 us/op 0.95
toHex 89.866 ns/op 95.092 ns/op 0.95
Buffer.from 82.481 ns/op 86.862 ns/op 0.95
shared Buffer 55.457 ns/op 59.096 ns/op 0.94
fastMsgIdFn sha256 / 200 bytes 1.4140 us/op 1.5030 us/op 0.94
fastMsgIdFn h32 xxhash / 200 bytes 147.00 ns/op 154.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 200 bytes 201.00 ns/op 207.00 ns/op 0.97
fastMsgIdFn sha256 / 1000 bytes 4.5420 us/op 4.8080 us/op 0.94
fastMsgIdFn h32 xxhash / 1000 bytes 235.00 ns/op 245.00 ns/op 0.96
fastMsgIdFn h64 xxhash / 1000 bytes 245.00 ns/op 250.00 ns/op 0.98
fastMsgIdFn sha256 / 10000 bytes 40.124 us/op 42.542 us/op 0.94
fastMsgIdFn h32 xxhash / 10000 bytes 1.2210 us/op 1.3220 us/op 0.92
fastMsgIdFn h64 xxhash / 10000 bytes 781.00 ns/op 855.00 ns/op 0.91
send data - 1000 256B messages 4.1029 ms/op 4.1753 ms/op 0.98
send data - 1000 512B messages 5.1208 ms/op 5.2114 ms/op 0.98
send data - 1000 1024B messages 5.4387 ms/op 5.5835 ms/op 0.97
send data - 1000 1200B messages 5.9931 ms/op 8.8017 ms/op 0.68
send data - 1000 2048B messages 13.753 ms/op 11.986 ms/op 1.15
send data - 1000 4096B messages 44.088 ms/op 88.831 ms/op 0.50
send data - 1000 16384B messages 484.39 ms/op 382.98 ms/op 1.26
send data - 1000 65536B messages 1.5505 s/op 1.6882 s/op 0.92
enrSubnets - fastDeserialize 64 bits 745.00 ns/op 755.00 ns/op 0.99
enrSubnets - ssz BitVector 64 bits 266.00 ns/op 275.00 ns/op 0.97
enrSubnets - fastDeserialize 4 bits 104.00 ns/op 100.00 ns/op 1.04
enrSubnets - ssz BitVector 4 bits 263.00 ns/op 267.00 ns/op 0.99
prioritizePeers score -10:0 att 32-0.1 sync 2-0 196.28 us/op 203.37 us/op 0.97
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 223.83 us/op 230.57 us/op 0.97
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 326.24 us/op 339.10 us/op 0.96
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 576.04 us/op 593.58 us/op 0.97
prioritizePeers score 0:0 att 64-1 sync 4-1 654.33 us/op 711.17 us/op 0.92
array of 16000 items push then shift 1.1989 us/op 1.3074 us/op 0.92
LinkedList of 16000 items push then shift 7.1110 ns/op 7.3520 ns/op 0.97
array of 16000 items push then pop 68.706 ns/op 79.330 ns/op 0.87
LinkedList of 16000 items push then pop 5.6570 ns/op 6.0460 ns/op 0.94
array of 24000 items push then shift 1.7617 us/op 1.9429 us/op 0.91
LinkedList of 24000 items push then shift 6.7050 ns/op 6.9380 ns/op 0.97
array of 24000 items push then pop 99.173 ns/op 113.44 ns/op 0.87
LinkedList of 24000 items push then pop 5.7750 ns/op 5.9930 ns/op 0.96
intersect bitArray bitLen 8 3.7260 ns/op 3.9120 ns/op 0.95
intersect array and set length 8 28.339 ns/op 29.542 ns/op 0.96
intersect bitArray bitLen 128 22.192 ns/op 23.249 ns/op 0.95
intersect array and set length 128 479.80 ns/op 501.50 ns/op 0.96
bitArray.getTrueBitIndexes() bitLen 128 901.00 ns/op 922.00 ns/op 0.98
bitArray.getTrueBitIndexes() bitLen 248 1.6270 us/op 1.6670 us/op 0.98
bitArray.getTrueBitIndexes() bitLen 512 3.4270 us/op 3.5120 us/op 0.98
Full columns - reconstruct all 6 blobs 164.88 us/op 140.20 us/op 1.18
Full columns - reconstruct half of the blobs out of 6 84.444 us/op 106.30 us/op 0.79
Full columns - reconstruct single blob out of 6 31.594 us/op 40.290 us/op 0.78
Half columns - reconstruct all 6 blobs 375.42 ms/op 400.39 ms/op 0.94
Half columns - reconstruct half of the blobs out of 6 189.07 ms/op 200.93 ms/op 0.94
Half columns - reconstruct single blob out of 6 67.517 ms/op 71.457 ms/op 0.94
Set add up to 64 items then delete first 1.5725 us/op 1.6846 us/op 0.93
OrderedSet add up to 64 items then delete first 2.3984 us/op 2.5198 us/op 0.95
Set add up to 64 items then delete last 1.7545 us/op 1.8703 us/op 0.94
OrderedSet add up to 64 items then delete last 2.6244 us/op 2.8058 us/op 0.94
Set add up to 64 items then delete middle 1.7583 us/op 1.8870 us/op 0.93
OrderedSet add up to 64 items then delete middle 4.0599 us/op 4.2701 us/op 0.95
Set add up to 128 items then delete first 3.5411 us/op 3.7455 us/op 0.95
OrderedSet add up to 128 items then delete first 5.4998 us/op 5.7156 us/op 0.96
Set add up to 128 items then delete last 3.3649 us/op 3.5609 us/op 0.94
OrderedSet add up to 128 items then delete last 7.2240 us/op 5.3688 us/op 1.35
Set add up to 128 items then delete middle 3.4258 us/op 3.6200 us/op 0.95
OrderedSet add up to 128 items then delete middle 10.690 us/op 11.386 us/op 0.94
Set add up to 256 items then delete first 6.9521 us/op 7.4600 us/op 0.93
OrderedSet add up to 256 items then delete first 11.687 us/op 12.253 us/op 0.95
Set add up to 256 items then delete last 7.0739 us/op 7.1353 us/op 0.99
OrderedSet add up to 256 items then delete last 11.024 us/op 11.565 us/op 0.95
Set add up to 256 items then delete middle 6.8156 us/op 7.1805 us/op 0.95
OrderedSet add up to 256 items then delete middle 34.007 us/op 35.222 us/op 0.97
runFastConfirmationRules vc:100000 bc:96 eq:0 4.3205 ms/op 4.5729 ms/op 0.94
runFastConfirmationRules vc:600000 bc:96 eq:0 34.259 ms/op 35.123 ms/op 0.98
runFastConfirmationRules vc:1000000 bc:96 eq:0 56.236 ms/op 59.425 ms/op 0.95
runFastConfirmationRules vc:600000 bc:320 eq:0 34.286 ms/op 36.254 ms/op 0.95
runFastConfirmationRules vc:100000 bc:96 eq:1000 1.0892 s/op 1.1586 s/op 0.94
pass gossip attestations to forkchoice per slot 2.4929 ms/op 2.5817 ms/op 0.97
forkChoice updateHead vc 100000 bc 64 eq 0 385.36 us/op 415.45 us/op 0.93
forkChoice updateHead vc 600000 bc 64 eq 0 2.3240 ms/op 2.5213 ms/op 0.92
forkChoice updateHead vc 1000000 bc 64 eq 0 3.9232 ms/op 4.1436 ms/op 0.95
forkChoice updateHead vc 600000 bc 320 eq 0 2.3632 ms/op 2.4839 ms/op 0.95
forkChoice updateHead vc 600000 bc 1200 eq 0 2.3930 ms/op 2.5196 ms/op 0.95
forkChoice updateHead vc 600000 bc 7200 eq 0 3.1394 ms/op 3.0395 ms/op 1.03
forkChoice updateHead vc 600000 bc 64 eq 1000 2.3724 ms/op 2.4659 ms/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 10000 2.4221 ms/op 2.5545 ms/op 0.95
forkChoice updateHead vc 600000 bc 64 eq 300000 6.4153 ms/op 6.9754 ms/op 0.92
forkChoice updateHead vc 600000 bc 64 eq 0 gloas boosted 2.3529 ms/op 2.4793 ms/op 0.95
computeDeltas 1400000 validators 0% inactive 12.115 ms/op 12.212 ms/op 0.99
computeDeltas 1400000 validators 10% inactive 11.396 ms/op 11.602 ms/op 0.98
computeDeltas 1400000 validators 20% inactive 10.839 ms/op 10.945 ms/op 0.99
computeDeltas 1400000 validators 50% inactive 8.7072 ms/op 8.9508 ms/op 0.97
computeDeltas 2100000 validators 0% inactive 18.221 ms/op 18.417 ms/op 0.99
computeDeltas 2100000 validators 10% inactive 17.250 ms/op 17.322 ms/op 1.00
computeDeltas 2100000 validators 20% inactive 16.116 ms/op 16.405 ms/op 0.98
computeDeltas 2100000 validators 50% inactive 10.489 ms/op 11.171 ms/op 0.94
altair processAttestation - 250000 vs - 7PWei normalcase 2.0421 ms/op 1.6057 ms/op 1.27
altair processAttestation - 250000 vs - 7PWei worstcase 3.0342 ms/op 2.4956 ms/op 1.22
altair processAttestation - setStatus - 1/6 committees join 105.99 us/op 101.75 us/op 1.04
altair processAttestation - setStatus - 1/3 committees join 207.24 us/op 207.67 us/op 1.00
altair processAttestation - setStatus - 1/2 committees join 281.29 us/op 294.68 us/op 0.95
altair processAttestation - setStatus - 2/3 committees join 365.44 us/op 373.22 us/op 0.98
altair processAttestation - setStatus - 4/5 committees join 493.59 us/op 527.40 us/op 0.94
altair processAttestation - setStatus - 100% committees join 583.60 us/op 619.18 us/op 0.94
altair processBlock - 250000 vs - 7PWei normalcase 4.8106 ms/op 4.0808 ms/op 1.18
altair processBlock - 250000 vs - 7PWei normalcase hashState 21.028 ms/op 22.174 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase 23.845 ms/op 24.993 ms/op 0.95
altair processBlock - 250000 vs - 7PWei worstcase hashState 51.332 ms/op 50.086 ms/op 1.02
phase0 processBlock - 250000 vs - 7PWei normalcase 1.3054 ms/op 1.3744 ms/op 0.95
phase0 processBlock - 250000 vs - 7PWei worstcase 20.424 ms/op 20.392 ms/op 1.00
altair processEth1Data - 250000 vs - 7PWei normalcase 283.57 us/op 295.86 us/op 0.96
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:16 3.5490 us/op 3.0740 us/op 1.15
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:220 19.738 us/op 19.818 us/op 1.00
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:43 7.1940 us/op 5.6130 us/op 1.28
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:19 3.7630 us/op 3.6170 us/op 1.04
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1021 91.860 us/op 89.540 us/op 1.03
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11778 1.3788 ms/op 1.4371 ms/op 0.96
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.7493 ms/op 1.9720 ms/op 0.89
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.7728 ms/op 1.8719 ms/op 0.95
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 4.2655 ms/op 3.6843 ms/op 1.16
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 2.0418 ms/op 2.1361 ms/op 0.96
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 4.8368 ms/op 4.1393 ms/op 1.17
Tree 40 250000 create 342.98 ms/op 313.16 ms/op 1.10
Tree 40 250000 get(125000) 90.309 ns/op 95.291 ns/op 0.95
Tree 40 250000 set(125000) 982.91 ns/op 1.0286 us/op 0.96
Tree 40 250000 toArray() 16.350 ms/op 10.321 ms/op 1.58
Tree 40 250000 iterate all - toArray() + loop 16.644 ms/op 13.598 ms/op 1.22
Tree 40 250000 iterate all - get(i) 40.814 ms/op 37.842 ms/op 1.08
Array 250000 create 2.2819 ms/op 2.5650 ms/op 0.89
Array 250000 clone - spread 703.92 us/op 802.28 us/op 0.88
Array 250000 get(125000) 0.29500 ns/op 0.30200 ns/op 0.98
Array 250000 set(125000) 0.29100 ns/op 0.30200 ns/op 0.96
Array 250000 iterate all - loop 56.484 us/op 58.656 us/op 0.96
phase0 afterProcessEpoch - 250000 vs - 7PWei 38.835 ms/op 39.627 ms/op 0.98
Array.fill - length 1000000 9.3548 ms/op 4.2429 ms/op 2.20
Array push - length 1000000 14.199 ms/op 13.421 ms/op 1.06
Array.get 0.20448 ns/op 0.20979 ns/op 0.97
Uint8Array.get 0.24488 ns/op 0.25320 ns/op 0.97
phase0 beforeProcessEpoch - 250000 vs - 7PWei 22.442 ms/op 19.009 ms/op 1.18
altair processEpoch - mainnet_e81889 254.16 ms/op 208.76 ms/op 1.22
mainnet_e81889 - altair beforeProcessEpoch 36.363 ms/op 46.449 ms/op 0.78
mainnet_e81889 - altair processJustificationAndFinalization 5.7930 us/op 4.6630 us/op 1.24
mainnet_e81889 - altair processInactivityUpdates 6.5113 ms/op 3.5133 ms/op 1.85
mainnet_e81889 - altair processRewardsAndPenalties 18.257 ms/op 17.016 ms/op 1.07
mainnet_e81889 - altair processRegistryUpdates 572.00 ns/op 582.00 ns/op 0.98
mainnet_e81889 - altair processSlashings 144.00 ns/op 144.00 ns/op 1.00
mainnet_e81889 - altair processEth1DataReset 137.00 ns/op 136.00 ns/op 1.01
mainnet_e81889 - altair processEffectiveBalanceUpdates 6.8520 ms/op 1.2984 ms/op 5.28
mainnet_e81889 - altair processSlashingsReset 702.00 ns/op 712.00 ns/op 0.99
mainnet_e81889 - altair processRandaoMixesReset 975.00 ns/op 919.00 ns/op 1.06
mainnet_e81889 - altair processHistoricalRootsUpdate 142.00 ns/op 143.00 ns/op 0.99
mainnet_e81889 - altair processParticipationFlagUpdates 449.00 ns/op 444.00 ns/op 1.01
mainnet_e81889 - altair processSyncCommitteeUpdates 115.00 ns/op 110.00 ns/op 1.05
mainnet_e81889 - altair afterProcessEpoch 40.566 ms/op 41.643 ms/op 0.97
capella processEpoch - mainnet_e217614 752.98 ms/op 655.36 ms/op 1.15
mainnet_e217614 - capella beforeProcessEpoch 75.661 ms/op 78.668 ms/op 0.96
mainnet_e217614 - capella processJustificationAndFinalization 5.2350 us/op 5.2270 us/op 1.00
mainnet_e217614 - capella processInactivityUpdates 14.615 ms/op 14.591 ms/op 1.00
mainnet_e217614 - capella processRewardsAndPenalties 91.277 ms/op 99.666 ms/op 0.92
mainnet_e217614 - capella processRegistryUpdates 4.4090 us/op 4.4980 us/op 0.98
mainnet_e217614 - capella processSlashings 141.00 ns/op 143.00 ns/op 0.99
mainnet_e217614 - capella processEth1DataReset 135.00 ns/op 137.00 ns/op 0.99
mainnet_e217614 - capella processEffectiveBalanceUpdates 16.704 ms/op 14.447 ms/op 1.16
mainnet_e217614 - capella processSlashingsReset 656.00 ns/op 686.00 ns/op 0.96
mainnet_e217614 - capella processRandaoMixesReset 917.00 ns/op 931.00 ns/op 0.98
mainnet_e217614 - capella processHistoricalRootsUpdate 135.00 ns/op 146.00 ns/op 0.92
mainnet_e217614 - capella processParticipationFlagUpdates 428.00 ns/op 457.00 ns/op 0.94
mainnet_e217614 - capella afterProcessEpoch 106.35 ms/op 110.02 ms/op 0.97
phase0 processEpoch - mainnet_e58758 228.56 ms/op 200.11 ms/op 1.14
mainnet_e58758 - phase0 beforeProcessEpoch 53.279 ms/op 44.841 ms/op 1.19
mainnet_e58758 - phase0 processJustificationAndFinalization 5.0450 us/op 4.6240 us/op 1.09
mainnet_e58758 - phase0 processRewardsAndPenalties 17.461 ms/op 16.070 ms/op 1.09
mainnet_e58758 - phase0 processRegistryUpdates 2.2470 us/op 2.3240 us/op 0.97
mainnet_e58758 - phase0 processSlashings 140.00 ns/op 150.00 ns/op 0.93
mainnet_e58758 - phase0 processEth1DataReset 240.00 ns/op 277.00 ns/op 0.87
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 812.50 us/op 826.68 us/op 0.98
mainnet_e58758 - phase0 processSlashingsReset 863.00 ns/op 835.00 ns/op 1.03
mainnet_e58758 - phase0 processRandaoMixesReset 1.0110 us/op 942.00 ns/op 1.07
mainnet_e58758 - phase0 processHistoricalRootsUpdate 136.00 ns/op 145.00 ns/op 0.94
mainnet_e58758 - phase0 processParticipationRecordUpdates 975.00 ns/op 929.00 ns/op 1.05
mainnet_e58758 - phase0 afterProcessEpoch 32.865 ms/op 33.876 ms/op 0.97
phase0 processEffectiveBalanceUpdates - 250000 normalcase 979.99 us/op 1.0100 ms/op 0.97
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.5565 ms/op 1.2256 ms/op 1.27
altair processInactivityUpdates - 250000 normalcase 13.461 ms/op 10.903 ms/op 1.23
altair processInactivityUpdates - 250000 worstcase 13.778 ms/op 12.928 ms/op 1.07
phase0 processRegistryUpdates - 250000 normalcase 2.4110 us/op 2.4230 us/op 1.00
phase0 processRegistryUpdates - 250000 badcase_full_deposits 136.99 us/op 144.36 us/op 0.95
phase0 processRegistryUpdates - 250000 worstcase 0.5 64.740 ms/op 64.613 ms/op 1.00
altair processRewardsAndPenalties - 250000 normalcase 13.907 ms/op 13.524 ms/op 1.03
altair processRewardsAndPenalties - 250000 worstcase 13.618 ms/op 13.910 ms/op 0.98
phase0 getAttestationDeltas - 250000 normalcase 5.1904 ms/op 5.6997 ms/op 0.91
phase0 getAttestationDeltas - 250000 worstcase 5.4858 ms/op 5.8175 ms/op 0.94
phase0 processSlashings - 250000 worstcase 60.505 us/op 65.474 us/op 0.92
altair processSyncCommitteeUpdates - 250000 11.614 ms/op 10.115 ms/op 1.15
BeaconState.hashTreeRoot - No change 168.00 ns/op 172.00 ns/op 0.98
BeaconState.hashTreeRoot - 1 full validator 79.342 us/op 60.367 us/op 1.31
BeaconState.hashTreeRoot - 32 full validator 845.43 us/op 846.75 us/op 1.00
BeaconState.hashTreeRoot - 512 full validator 8.1502 ms/op 8.9661 ms/op 0.91
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 101.23 us/op 98.681 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3959 ms/op 1.5387 ms/op 0.91
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.768 ms/op 14.222 ms/op 1.25
BeaconState.hashTreeRoot - 1 balances 84.907 us/op 78.607 us/op 1.08
BeaconState.hashTreeRoot - 32 balances 735.58 us/op 871.79 us/op 0.84
BeaconState.hashTreeRoot - 512 balances 5.5771 ms/op 7.4407 ms/op 0.75
BeaconState.hashTreeRoot - 250000 balances 127.99 ms/op 137.09 ms/op 0.93
aggregationBits - 2048 els - zipIndexesInBitList 18.924 us/op 20.510 us/op 0.92
regular array get 100000 times 21.984 us/op 23.290 us/op 0.94
wrappedArray get 100000 times 21.955 us/op 23.346 us/op 0.94
arrayWithProxy get 100000 times 17.145 ms/op 10.351 ms/op 1.66
ssz.Root.equals 20.629 ns/op 21.463 ns/op 0.96
byteArrayEquals 20.542 ns/op 21.201 ns/op 0.97
Buffer.compare 8.7580 ns/op 9.2290 ns/op 0.95
processSlot - 1 slots 10.204 us/op 8.4880 us/op 1.20
processSlot - 32 slots 2.1234 ms/op 1.9683 ms/op 1.08
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 5.7271 ms/op 4.6822 ms/op 1.22
getCommitteeAssignments - req 1 vs - 250000 vc 1.5928 ms/op 1.6790 ms/op 0.95
getCommitteeAssignments - req 100 vs - 250000 vc 3.2479 ms/op 3.4080 ms/op 0.95
getCommitteeAssignments - req 1000 vs - 250000 vc 3.5080 ms/op 3.6717 ms/op 0.96
findModifiedValidators - 10000 modified validators 693.65 ms/op 819.57 ms/op 0.85
findModifiedValidators - 1000 modified validators 541.42 ms/op 640.94 ms/op 0.84
findModifiedValidators - 100 modified validators 334.04 ms/op 436.07 ms/op 0.77
findModifiedValidators - 10 modified validators 266.51 ms/op 339.20 ms/op 0.79
findModifiedValidators - 1 modified validators 223.06 ms/op 269.48 ms/op 0.83
findModifiedValidators - no difference 178.33 ms/op 201.37 ms/op 0.89
migrate state 1500000 validators, 3400 modified, 2000 new 3.2123 s/op 3.0636 s/op 1.05
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 3.7800 ns/op 3.7500 ns/op 1.01
state getBlockRootAtSlot - 250000 vs - 7PWei 424.59 ns/op 288.10 ns/op 1.47
computeProposerIndex 100000 validators 1.3846 ms/op 1.3718 ms/op 1.01
getNextSyncCommitteeIndices 1000 validators 2.8981 ms/op 2.9125 ms/op 1.00
getNextSyncCommitteeIndices 10000 validators 25.713 ms/op 26.258 ms/op 0.98
getNextSyncCommitteeIndices 100000 validators 89.035 ms/op 89.003 ms/op 1.00
computeProposers - vc 250000 574.18 us/op 570.62 us/op 1.01
computeEpochShuffling - vc 250000 38.968 ms/op 39.450 ms/op 0.99
getNextSyncCommittee - vc 250000 9.8916 ms/op 9.6063 ms/op 1.03
nodejs block root to RootHex using toHex 105.18 ns/op 100.79 ns/op 1.04
nodejs block root to RootHex using toRootHex 61.981 ns/op 64.908 ns/op 0.95
nodejs fromHex(blob) 910.35 us/op 1.1784 ms/op 0.77
nodejs fromHexInto(blob) 635.67 us/op 637.67 us/op 1.00
nodejs block root to RootHex using the deprecated toHexString 585.42 ns/op 600.73 ns/op 0.97
nodejs byteArrayEquals 32 bytes (block root) 25.183 ns/op 26.217 ns/op 0.96
nodejs byteArrayEquals 48 bytes (pubkey) 36.092 ns/op 37.815 ns/op 0.95
nodejs byteArrayEquals 96 bytes (signature) 34.236 ns/op 37.405 ns/op 0.92
nodejs byteArrayEquals 1024 bytes 40.208 ns/op 45.231 ns/op 0.89
nodejs byteArrayEquals 131072 bytes (blob) 1.7526 us/op 1.7643 us/op 0.99
browser block root to RootHex using toHex 142.05 ns/op 146.47 ns/op 0.97
browser block root to RootHex using toRootHex 125.65 ns/op 130.73 ns/op 0.96
browser fromHex(blob) 1.7874 ms/op 1.9588 ms/op 0.91
browser fromHexInto(blob) 607.17 us/op 638.54 us/op 0.95
browser block root to RootHex using the deprecated toHexString 388.91 ns/op 397.01 ns/op 0.98
browser byteArrayEquals 32 bytes (block root) 26.963 ns/op 28.385 ns/op 0.95
browser byteArrayEquals 48 bytes (pubkey) 37.988 ns/op 40.172 ns/op 0.95
browser byteArrayEquals 96 bytes (signature) 71.200 ns/op 74.197 ns/op 0.96
browser byteArrayEquals 1024 bytes 724.57 ns/op 751.28 ns/op 0.96
browser byteArrayEquals 131072 bytes (blob) 91.217 us/op 94.835 us/op 0.96

by benchmarkbot/action

@nflaig
nflaig marked this pull request as ready for review August 31, 2026 11:44
chatgpt-codex-connector[bot]
chatgpt-codex-connector Bot reviewed Aug 31, 2026
View reviewed changes

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 58cfb4c899

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/beacon-node/test/unit/chain/prepareNextSlot.test.ts
@nflaig

nflaig commented Aug 31, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 8a94a25c42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@nflaig nflaig mentioned this pull request Aug 31, 2026
Open
nflaig
nflaig commented Aug 31, 2026
View reviewed changes
Comment thread packages/beacon-node/src/chain/prepareNextSlot.ts
* 1667 = 16.67% of slot (2s on mainnet). Must stay under the few seconds an idle connection survives,
* while leaving room for a round trip to a distant builder to complete before the slot begins.
*/
const BUILDER_STATUS_CHECK_BEFORE_SLOT_BPS = 1667;

@nflaig nflaig Aug 31, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could also just me a _MS value as it's not really related to slot duration, but either way seems fine to me

This was referenced Aug 31, 2026
Open
Open
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

Lodestar Team Coordination
Status: In Progress

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nflaig