snap sync - apply BALs before flat db heal#10151
Merged
matkt merged 7 commits intobesu-eth:mainfrom Apr 9, 2026
Merged
Conversation
Signed-off-by: Miroslav Kovar <[email protected]>
Signed-off-by: Miroslav Kovar <[email protected]>
mirgee
force-pushed
the
snap-2-apply-bals
branch
2 times, most recently
from
57f04b8 to
069462e
Compare
Signed-off-by: Miroslav Kovar <[email protected]>
mirgee
force-pushed
the
snap-2-apply-bals
branch
from
069462e to
4bd1a47
Compare
matkt
requested changes
Apr 3, 2026
...java/org/hyperledger/besu/ethereum/eth/sync/snapsync/request/BlockAccessListDataRequest.java
Outdated
Show resolved
Hide resolved
...java/org/hyperledger/besu/ethereum/eth/sync/snapsync/request/BlockAccessListDataRequest.java
Outdated
Show resolved
Hide resolved
...java/org/hyperledger/besu/ethereum/eth/sync/snapsync/request/BlockAccessListDataRequest.java
Outdated
Show resolved
Hide resolved
...java/org/hyperledger/besu/ethereum/eth/sync/snapsync/request/BlockAccessListDataRequest.java
Outdated
Show resolved
Hide resolved
| protected PivotSyncState storeState(final PivotSyncState fastSyncState) { | ||
| final Optional<BlockHeader> firstPivotBlockHeader = | ||
| initialPivotSyncState instanceof SnapSyncProcessState snapSyncState | ||
| ? snapSyncState.getFirstPivotBlockHeader().or(fastSyncState::getPivotBlockHeader) |
Contributor
There was a problem hiding this comment.
if we keep all the pivot blocks used during snapsync you can completely skip the fladb healing if all the pivot blocks used are canonical and only trigger the heal if one of the block is not canonical
Contributor
Author
There was a problem hiding this comment.
Not necessarily all pivots, as we already discussed. It's a valid point but hopefully you don't want to address it in this PR :)
Signed-off-by: Miroslav Kovar <[email protected]>
Signed-off-by: Miroslav Kovar <[email protected]>
mirgee
force-pushed
the
snap-2-apply-bals
branch
from
65c1ff7 to
4ee4c1f
Compare
matkt
reviewed
Apr 8, 2026
| final Optional<BlockHeader> maybeBlockHeader = blockchain.getBlockHeader(blockNumber); | ||
| if (maybeBlockHeader.isPresent()) { | ||
| final BlockHeader blockHeader = maybeBlockHeader.get(); | ||
| enqueueRequest(createBlockAccessListDataRequest(blockHeader.getStateRoot(), blockHeader)); |
Contributor
There was a problem hiding this comment.
in another PR I think it will be better to download the BAL before in the pipeline where we are downloading header, body
daniellehrner
added a commit
that referenced
this pull request
Apr 9, 2026
* Add SHL, SHR and SAR shift operations for EVM v2 (#10154) * Add SHL, SHR and SAR implementations and benchmarks for EVM v2 Signed-off-by: Ameziane H. <[email protected]> * Upgrade RocksDB version from 9.7.3 to 10.6.2 (#9767) * Upgrade RocksDB version from 9.7.3 to 10.6.2 * Fix JNI SIGSEGV crashes Signed-off-by: Ameziane H. <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> * Add missing verification metadata (#10198) Signed-off-by: Fabio Di Fabio <[email protected]> * Stream debug_traceBlock* responses directly to avoid OOM on large blocks (#9848) * stream block traces on op code level Signed-off-by: daniellehrner <[email protected]> * correctly parse default setting for memory tracing Signed-off-by: daniellehrner <[email protected]> * fix initcode capture for failed create op codes Signed-off-by: daniellehrner <[email protected]> * created separate streaming debug tracer, for batch request fall back to accumulation in memory, adddress pr comments Signed-off-by: daniellehrner <[email protected]> * execute tests from genesis and verify full trace Signed-off-by: daniellehrner <[email protected]> * addressed pr comments Signed-off-by: daniellehrner <[email protected]> * spotless Signed-off-by: daniellehrner <[email protected]> * optimize trace streaming and struct log handling Signed-off-by: Ameziane H. <[email protected]> * spotless Signed-off-by: Ameziane H. <[email protected]> * Fix remaining issues and add unit tests Signed-off-by: Ameziane H. <[email protected]> * added back pressure when writing to the socket and reduced the buffer size to work better with netty's default buffer size Signed-off-by: daniellehrner <[email protected]> * improve error handling by deferring to send the header only when data is available, allows to send the proper error codes during setup Signed-off-by: daniellehrner <[email protected]> * compactHex candidate comparison Signed-off-by: daniellehrner <[email protected]> * wire in more performant hex writer Signed-off-by: daniellehrner <[email protected]> * introduce separate timeout for streaming calls, defaults to 10 minutes Signed-off-by: daniellehrner <[email protected]> * spotless Signed-off-by: daniellehrner <[email protected]> * Fix streamin/accumulating output parity, added missing refund field, corrected error format, reason encoding, returnValue prefix, and precompile gasCost, with equivalence tests between both Signed-off-by: daniellehrner <[email protected]> * revert accidental removal of 0x prefix Signed-off-by: daniellehrner <[email protected]> * pad memory bytes to 32 bytes Signed-off-by: daniellehrner <[email protected]> --------- Signed-off-by: daniellehrner <[email protected]> Signed-off-by: Ameziane H. <[email protected]> Co-authored-by: Ameziane H. <[email protected]> * Optimize performance and reduce memory when creating Quantity from scalar (#10134) * Optimize performance and reduce memory when creating Quantity from scalar Signed-off-by: Fabio Di Fabio <[email protected]> * Benchmark other implementations Signed-off-by: Fabio Di Fabio <[email protected]> --------- Signed-off-by: Fabio Di Fabio <[email protected]> * snap sync - apply BALs before flat db heal (#10151) Signed-off-by: Miroslav Kovar <[email protected]> * Remove dryRunDetector workaround methods from unit tests (#10201) * Remove dryRunDetector workaround methods from unit tests The dryRunDetector methods were added as a workaround for a Gradle issue that prevented @ParameterizedTest classes from being selected when running with --dry-run. Since the issue is fixed and --dry-run is no longer used, these methods are no longer needed. Signed-off-by: Fabio Di Fabio <[email protected]> * Remove dryRunDetector workaround from acceptance tests too The Gradle issue is confirmed fixed, so the workaround is no longer needed anywhere, including acceptance tests. Signed-off-by: Fabio Di Fabio <[email protected]> --------- Signed-off-by: Fabio Di Fabio <[email protected]> * preserve state gas reservoir for the top level frame in case of OOG (#10205) Signed-off-by: daniellehrner <[email protected]> --------- Signed-off-by: Ameziane H. <[email protected]> Signed-off-by: Fabio Di Fabio <[email protected]> Signed-off-by: daniellehrner <[email protected]> Signed-off-by: Miroslav Kovar <[email protected]> Co-authored-by: ahamlat <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> Co-authored-by: Fabio Di Fabio <[email protected]> Co-authored-by: Miroslav Kovář <[email protected]>
daniellehrner
added a commit
that referenced
this pull request
Apr 10, 2026
* Add SHL, SHR and SAR shift operations for EVM v2 (#10154) * Add SHL, SHR and SAR implementations and benchmarks for EVM v2 Signed-off-by: Ameziane H. <[email protected]> * Upgrade RocksDB version from 9.7.3 to 10.6.2 (#9767) * Upgrade RocksDB version from 9.7.3 to 10.6.2 * Fix JNI SIGSEGV crashes Signed-off-by: Ameziane H. <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> * Add missing verification metadata (#10198) Signed-off-by: Fabio Di Fabio <[email protected]> * Stream debug_traceBlock* responses directly to avoid OOM on large blocks (#9848) * stream block traces on op code level Signed-off-by: daniellehrner <[email protected]> * correctly parse default setting for memory tracing Signed-off-by: daniellehrner <[email protected]> * fix initcode capture for failed create op codes Signed-off-by: daniellehrner <[email protected]> * created separate streaming debug tracer, for batch request fall back to accumulation in memory, adddress pr comments Signed-off-by: daniellehrner <[email protected]> * execute tests from genesis and verify full trace Signed-off-by: daniellehrner <[email protected]> * addressed pr comments Signed-off-by: daniellehrner <[email protected]> * spotless Signed-off-by: daniellehrner <[email protected]> * optimize trace streaming and struct log handling Signed-off-by: Ameziane H. <[email protected]> * spotless Signed-off-by: Ameziane H. <[email protected]> * Fix remaining issues and add unit tests Signed-off-by: Ameziane H. <[email protected]> * added back pressure when writing to the socket and reduced the buffer size to work better with netty's default buffer size Signed-off-by: daniellehrner <[email protected]> * improve error handling by deferring to send the header only when data is available, allows to send the proper error codes during setup Signed-off-by: daniellehrner <[email protected]> * compactHex candidate comparison Signed-off-by: daniellehrner <[email protected]> * wire in more performant hex writer Signed-off-by: daniellehrner <[email protected]> * introduce separate timeout for streaming calls, defaults to 10 minutes Signed-off-by: daniellehrner <[email protected]> * spotless Signed-off-by: daniellehrner <[email protected]> * Fix streamin/accumulating output parity, added missing refund field, corrected error format, reason encoding, returnValue prefix, and precompile gasCost, with equivalence tests between both Signed-off-by: daniellehrner <[email protected]> * revert accidental removal of 0x prefix Signed-off-by: daniellehrner <[email protected]> * pad memory bytes to 32 bytes Signed-off-by: daniellehrner <[email protected]> --------- Signed-off-by: daniellehrner <[email protected]> Signed-off-by: Ameziane H. <[email protected]> Co-authored-by: Ameziane H. <[email protected]> * Optimize performance and reduce memory when creating Quantity from scalar (#10134) * Optimize performance and reduce memory when creating Quantity from scalar Signed-off-by: Fabio Di Fabio <[email protected]> * Benchmark other implementations Signed-off-by: Fabio Di Fabio <[email protected]> --------- Signed-off-by: Fabio Di Fabio <[email protected]> * snap sync - apply BALs before flat db heal (#10151) Signed-off-by: Miroslav Kovar <[email protected]> * Remove dryRunDetector workaround methods from unit tests (#10201) * Remove dryRunDetector workaround methods from unit tests The dryRunDetector methods were added as a workaround for a Gradle issue that prevented @ParameterizedTest classes from being selected when running with --dry-run. Since the issue is fixed and --dry-run is no longer used, these methods are no longer needed. Signed-off-by: Fabio Di Fabio <[email protected]> * Remove dryRunDetector workaround from acceptance tests too The Gradle issue is confirmed fixed, so the workaround is no longer needed anywhere, including acceptance tests. Signed-off-by: Fabio Di Fabio <[email protected]> --------- Signed-off-by: Fabio Di Fabio <[email protected]> * preserve state gas reservoir for the top level frame in case of OOG (#10205) Signed-off-by: daniellehrner <[email protected]> * Enable execution processor on PoA networks with system contract addresses (#10196) * enable the prague execution processor for poa networks that have the systems contract addresses set in their genesis file Signed-off-by: daniellehrner <[email protected]> * Fix engine_getPayloadV1 to return -38001 instead of -32001 for unknown payloadId (#10179) The Engine API spec requires error code -38001 (Unknown payload) when engine_getPayloadV1 is called with an unrecognized payloadId. Besu was incorrectly returning -32001 (Resource not found), which is a non-standard error code that may cause interoperability issues with consensus layer clients. Fixes #10174 Signed-off-by: Vivek Singh Solanki <[email protected]> * Exclude IntelliJ bin/default output from Spotless shell script check (#10210) When IntelliJ syncs a Gradle project without build delegation, it copies processed resources (including reference test shell scripts from the submodule) into bin/default/. Spotless then finds these copies and incorrectly flags them for missing license headers, while CI never sees bin/default/ since it runs bare Gradle. Add '**/bin/default/**' to the ShellScripts targetExclude, matching the existing pattern used for other generated/external content. Signed-off-by: Simon Dudley <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> * Missing memory presence check (#10213) * Call lastFrame.getMemory().isPresent() before calling lastFrame.getMemory().get().length to avoid NPE Signed-off-by: daniellehrner <[email protected]> Co-authored-by: Copilot <[email protected]> --------- Signed-off-by: Ameziane H. <[email protected]> Signed-off-by: Fabio Di Fabio <[email protected]> Signed-off-by: daniellehrner <[email protected]> Signed-off-by: Miroslav Kovar <[email protected]> Signed-off-by: Vivek Singh Solanki <[email protected]> Signed-off-by: Simon Dudley <[email protected]> Co-authored-by: ahamlat <[email protected]> Co-authored-by: Sally MacFarlane <[email protected]> Co-authored-by: Fabio Di Fabio <[email protected]> Co-authored-by: Miroslav Kovář <[email protected]> Co-authored-by: Vivek Singh Solanki <[email protected]> Co-authored-by: Simon Dudley <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Copilot <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pass the first pivot, stored in
ChainSyncStateand persisted inChainSyncStateStorage, throughSnapSyncProcessStatetoSnapWorldDownloadState(+ make sure to preserve it when recreatingSnapSyncProcessStateinSnapSyncDownloader#storeState) where it is used incheckCompletionto decide whether BAL requests should be enqueued. Due to dependencies between request queues, enqueueing BAL requests blocks flat db requests. This means that BALs are applied (when necessary) strictly after trie healing and before flat db healing phase. Moreover, BAL request enqueuing is gated behindAtomicBoolean(+checkCompletionis synchronized) to happen only once.Batched BAL responses are then passed through
PersistDataStep#persisttoBlockAccessListDataRequest#doPersistwhere they are applied to the providedBonsaiWorldStateKeyValueStorage.Updaterand later committed to storage. We make sure to recompute the storage root if any slots were modified.Some functionality shared with
BalStateRootCalculator(where BALs are applied to world state in a similar way) was extracted toBlockAccessListChanges. This results in extra copy - not sure about performance impact.Follow up PR will introduce a functionality to the
Pipelinewhich will allow to aggregate batches of items into a single batch, so that all changes from BALs are applied to the same updater and committed in one atomic operation.