stagedsync: replace TxLookup prune full-table scan with block-based deletion#19975
Closed
stagedsync: replace TxLookup prune full-table scan with block-based deletion#19975
Conversation
…-based deletion PruneTxLookup previously used TableScanningPrune which scans the entire TxLookup table (keyed by random tx hashes) to find entries within a txNum range — O(table_size). At chain tip this burned a constant ~2s per block even when there was nothing to prune, wasting ~17% of the slot budget. Replace with deleteTxLookupRange which iterates HeaderCanonical for the block range, looks up each block's transactions, and deletes their TxLookup entries directly — O(txs_to_delete). This is the same approach already used by UnwindTxLookup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bound the work done inside a single DB transaction to avoid occupying too much of the 12-second slot budget when the node falls slightly behind. During initial sync the limit is not applied. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
domiwei
requested review from
AskAlexSharov and
JkLondon
and removed request for
JkLondon,
mh0lt and
yperbasis
Member
|
it's not a full scan in one iteration. We have progress tracking of scan exactly because we want to avoid this. So, probably, the reason of bug is that txTo changes to often or table is just enormously large need to investigate. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#19890
TableScanningPrunewhich scans the entireTxLookup table (keyed by random tx hashes) to find entries within a txNum
range — O(table_size). At chain tip this burned a constant ~2s per block
even when there was nothing to prune, wasting ~17% of the 12s slot budget.
deleteTxLookupRangewhich iterates HeaderCanonical for theblock range, looks up each block's transactions, and deletes their TxLookup
entries directly — O(txs_to_delete). This is the same approach already used
by UnwindTxLookup.
Background
PR #19179 ("prune unification") moved TxLookup to the generic
TableScanningPruneframework. That framework works well for tables whose keys contain txNum/step
(domain, history, inverted_index), but TxLookup keys are random tx hashes with
txNum buried in the value — forcing a full table scan even when 0 entries need
deleting.
Test plan
make lintmake test-shortmilliseconds instead of ~2s at chain tip