feat(mempool): incremental recheck #909
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…mpletion) and into a seperate Cancel function
| for addr, txs := range t.txs { | ||
| sort.Sort(types.TxByNonce(txs)) | ||
|
|
||
| // Filter by minimum tip if configured | ||
| if minTipBig != nil { | ||
| for i, tx := range txs { | ||
| if tx.EffectiveGasTipIntCmp(minTipBig, baseFeeBig) < 0 { | ||
| txs = txs[:i] | ||
| break | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // Convert to lazy transactions | ||
| if len(txs) > 0 { | ||
| lazies := make([]*txpool.LazyTransaction, len(txs)) | ||
| for i, tx := range txs { | ||
| lazies[i] = &txpool.LazyTransaction{ | ||
| Hash: tx.Hash(), | ||
| Tx: tx, | ||
| Time: tx.Time(), | ||
| GasFeeCap: uint256.MustFromBig(tx.GasFeeCap()), | ||
| GasTipCap: uint256.MustFromBig(tx.GasTipCap()), | ||
| Gas: tx.Gas(), | ||
| BlobGas: tx.BlobGas(), | ||
| } | ||
| } | ||
| numSelected += len(lazies) | ||
| pending[addr] = lazies | ||
| } | ||
| } |
Check warning
Code scanning / CodeQL
Iteration over map Warning
| } | ||
|
|
||
| evmPendingTxs := m.txPool.Pending(txpool.PendingFilter{ | ||
| ctx, cancel := context.WithTimeout(ctx, time.Millisecond*100) |
There was a problem hiding this comment.
Q: Can we fetch consensus params from the context and replace ms with something that correlates with the configured propose timeout?
There was a problem hiding this comment.
yes good idea, still trying to figure out what a good value for this is
|
|
||
| // runReorg runs reset and promoteExecutables on behalf of scheduleReorgLoop. | ||
| func (pool *LegacyPool) runReorg(done chan struct{}, reset *txpoolResetRequest, dirtyAccounts *accountSet, events map[common.Address]*SortedMap) { | ||
| func (pool *LegacyPool) runReorg(done chan struct{}, cancelReset chan struct{}, reset *txpoolResetRequest, dirtyAccounts *accountSet, events map[common.Address]*SortedMap) { |
There was a problem hiding this comment.
Should we replace these args with reorgInput{}?
There was a problem hiding this comment.
yeah there is a lot i can consolidate to a struct
|
|
||
| // isResetCancelled returns true if the pool is resetting and it has been | ||
| // signaled to cancel the reset. | ||
| func (pool *LegacyPool) isReorgCancelled(reset *txpoolResetRequest, cancelled chan struct{}) bool { |
There was a problem hiding this comment.
This can be moved to a standalone function as it doesn't use the pool
| } | ||
|
|
||
| // add adds txs to the current set. | ||
| func (t *txs) Add(addr common.Address, txs types.Transactions) { |
There was a problem hiding this comment.
It's possible to write a duplicate here. How does the caller ensure that no duplicates are written?
There was a problem hiding this comment.
hm you're right. I think a scenario like:
- new block arrives
- new tx arrives that replaces a tx in the pending queue (this will get added to txs struct via markTxPromoted)
- reorg loop runs with reset != nil
- tx that just replaced the tx in pending is validated again during demote unexecutables and added to txs structure again
- ahhh
I can add in protection to make sure no duplicates are added.
mattac21
force-pushed
the
ma/incremental-recheck
branch
from
3c991da to
392848e
Compare
mattac21
force-pushed
the
ma/incremental-recheck
branch
from
b8e1b2b to
9134baf
Compare
| // pending limit. The algorithm tries to reduce transaction counts by an approximately | ||
| // equal number for all for accounts with many pending transactions. | ||
| func (pool *LegacyPool) truncatePending() { | ||
| defer func(t0 time.Time) { pendingTruncateTimer.UpdateSince(t0) }(time.Now()) |
Check warning
Code scanning / CodeQL
Calling the system time Warning
| func (pool *LegacyPool) markTxRemoved(tx *types.Transaction, p PoolType) { | ||
| func (pool *LegacyPool) markTxRemoved(addr common.Address, tx *types.Transaction, p PoolType) { | ||
| if p == Pending { | ||
| defer func(t0 time.Time) { pendingRemoveCBTimer.UpdateSince(t0) }(time.Now()) |
Check warning
Code scanning / CodeQL
Calling the system time Warning
|
|
||
| // RemoveTx removes a tx from the collector. | ||
| func (c *txCollector) RemoveTx(addr common.Address, tx *types.Transaction) { | ||
| defer func(t0 time.Time) { collectorRemoveDuraiton.UpdateSince(t0) }(time.Now()) |
Check warning
Code scanning / CodeQL
Calling the system time Warning
mattac21
force-pushed
the
ma/incremental-recheck
branch
2 times, most recently
from
e87dacf to
9134baf
Compare
Description
This adds the ability for proposals to be created out of txs in the pending pool while the pool is resetting via a new block. Previously, this resetting would hold the lock on the pool for the entire duration of the reset, which would need to call recheck on every tx in the pending pool. This may take multiple seconds, which will block a call to
Pendingand therefore blockPrepareProposal.This PR addresses this by pushing txs into a
txCollectoras they are validated viademoteUnexecutables. Creating a 'snapshot' of the pending txs that have been validated for a height at any given time. This snapshot no longer shares the same global legacypool lock, thus unblockingPrepareProposal's call toPending.This PR also adds functionality where the reorg loop will be cancelled if it is still processing a reset (new block seen), and a new block arrives. This is to prevent the case where
demoteUnexecutablestakes such a long time to process txs that processing the previous height, always takes longer than the timeout the caller is willing to wait inPending, thus preventing any txs from ever making it into a proposal.Closes: STACK-2008
Author Checklist
All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.
I have...
mainbranch