perf(mempool): make transaction acceptance cost independent of mempool size

[rabbitson87]

Two commits, both about the same thing: accepting a transaction cost time quadratic in mempool size, on the path that accepts transactions from peers. Anyone able to fill the mempool made every subsequent acceptance more expensive.

1. The priority index re-sorted itself on every insert

ParetoFront::insert held its keys in a flat vector and did a linear remove followed by sort_by over the whole index. Filling an index of n entries was O(n² log n).

Replaced with an ordered set keyed by the priority comparison, plus a map from entry id to the key currently indexed for it. Both operations are O(log n).

Entries	before	after	ratio
1,000	2.064 ms	151.0 µs	13.7x
4,000	45.65 ms	626.2 µs	72.9x
16,000	464.7 ms	3.046 ms	152.6x
50,000	4.497 s	10.45 ms	430.4x

Measured exponent 1.97 for the old arm — n². Both arms run in one process over one fixture.

The id→key map is not redundant: removals arrive as an entry id while the ordered set is keyed by priority, so without it a removal would have to search the set. The ordering's final tiebreak on entry id was cosmetic before and is now load-bearing — the index is a set of keys, so two entries whose keys compared equal would collapse into one and a transaction would silently leave the priority index.

2. That alone did not close it

insert_entry also called recompute_all_metadata, which walked every entry, and then total_vsize(), which folded every entry again. With the index fixed and nothing else changed, insertion still measured an exponent of 2.17.

The metadata refresh is now incremental. Linking one transaction into the spend graph changes package totals for its transitive ancestors, itself, and its transitive descendants — and nothing else. An entry x outside that set gains no new ancestor, because x is not a descendant of the seed; and gains no new descendant, because every new path runs through the seed, which would put x among its ancestors. The closure's size is bounded by the ancestor and descendant policy limits (25 each by default), not by the pool.

Order matters in both directions, and both are commented at the call sites: the closure is taken after the entry enters the spend indexes on an insertion (a transaction can arrive after something that already spends its outputs), and before the removal on a removal (a removed entry's ancestors cannot be walked once it is gone).

total_vsize and aggregate_fees are now running sums, guarded by debug_asserts against the folds they replaced.

Transactions	before	after	ratio	per tx, after
200	2.572 ms	373.0 µs	6.9x	1.87 µs
800	51.27 ms	1.760 ms	29.1x	2.20 µs
3,200	1.057 s	7.079 ms	149.3x	2.21 µs
12,800	not measurable	37.87 ms	—	2.96 µs
51,200	not measurable	211.5 ms	—	4.13 µs

The last two sizes could not be measured before — at exponent 2.17, 51,200 transactions is about seven minutes per sample. Per-transaction cost is now nearly flat across 256x the entries, and the exponent over the final leg is 1.24: n log n for the fill, so about O(log n) per accepted transaction.

A defect fixed on the way

prioritise applied its fee delta to each descendant's ancestor_fee by hand and then never reindexed those descendants, so a descendant kept the priority key it had from before its ancestor was bumped. Since prioritisetransaction exists to move transactions in the miner's template, leaving descendants ranked on the pre-bump figure defeated it for exactly the packages it was aimed at. The three delta loops are now one call to the same refresh an insertion does.

Correctness

Two oracles, both retained rather than deleted:

• SortedParetoFront — the flat-vector index, the before arm and the ordering oracle.
• recompute_all_metadata — kept under cfg(test). Nothing in the pool reaches it, so production cannot drift away from it. Each equivalence test drives the incremental path, then runs the full recompute, and asserts nothing moved.

Mutation audit — thirteen mutations, all killed on their named tests:

Mutation	Result
a replacement leaves the stale key in the ordered set	2 failed
the ordering drops its entry-id tiebreak	3 failed
the ordering puts the lowest fee rate first	3 failed
remove forgets the ordered set	2 failed
the refresh closure names only the seed	5 failed
the closure forgets descendants	3 failed
the closure forgets ancestors	2 failed
a removal takes its closure after the removal	1 failed
descendant totals count only the entry itself	5 failed
the refresh skips the priority reindex	2 failed
an insertion forgets the running vsize total	29 failed
a removal forgets the running fee total	1 failed
prioritise forgets the running fee total	1 failed

Three of these survived a first pass and are why the audit was worth running.

The oracle originally shared ParetoKey's Ord with the replacement, which looked tidy and made it worthless: under the reversed-ordering mutation both implementations agreed with each other, so both equivalence tests stayed green while the index was ordered backwards. The oracle now keeps its own verbatim copy of the comparison. An oracle that shares code with the implementation cannot disagree with it.

The last two rows also survived. total_vsize and aggregate_fees each guard themselves with a debug_assert, but a guard only fires when something calls it. insert_entry consults total_vsize() on every acceptance, so its bookkeeping was covered by accident — the 29 failures are that accident. aggregate_fees() is only reached through stats(), and no test called it after a removal or a fee bump, so deleting the bookkeeping in both paths turned nothing red. Now covered, and compared against an independent fold so the check survives a release build where the debug_asserts are compiled out.

Verification

cargo test -p bitcoin-rs-mempool -p bitcoin-rs-mining -p bitcoin-rs-electrum --no-fail-fast green (89 lib + 5 pareto_ordering + downstream). cargo fmt --check and cargo clippy -p bitcoin-rs-mempool -p bitcoin-rs-mining -p bitcoin-rs-electrum -p bitcoin-rs-rpc -- -D warnings clean. The 22 expect_used findings under --all-targets are pre-existing on main (verified by stashing).

Full write-up: docs/benchmarks/mempool-pareto.md.

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bf2eef24-7f86-4908-880d-bbd2cf82ca37

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}