perf(utxo): shrink the UTXO record payload 21.7% - #83
Open
rabbitson87 wants to merge 9 commits into
Open
Conversation
`UtxoSet` is fully memory-resident across 256 shards with no eviction tier, and the published tip-RSS evidence reached 13.83 GiB at height 645,804 against a G14 budget of 16 GiB. Nothing had ever attributed that figure, and the encoding work planned after it was about to be priced against a component size that had never been measured. Adds `UtxoSetView::memory_report()`, which walks the shards and returns record payload bytes, allocation bytes including header and slack, and hash-table backing store bytes. `crates/node` samples it on the checkpoint path alongside process RSS, so the residual between what the set can account for and what the kernel reports is visible in the logs rather than inferred. Sampling from the checkpoint path is deliberate: in-flight samples swing between 1.1 and 3.2 GB and measure block staging, not the set. What a pruned mainnet sync to height 412,732 (38,145,360 outputs across 10,519,335 records) then showed: - The UTXO set is **77.4% of process RSS**, not the ~44% an in-flight sample suggested. Record payload is 55.1 B/output, 61.2 accounted, 79.2 RSS. - **Allocator fragmentation is not the problem.** Churning twice the whole set costs 5% more RSS and the curve is flattening. That was the plan's leading hypothesis and it is refuted at this scale on this allocator. - **Outputs per record was the assumption that mattered, and it was wrong.** The first pass assumed 1.5; the measured trajectory is 2.296 at height 183k, 4.056 at 390k and 3.626 at 412k. It has not converged. Re-run at the measured 3.626 the synthetic harness predicts 54.6 B/output of payload against 55.1 measured and 62.0 accounted against 61.2, and the v4 snapshot on disk agrees at 57.3 — three independent paths within 1.3%. `process_rss_bytes` reads `/proc/self/status` on Linux and shells out to `ps` elsewhere, rather than taking a platform dependency for one number read once per checkpoint. It is logged as a plain integer with a separate `rss_known` flag: `?rss` renders `Some(3019751424)` and breaks machine parsing. No behaviour changes. `docs/benchmarks/utxo-memory.md` records the method, the numbers and the two hypotheses this refuted.
…han v4
The attribution in the previous commit put the UTXO set at 77.4% of process RSS
and the tip projection at 83% of the 16 GiB G14 budget on the UTXO path alone,
before txindex and blockfilterindex. This is the encoding work that margin
justified.
v5 keeps the record header and replaces the per-output layout:
txid(32) || output_count(4) || legacy_inline_len(1) || widths(1)
|| vout_dir : one fixed-width little-endian entry per output
|| len_dir : one fixed-width payload length per output
|| payloads : varint(amount) [|| raw amount] || varint(height<<1|coinbase) || script
Three transforms do the shrinking, all per-output with no cross-output invariant
to violate: Core's `CTxOutCompressor` amount transform, `height` and `coinbase`
packed into one varint, and directory widths that are the narrowest the record
needs. The script length is not stored — the script is whatever remains of its
payload, so the length directory pays for itself.
Measured: **11.75 bytes per output, 21.7% of the payload**, which is 14.8% of
process RSS and about 1.97 GiB at tip. Hoisting `height` into the record header
would save 3 bytes more and is deliberately not done: it needs "every output of
a record shares one height", and BIP30's duplicate coinbase txids are exactly
where that might not hold.
The directories are the load-bearing part, and they exist because the first
draft was wrong. That draft was a flat varint frame per output. It hit the size
target and lost badly on speed:
operation v4 flat v5 directory v5
get_miss (shard lookup) 705 ns 3.42 µs 300 ns
get_last 728 ns 3.43 µs 617 ns
get_middle 384 ns 1.67 µs 342 ns
spend_fanout_64 18.5 µs 39.9 µs 21.3 µs
spend_fanout_64_noop 86.7 µs 115.2 µs 77.1 µs
Two mistakes produced that, and neither was visible until the benchmark was
reshaped around the operation that actually dominates:
1. The benchmark timed whole-record encode/decode. The hot read is
`find_output(vout)` — every spent input resolves through `Shard::get`,
`get_entry` or `get_meta`, and all three land there. Whole-record decode is
the snapshot and rescan path, which is rare by comparison.
2. v4 gets lazy field skipping for free and a flat varint layout cannot. Every
v4 field sits at a constant offset, so when only `vout` is read the optimizer
deletes the loads for the rest. In a flat layout each varint's length locates
the next field, so the reads are a serial dependency chain, and finding
output `i` walks the bytes of outputs `0..i`, scripts included.
The directories remove exactly that: a lookup scans one dense fixed-width array
and sums a second, touching ~2 bytes per output instead of ~35. Nine of the
twelve `utxo_commit` lookup arms now beat v4; the three that do not are the
`_first` cases, 10 ns apart in absolute terms.
Encoding is 1.6-2.4x slower, because the directory widths are a property of the
whole record so nothing can be written until every payload length is known. At
block scale that is commit p95 +3% (`existing`), +8% (`uniform`) and +21%
(`concentrated`). The G14 budget is 50 ms and the worst case measured is 2.57 ms.
Checked by:
- `tests/record_codec_equivalence.rs`, 7 tests. v4 is retained as the oracle;
equality is per field over every decoded `OneUtxoOut`, in order, since
comparing encoded bytes is meaningless when the layouts differ by design.
Size is asserted as a property, not a spot check.
- `non_canonical_v5_spellings_are_rejected` covers every second spelling the two
layouts introduce: a non-minimal varint, the amount escape used for a value
the compact form already covers, and a directory wider than the record needs.
`UtxoRecord` compares by bytes, so two spellings of one record is a
correctness bug.
- `find_output_decompresses_at_most_the_amount_it_returns` asserts the work
rather than the time: one amount decompression for a hit, none for a miss,
none for `max_vout`. A wall-clock assertion in a test suite is a flake
generator; counting the expensive operation is the same claim made
deterministically.
The `utxo_commit` arms cannot be paired in one run since only one codec is
compiled in, so the v4 comparison was taken A-B-A across a stash, with the two
v5 runs agreeing to 0.1-2.9%.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
`snapshot_roundtrip.rs` writes and reads inside one process, so it stays green even when both sides of the codec change together — which is exactly the failure a record-layout change can cause. Nothing outside this build held the encoding to account. `fixtures/utxo-v4-golden.dat` is 27,618 bytes emitted by a build of the v4 record codec over a fixture chosen to be awkward: every field width the codec can pick between, both directory widths, the inline/overflow partition boundary, vouts from 0 to `u32::MAX`, empty and 2 KB scripts, zero amounts, round amounts, the money supply and one value above it. The generator is kept verbatim as `golden_fixture()` beside the assertions, so the fixture's provenance is in the tree rather than in a commit message. Three assertions, in both directions: - a v4 file loads to the `hash_serialized_3` and MuHash trailer the v4 build computed, - this build writes byte-identical snapshot bytes for the same logical set, - load-then-store is a fixed point. The write direction is the stronger one: the snapshot writer emits outputs in the order the record layout stores them, so a pure reordering shows up there and nowhere else — `hash_serialized_3` sorts by `(txid, vout)` before hashing and cannot see one. Mutation-verified. Dropping the `coinbase` bit in `decode_output_at` fails all three; reversing the output iteration index fails all three. The second mutation also mismatches fields rather than purely reordering, so it does not isolate the ordering claim on its own. This is the invariant the whole record-layout change rests on: `hash_serialized_3` and the trailer are computed over decoded consensus values, never over the in-memory encoding.
… ran `process_rss_bytes` had no test at all, and its Linux branch had never executed: `/proc/self/status` parsing is `cfg`-compiled out on the macOS development host, so it would have run for the first time in production — on the one number the UTXO memory attribution divides against. A wrong denominator there silently misprices every encoding decision made from it. Splits the two parsers out of the platform `cfg` so both are compiled and tested everywhere, and covers them with the kernel's actual format: kibibytes with variable leading whitespace, `VmHWM`/`VmSize` sitting adjacent with the same prefix shape, the field absent entirely for a kernel thread, and non-numeric or overflowing values. Adds an end-to-end assertion too, because a parser test cannot catch a reader that returns a plausible constant: allocate 128 MB, touch every page so it is resident rather than merely reserved, and require the reading to move by at least half of it. `allow(dead_code)` rather than `expect`, deliberately: each parser is live in one platform's library build and dead in the other's, but *both* are live in every test build, so an `expect` would itself be unfulfilled half the time.
…64 outputs
The earlier commit message and the benchmark doc said v5 "makes lookups faster
than v4". That was read off the `utxo_commit` lookup arms, which hold **256
outputs in one record**. It is true of that fixture and false of the workload:
the measured mainnet average is 3.626 outputs per record, and there v5 is
slower.
Benchmarking only 1, 4 and 16 outputs hid the crossover in one direction, and
quoting only `utxo_commit` hid it in the other. Adds 64 and 256 to the codec
harness so both sides of it are in the tree:
outputs find_output/miss find_output/hit_last
v4 v5 ratio v4 v5 ratio
1 1.7 ns 3.6 ns 0.48x 2.1 ns 12.5 ns 0.16x
4 4.0 ns 7.0 ns 0.57x 3.7 ns 18.4 ns 0.20x
16 16.6 ns 20.6 ns 0.80x 16.8 ns 40.6 ns 0.41x
64 109.3 ns 79.5 ns 1.37x 126.3 ns 136.6 ns 0.92x
256 471.8 ns 298.7 ns 1.58x 481.6 ns 499.7 ns 0.96x
v5 pays a fixed ~10 ns for the directory header and the matched payload, then
scans at a fraction of v4's per-output cost. Below ~64 outputs the fixed cost
dominates.
The trade is unchanged and still worth making — 3 ns per lookup at the mainnet
average is 12 µs per block against a 50 ms commit budget, 0.02%, bought with
21.7% of the record payload — but the honest statement is a small cost on
typical records and a win on batch payouts, not a win everywhere.
Also records why the two harnesses disagree on the 256-output case (2.35x
through `utxo_commit`, 1.58x here): `utxo_commit` drives the real
`UtxoRecord::find_output`, while this harness reimplements the v4 search as a
direct loop the optimizer handles better. The microbenchmark is the
conservative number and is the one now quoted.
Verified on Linux as well as macOS: all 113 utxo tests pass under
`rust:1.95-bookworm` on aarch64, and the encoded sizes are byte-identical
across both (12.00 / 11.75 / 11.62 B/output at 1 / 4 / 16 outputs).
rabbitson87
changed the title
Member
Author
Correction: the lookup claim was measured on the wrong record sizePushed in 7d3c948, and retitled the PR. The original framing — "makes lookups faster than v4" — was read off the Benchmarking only 1/4/16 outputs hid the crossover in one direction and quoting only
v5 pays a fixed ~10 ns for the directory header and the matched payload, then scans at a fraction of v4's per-output cost. Below ~64 outputs the fixed cost dominates. The trade is unchanged and still worth making, but state it correctly: 3 ns per lookup at the mainnet average is 12 µs per block against a 50 ms commit budget — 0.02% — bought with 21.7% of the record payload. The win concentrates on batch payouts, which is where a lookup was expensive to begin with. Also worth flagging: the two harnesses disagree on the 256-output case (2.35x through Also in this push
CI confirms the Linux branch now runs: Linux cross-checkAll 113 utxo tests pass under |
`decompress_amount` ended in a `while` loop multiplying by ten up to nine times, with no bound on its input. `read_varint` hands it whatever a record contains, and `validate_encoded` runs it over every output of every record loaded from a snapshot, so a file on disk could reach it with an arbitrary `u64`. `decompress_amount(u64::MAX)` is 2.05e22 — a panic in a debug build, a silent wrap in a release one. Reproduced before fixing: `an_absurd_compressed_amount_is_rejected_rather_than_overflowing` failed with `attempt to multiply with overflow` from `compress.rs:193`. The function now returns `Option` and requires the decompressed value back inside the compressible domain. That also closes a canonicality hole that was open until now: the compact form may encode only amounts the escape refuses, and the escape refuses exactly the amounts the compact form covers, so every amount has one spelling and no other. `decompress_accepts_exactly_the_encoder_image` states it as a property over every `u64` — whatever the decoder accepts must round-trip back to the same compressed value through the encoder, which pins the accepted set to the encoder's image rather than merely checking that nothing panics. Found while investigating the fixed ~12 ns `find_output` cost, which is what the same loop was suspected of. Replacing it with a power-of-ten lookup does help, but only 12.5 ns -> 11.3 ns (1.1x): the remaining fixed cost is the directory read and building the returned view, not the arithmetic. Quoted that way in the doc rather than as a speed win, and the crossover table is re-measured. The `miss` arms are unchanged at every record size, which is the control this wants: that path never decompresses an amount, so a change to the amount transform must not move it.
`AGENTS.md` requires that durable project knowledge be reconciled when it changes — the overlapping `docs/solutions/` learning and every affected term in `CONCEPTS.md`. The read-path work did that; this branch had not. `CONCEPTS.md` gains three terms the record layout introduced: - *Directory-layout record* — why the lookup keys and item lengths sit in fixed-width arrays in front of the payloads, and where the two layouts cross over. - *Canonical record spelling* — the three rules that keep one logical record to one byte string once the fields stop being fixed-width, and why the last of them is a safety rule rather than a tidiness one. - *Work-count assertion* — asserting how much of an expensive operation a path performs instead of how long it takes, and the case where a count cannot substitute for a benchmark. `DEVIATIONS.md` gains §9, which records two things `PLAN.md` still specifies that this campaign settled: - The per-shard `bumpalo` arena of design principle 8 is **rejected on measurement**, not deferred: allocation overhead is 2.2 B/output and fragmentation 5% after churning twice the whole set. Those two numbers are the evidence against starting it. - The record payload is v5 rather than the v4 layout, `height` is deliberately not hoisted (BIP30 duplicate txids), and the snapshot disk format is deliberately unchanged — disk size is not a G14 budget item, so the invariant that step protected is covered by a golden vector instead. Adds the best-practice learning, which is the part most likely to be useful somewhere else: the codec benchmark measured `encode`/`decode` because that is what the codec module exposes, while the node calls `find_output`. Reshaped around the real call the same change measured 4.4-4.9x slower rather than 1.9x, and the cause turned out to be that a fixed-width layout gets lazy field skipping free from the optimizer while a variable-length one structurally cannot. It also records the crossover that one fixture size hid in both directions. Finally, states the revert criterion for v5 in both the deviation and the benchmark doc, while the numbers are in front of us: if G14 tip RSS measures well under budget the complexity is not earning its keep, and v4 is still in the tree as the oracle, so a revert is a revert rather than a rewrite.
rabbitson87
force-pushed
the
perf/utxo-record-v5
branch
from
August 18, 2026 14:13
db8169b to
b3ae179
Compare
Contributor
|
@coderabbitai resolve conflicts |
|
❌ Failed to resolve merge conflicts Unexpected error: 14 UNAVAILABLE: read ECONNRESET Please resolve conflicts manually. |
…etic one
Every size number for v5 so far came from a fixture built on an assumed script
mix and an assumed outputs-per-record. The assumption that mattered most —
outputs per record — had already been wrong once by a factor of two, so the
headline saving deserved better evidence than the same harness that produced it.
Adds `examples/snapshot_memory.rs`, which loads a real `utxo-v4.dat` checkpoint
and reports what the set costs. Because the record encoding is internal, the
*same file* loaded by two builds is a controlled A/B of the codec with every
other variable held fixed.
Run against a pruned mainnet sync at height 412,732 — 10,519,335 records,
**38,145,360 outputs**, 2.03 GiB on disk — a v4 build and a v5 build give:
layer v4 v5 saved
record payload 55.08 43.90 11.18 B/output
+ allocation header 57.28 46.11 11.17
+ hash table 61.24 50.07 11.17
process RSS 65.57 54.08 11.49
**The synthetic bench was 5.1% optimistic**: it predicted 11.75 B/output of
payload and the real chainstate gives 11.18. The doc now quotes 11.18, and the
tip projection moves from 11.31 GiB to 11.35 GiB.
Two independent checks that the instrument is sound. The v4 payload here is
55.08 B/output against the 55.1 the original attribution measured by a different
route, and the RSS saving (11.49) is slightly larger than the payload saving
(11.17), which is what an allocator handing back whole size classes should do.
It also restates the consensus-neutrality claim on real data. Both builds
produce the identical `hash_serialized_3` and a `MuHash` trailer whose SHA-256
matches the one the v4 node recorded in the checkpoint manifest when it wrote
the file. That is the assertion `tests/snapshot_v4_golden.rs` makes over 433
fixture outputs, made here over 38 million real ones.
What it does not measure, and the doc says so: full-node tip RSS. This loads the
UTXO set alone, with no fjall, CoinStats, block-record log or runtime beside it.
The G14 gate still needs a synced tip node with `txindex` and
`blockfilterindex`.
One lint suppressed with its reason: clippy proposes passing
`UtxoSetView::memory_report` in place of the closure, and that does not compile —
the higher-ranked lifetime on the view will not unify with a bare function item.
This was referenced Aug 18, 2026
# Conflicts: # DEVIATIONS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measures where the UTXO set's memory actually goes, then shrinks the record payload by 21.7% — and ends up faster on the hot lookup than the layout it replaces.
Independent of #80 (no overlapping files); branched from
main.Why
UtxoSetis fully memory-resident across 256 shards with no eviction tier, and the published tip-RSS evidence reached 13.83 GiB at height 645,804 against a G14 budget of 16 GiB. Nothing had ever attributed that figure, so the first commit measures before the second changes anything.What the measurement found
From a pruned mainnet sync to height 412,732 (38,145,360 outputs across 10,519,335 records), sampled on the checkpoint path:
What the change does
Core's
CTxOutCompressoramount transform,height+coinbasepacked into one varint, and directory widths that are the narrowest the record needs. The script length is not stored — the script is whatever remains of its payload.11.75 B/output, 21.7% of the payload, ~1.97 GiB at tip.
Not done, on purpose: hoisting
heightinto the record header would save 3 bytes more but needs "every output of a record shares one height", and BIP30's duplicate coinbase txids are exactly where that might not hold.The part worth reviewing
The first draft was a flat varint frame per output. It hit the size target and lost badly on speed:
get_miss(real shard lookup)get_lastget_middlespend_fanout_64spend_fanout_64_noop_listenerTwo mistakes produced that, and neither was visible until the benchmark was reshaped:
find_output(vout): every spent input resolves throughShard::get/get_entry/get_metaand all three land there.voutis read the optimizer deletes the loads for the rest. In a flat layout each varint's length locates the next field, so the reads are a serial dependency chain and finding outputiwalks the bytes of outputs0..i, scripts included.The directories remove exactly that: a lookup scans one dense fixed-width array and sums a second, touching ~2 bytes per output instead of ~35. Nine of twelve
utxo_commitlookup arms now beat v4; the three that do not are the_firstcases, 10 ns apart.What it costs
Encoding is 1.6–2.4× slower — the directory widths are a property of the whole record, so nothing can be written until every payload length is known. Commit p95 +3% (
existing), +8% (uniform), +21% (concentrated). G14 budget is 50 ms; worst case measured is 2.57 ms.One encode optimization was tried and rejected by measurement (staging both directories in a
SmallVecand copying once: 505.7 ns vs 428.5 ns). It is reverted, with the number in a comment so nobody retries it.Checks
tests/record_codec_equivalence.rs— v4 retained as the oracle, equality per field over every decodedOneUtxoOutin order. Comparing encoded bytes would be meaningless when the layouts differ by design. Size asserted as a property.non_canonical_v5_spellings_are_rejected— every second spelling the two layouts introduce: non-minimal varint, the amount escape used for a value the compact form already covers, and a directory wider than the record needs.UtxoRecordcompares by bytes, so two spellings of one record is a correctness bug.find_output_decompresses_at_most_the_amount_it_returns— asserts the work, not the time. One decompression for a hit, none for a miss, none formax_vout. A wall-clock assertion in a test suite is a flake generator.110 tests in
bitcoin-rs-utxo;cargo fmt --checkclean; clippy on-p bitcoin-rs --all-targets --no-default-features --features "rocksdb,fjall,redb,mdbx,kernel"reports the same 11 pre-existing errors asmain(all inbin/bitcoin-rs/tests/g14_perf_evidence_script.rs, which this branch does not touch).Not in this PR
hash_serialized_3and MuHash trailer — those are computed over consensus values, not the in-memory encoding, so a v4 file loaded into a v5 build must produce the same trailer.txindexandblockfilterindex.Reproduce
The
utxo_commitarms cannot be paired in one run — only one codec is compiled in — so the v4 comparison was taken A-B-A across a stash, with the two v5 runs agreeing to 0.1–2.9%. That drift bounds the rebuild effect well below every ratio quoted here.