perf(utxo): shrink the UTXO record payload 21.7%

CONCEPTS.md

@@ -282,3 +282,12 @@ A Criterion group holding the before and after implementations of one change ove
 
 ### Resolution-time sampling
 Recording a statistic when its outcome is known rather than when the subject arrives. The fee estimator counted a transaction against every confirmation target the moment it entered, so a fresh arrival was already a failure at every target and a burst silenced the estimator before anything had missed a deadline. It also broke the decay: the denominator had been decaying since entry while a confirmation arrived undecayed, reporting 81 successes in 100 as roughly 85%. Sampling numerator and denominator together at the moment a target resolves fixes both, because they then decay from the same block. The counterpart rule is that a subject leaving for an unrelated reason is untracked without being sampled: an eviction says something about the mempool, not about whether the transaction would have confirmed.
+
+### Directory-layout record
+A record that keeps its per-item lookup keys and item lengths in fixed-width arrays in front of the items, rather than inline with them. `UtxoRecord` v5 is `txid || output_count || legacy_inline_len || widths || vout_dir || len_dir || payloads`, where each directory entry is the narrowest little-endian width the record needs. The reason is random access: the hot read is `find_output(vout)` — every spent input resolves through `Shard::get`/`get_entry`/`get_meta` — and in a flat variable-length layout each field's length is what locates the next one, so finding output `i` walks the bytes of outputs `0..i`, scripts included. That was measured at 4.4-4.9x slower than the fixed-width v4 it replaced. With directories a lookup scans one dense byte array and sums a second, touching about two bytes per output instead of thirty-five. The two layouts cross over near 64 outputs: below it v5's fixed setup dominates and it is about 3 ns slower at the measured mainnet average of 3.626 outputs per record, above it the scan dominates and v5 wins by up to 1.58x. Storing lengths rather than script lengths is what makes the directory free — the script is whatever remains of its payload, so no length is stored twice. See `docs/benchmarks/utxo-memory.md`.
+
+### Canonical record spelling
+The rule that one logical record has exactly one byte string. Fixed-width fields give this away for free; variable-width encodings must enforce it, and `UtxoRecord` compares and hashes by bytes, so a second spelling makes equal records unequal. v5 needs three rules to keep it: a varint must be minimal, because `[0x80, 0x00]` also decodes to zero; a directory width must be the narrowest that fits, because a wider one describes the same record; and the compact amount form and the escape must be exact complements, so the compact form may encode only amounts the escape refuses and the escape refuses only amounts the compact form covers. The last of these is also a safety rule rather than a tidiness one: `read_varint` hands `decompress_amount` whatever a record contains and `validate_encoded` runs it over every output loaded from a snapshot, and the transform multiplies by up to a billion, so an unbounded input panics a debug build and wraps silently in a release one. `decompress_accepts_exactly_the_encoder_image` states the whole rule as one property over every `u64`.
+
+### Work-count assertion
+Asserting how much of an expensive operation a code path performs, instead of how long it takes. A wall-clock assertion in a test suite is a flake generator, and an assertion that a function merely returns something passes for a stub. `find_output_decompresses_at_most_the_amount_it_returns` counts `decompress_amount` calls behind a `cfg(test)` thread-local and requires one for a hit, none for a miss and none for `max_vout`, at any record size — which is the algorithmic claim the layout rests on, stated deterministically. The counterpart is the case a count cannot make: where the claim really is about elapsed time, the assertion belongs in a paired-arm benchmark, not a test.

DEVIATIONS.md

@@ -305,3 +305,64 @@ position while keeping the rest. See the *All-or-scan position fallback* concept
 in `CONCEPTS.md`. The residual accepted: a stale offset landing exactly on a
 transaction boundary whose transaction also matches, while a different
 transaction in that block matches too.
+
+## §9 — UTXO record payload encoding, and the arena PLAN.md specified
+
+`PLAN.md` design principle 8 specifies a `bumpalo::Bump` arena per shard for
+UTXO record storage. The shipped implementation deviated earlier to one heap
+allocation per record via `ThinRecordBuf`; this section records why the arena
+is now **rejected on measurement** rather than merely deferred, and what
+replaced the record encoding instead.
+
+### The arena is rejected, not pending
+
+The arena's stated purpose was the per-record allocation overhead and the
+fragmentation expected from tens of millions of small allocations. Both were
+measured before any work started (`docs/benchmarks/utxo-memory.md`):
+
+- Allocation header plus slack is **2.2 bytes per output** on a real mainnet
+  chainstate at height 412,732 (55.1 B payload against 57.3 accounted).
+- Fragmentation is **5%** after churning twice the whole set, and the curve is
+  flattening rather than climbing. Uniform small allocations are the case a
+  size-class allocator handles well.
+
+An arena removes an overhead that measurement puts at a few percent, at the cost
+of a self-referential per-shard structure (`self_cell!` over a pinned `Bump`)
+plus the round-robin `defrag_one_shard` PLAN.md Task 5 Step 7 also specifies.
+Do not start it without new evidence; the two numbers above are the evidence
+against it.
+
+### The record payload is v5, not the v4 layout
+
+The same measurement found the UTXO set is **77.4% of process RSS**, which is
+where the encoding work went instead. Per-output metadata was a fixed 19 bytes
+(`vout(4) || value(8) || height(4) || coinbase(1) || script_len(2)`); it is now
+Core's `CTxOutCompressor` amount transform, `height` and `coinbase` packed into
+one varint, and two fixed-width directories in front of the payloads. Measured
+saving **11.75 bytes per output, 21.7% of the payload**, about 1.97 GiB at tip.
+
+Three things about this are deviations worth naming:
+
+- **A flat varint layout was built first and rejected.** It hit the size target
+  and lost 4.4-4.9x on `find_output`, the hot read. See the *Directory-layout
+  record* concept.
+- **`height` is not hoisted into the record header**, which would save three
+  bytes more. It needs "every output of a record shares one height" to hold, and
+  BIP30's duplicate coinbase txids are exactly where it might not.
+- **The snapshot disk format is unchanged.** `PLAN.md`'s successor step called
+  for a v5 file format; disk size is not a G14 budget item (the budgets are tip
+  RSS and Electrum p95), so the invariant that step really protects was covered
+  instead by a golden vector generated from a v4 build —
+  `crates/utxo/tests/snapshot_v4_golden.rs`. `hash_serialized_3` and the MuHash
+  trailer are computed over decoded consensus values, never over the in-memory
+  encoding, and that is asserted in both directions plus as a load/store fixed
+  point.
+
+### Revert criterion
+
+v5 costs about 3 ns per lookup at the measured mainnet average and 3-21% on
+block commit p95, against budgets with roughly twenty times the headroom. It
+buys 12 points of the 16 GiB tip-RSS budget. **If G14 tip RSS measures well
+under budget — say below 10 GiB — this complexity is not earning its keep and
+reverting is the right call.** v4 remains in the tree as the equivalence oracle
+and the benchmark's `before` arm, so a revert is a revert, not a rewrite.

crates/node/src/checkpoint.rs

@@ -815,6 +815,72 @@ fn checkpoint_best_tip_id(
     Ok(applied_id)
 }
 
+/// Logs and gauges what the UTXO set holds in memory, against process RSS.
+///
+/// Runs on the checkpoint path because a checkpoint already walks every record
+/// to serialize the snapshot, so a second pointer-only walk is cheap beside it,
+/// and because a checkpoint is the only moment the set is guaranteed stable.
+///
+/// The residual between `accounted` and RSS is the point: the set can only
+/// account for its own allocations, while the G14 budget is written against the
+/// process. See `docs/benchmarks/utxo-memory.md`.
+fn report_utxo_memory(utxo: &UtxoSet, height: u32) {
+    // Clippy suggests a method reference here; it does not compile, because
+    // `with_stable_view` needs a closure general over the view's lifetime.
+    #[allow(clippy::redundant_closure_for_method_calls)]
+    let report = utxo.with_stable_view(|view| view.memory_report());
+    let accounted = report.accounted_bytes();
+    let rss = crate::metrics::process_rss_bytes();
+
+    metrics::gauge!("node.utxo.records").set(metric_count(report.records));
+    metrics::gauge!("node.utxo.outputs").set(metric_count(report.outputs));
+    metrics::gauge!("node.utxo.record_payload_bytes")
+        .set(metric_count(report.record_payload_bytes));
+    metrics::gauge!("node.utxo.record_allocation_bytes")
+        .set(metric_count(report.record_allocation_bytes));
+    metrics::gauge!("node.utxo.table_bytes").set(metric_count(report.table_bytes));
+    metrics::gauge!("node.utxo.accounted_bytes").set(metric_count(accounted));
+    if let Some(rss) = rss {
+        metrics::gauge!("node.process.rss_bytes").set(metric_count_u64(rss));
+    }
+
+    tracing::info!(
+        height,
+        records = report.records,
+        outputs = report.outputs,
+        record_payload_bytes = report.record_payload_bytes,
+        record_allocation_bytes = report.record_allocation_bytes,
+        table_bytes = report.table_bytes,
+        accounted_bytes = accounted,
+        // Plain numbers, not `?rss`: Debug on an `Option` emits "Some(123)",
+        // which every downstream parser then has to strip.
+        rss_bytes = rss.unwrap_or_default(),
+        rss_known = rss.is_some(),
+        unaccounted_bytes = rss
+            .map(|rss| rss.saturating_sub(accounted.try_into().unwrap_or(u64::MAX)))
+            .unwrap_or_default(),
+        "utxo memory attribution"
+    );
+}
+
+#[expect(
+    clippy::as_conversions,
+    clippy::cast_precision_loss,
+    reason = "gauge values are f64 by the metrics crate's contract"
+)]
+fn metric_count(value: usize) -> f64 {
+    value as f64
+}
+
+#[expect(
+    clippy::as_conversions,
+    clippy::cast_precision_loss,
+    reason = "gauge values are f64 by the metrics crate's contract"
+)]
+fn metric_count_u64(value: u64) -> f64 {
+    value as f64
+}
+
 #[allow(clippy::too_many_arguments, clippy::too_many_lines)]
 fn write_checkpoint_inner(
     data_dir: &Dir,
@@ -875,6 +941,8 @@ fn write_checkpoint_inner(
     let (utxo_bytes, utxo_sha256) = utxo_writer.finish();
     sync_file(&utxo_file, failpoint, CheckpointFailpoint::UtxoSync)?;
 
+    report_utxo_memory(utxo, applied_tip.height);
+
     let listener_stats = coin_stats.snapshot();
     if listener_stats.height != applied_tip.height {
         return Err(CheckpointError::Invalid(format!(

crates/node/src/metrics.rs

@@ -170,6 +170,69 @@ impl HistogramFn for HistogramHandle {
     }
 }
 
+/// Resident set size of this process in bytes, or `None` where the platform
+/// cannot report it.
+///
+/// Used by the memory-attribution reporting on the checkpoint path. The G14
+/// budget is written against RSS, and the UTXO set can only account for its own
+/// allocations, so the residual between the two is the number that says whether
+/// an encoding change is worth making.
+#[must_use]
+pub fn process_rss_bytes() -> Option<u64> {
+    #[cfg(target_os = "linux")]
+    {
+        parse_proc_status_rss(&std::fs::read_to_string("/proc/self/status").ok()?)
+    }
+    #[cfg(not(target_os = "linux"))]
+    {
+        // No `/proc`; shell out rather than take a platform-specific dependency
+        // for one number read once per checkpoint.
+        let output = std::process::Command::new("ps")
+            .args(["-o", "rss=", "-p", &std::process::id().to_string()])
+            .output()
+            .ok()?;
+        parse_ps_rss(&String::from_utf8(output.stdout).ok()?)
+    }
+}
+
+/// Extracts `VmRSS` from `/proc/{pid}/status` content, in bytes.
+///
+/// Split out from the read so it is testable off Linux. Without this the parser
+/// is `cfg`-compiled out of every run on a macOS development host and first
+/// executes in production, on the one path whose output the memory attribution
+/// divides by.
+///
+/// The kernel reports `VmRSS:` in kibibytes with variable leading whitespace,
+/// and the field is absent for a kernel thread.
+// Unreachable off Linux, and deliberately still compiled there: the point of
+// splitting it out is that its tests run everywhere, which they cannot do if
+// the function is `cfg`-ed away with its caller.
+#[cfg_attr(
+    not(target_os = "linux"),
+    allow(dead_code, reason = "tested on every host, called only on Linux")
+)]
+fn parse_proc_status_rss(status: &str) -> Option<u64> {
+    status.lines().find_map(|line| {
+        line.strip_prefix("VmRSS:")?
+            .split_whitespace()
+            .next()?
+            .parse::<u64>()
+            .ok()?
+            .checked_mul(1024)
+    })
+}
+
+/// Extracts the kibibyte count `ps -o rss=` prints, in bytes.
+// Mirror of the note on `parse_proc_status_rss`: unreachable on Linux, still
+// compiled and still tested there.
+#[cfg_attr(
+    target_os = "linux",
+    allow(dead_code, reason = "tested on every host, called only off Linux")
+)]
+fn parse_ps_rss(output: &str) -> Option<u64> {
+    output.trim().parse::<u64>().ok()?.checked_mul(1024)
+}
+
 /// Installs in-memory process metrics and returns its handle when configured.
 ///
 /// The workspace pins `metrics-exporter-prometheus` without its HTTP listener.
@@ -228,6 +291,107 @@ mod tests {
 
     use super::*;
 
+    /// Real `/proc/{pid}/status` content, so the Linux parser is exercised on
+    /// every host rather than only wherever CI happens to run Linux.
+    ///
+    /// The field is kibibytes with variable leading whitespace, sits between
+    /// other `Vm*` keys that share its prefix shape, and is absent for a kernel
+    /// thread.
+    #[test]
+    fn proc_status_rss_is_parsed_from_the_kernel_format() {
+        const STATUS: &str = "\
+Name:\tbitcoin-rs
+Umask:\t0022
+State:\tS (sleeping)
+VmPeak:\t14680064 kB
+VmSize:\t14680064 kB
+VmLck:\t       0 kB
+VmHWM:\t 3019751 kB
+VmRSS:\t 2949952 kB
+RssAnon:\t 2900000 kB
+Threads:\t17
+";
+        assert_eq!(
+            super::parse_proc_status_rss(STATUS),
+            Some(2_949_952 * 1024),
+            "VmRSS must be read in kibibytes and returned in bytes"
+        );
+
+        // `VmHWM` and `VmSize` share the prefix shape and must not be taken.
+        assert_ne!(super::parse_proc_status_rss(STATUS), Some(3_019_751 * 1024));
+
+        // A kernel thread has no `VmRSS` at all.
+        assert_eq!(
+            super::parse_proc_status_rss("Name:\tkthreadd\nThreads:\t1\n"),
+            None
+        );
+        assert_eq!(super::parse_proc_status_rss(""), None);
+        assert_eq!(
+            super::parse_proc_status_rss("VmRSS:\tnot-a-number kB"),
+            None
+        );
+        assert_eq!(super::parse_proc_status_rss("VmRSS:\t"), None);
+    }
+
+    #[test]
+    fn ps_rss_output_is_parsed_in_kibibytes() {
+        assert_eq!(super::parse_ps_rss(" 2949952\n"), Some(2_949_952 * 1024));
+        assert_eq!(super::parse_ps_rss(""), None);
+        assert_eq!(super::parse_ps_rss("  "), None);
+        assert_eq!(super::parse_ps_rss("garbage"), None);
+        // A value large enough to overflow the kibibyte conversion.
+        assert_eq!(super::parse_ps_rss(&u64::MAX.to_string()), None);
+    }
+
+    /// The reading must track real resident memory, not merely return a number.
+    ///
+    /// Asserting only `is_some()` would pass for a stub returning a constant,
+    /// and this figure is what the memory-attribution reporting divides the
+    /// UTXO set against — a wrong denominator silently misprices every
+    /// encoding decision made from it. So the test allocates, touches every
+    /// page to make it resident rather than merely reserved, and requires the
+    /// reading to move.
+    ///
+    /// It is also the only coverage the Linux branch has: `/proc/self/status`
+    /// is `cfg`-compiled out on the development host, so this parser runs for
+    /// the first time wherever CI runs Linux.
+    #[test]
+    fn process_rss_bytes_tracks_a_real_allocation() {
+        const BALLAST_BYTES: u64 = 128 << 20;
+        const PAGE: usize = 4096;
+
+        let Some(before) = process_rss_bytes() else {
+            // A platform with neither `/proc` nor `ps` is a legitimate `None`.
+            return;
+        };
+        assert!(
+            before > (1 << 20),
+            "implausibly small RSS before allocating: {before} bytes"
+        );
+
+        let mut ballast = vec![0_u8; usize::try_from(BALLAST_BYTES).unwrap_or(0)];
+        for page in ballast.chunks_mut(PAGE) {
+            if let Some(first) = page.first_mut() {
+                *first = 1;
+            }
+        }
+
+        // `unwrap_or(0)` rather than `expect`: a `None` here fails the
+        // assertion below with the reading it produced, which says more than a
+        // panic message would.
+        let after = process_rss_bytes().unwrap_or(0);
+        assert!(
+            after >= before + (BALLAST_BYTES / 2),
+            "RSS did not track a {BALLAST_BYTES}-byte resident allocation: {before} -> {after}"
+        );
+
+        // Keep the ballast alive across the second reading.
+        assert_eq!(
+            u64::try_from(std::hint::black_box(&ballast).len()).unwrap_or(0),
+            BALLAST_BYTES
+        );
+    }
+
     #[test]
     fn install_metrics_returns_error_when_global_recorder_install_fails() {
         let bind = SocketAddr::new(IpAddr::V4(Ipv4Addr::LOCALHOST), 0);

crates/utxo/Cargo.toml

@@ -57,3 +57,7 @@ mimalloc.workspace = true
 [[bench]]
 name = "utxo_commit"
 harness = false
+
+[[bench]]
+name = "record_codec"
+harness = false