docs: reconcile the durable knowledge this campaign changed

`AGENTS.md` requires that durable project knowledge be reconciled when it
changes — the overlapping `docs/solutions/` learning and every affected term in
`CONCEPTS.md`. The read-path work did that; this branch had not.

`CONCEPTS.md` gains three terms the record layout introduced:

- *Directory-layout record* — why the lookup keys and item lengths sit in
  fixed-width arrays in front of the payloads, and where the two layouts cross
  over.
- *Canonical record spelling* — the three rules that keep one logical record to
  one byte string once the fields stop being fixed-width, and why the last of
  them is a safety rule rather than a tidiness one.
- *Work-count assertion* — asserting how much of an expensive operation a path
  performs instead of how long it takes, and the case where a count cannot
  substitute for a benchmark.

`DEVIATIONS.md` gains §9, which records two things `PLAN.md` still specifies
that this campaign settled:

- The per-shard `bumpalo` arena of design principle 8 is **rejected on
  measurement**, not deferred: allocation overhead is 2.2 B/output and
  fragmentation 5% after churning twice the whole set. Those two numbers are the
  evidence against starting it.
- The record payload is v5 rather than the v4 layout, `height` is deliberately
  not hoisted (BIP30 duplicate txids), and the snapshot disk format is
  deliberately unchanged — disk size is not a G14 budget item, so the invariant
  that step protected is covered by a golden vector instead.

Adds the best-practice learning, which is the part most likely to be useful
somewhere else: the codec benchmark measured `encode`/`decode` because that is
what the codec module exposes, while the node calls `find_output`. Reshaped
around the real call the same change measured 4.4-4.9x slower rather than 1.9x,
and the cause turned out to be that a fixed-width layout gets lazy field
skipping free from the optimizer while a variable-length one structurally
cannot. It also records the crossover that one fixture size hid in both
directions.

Finally, states the revert criterion for v5 in both the deviation and the
benchmark doc, while the numbers are in front of us: if G14 tip RSS measures
well under budget the complexity is not earning its keep, and v4 is still in the
tree as the oracle, so a revert is a revert rather than a rewrite.

Author: rabbitson87

CONCEPTS.md

@@ -269,3 +269,12 @@ A key that distinguishes which producer wrote a row, as opposed to one that mere
 
 ### Resolution-time sampling
 Recording a statistic when its outcome is known rather than when the subject arrives. The fee estimator counted a transaction against every confirmation target the moment it entered, so a fresh arrival was already a failure at every target and a burst silenced the estimator before anything had missed a deadline. It also broke the decay: the denominator had been decaying since entry while a confirmation arrived undecayed, reporting 81 successes in 100 as roughly 85%. Sampling numerator and denominator together at the moment a target resolves fixes both, because they then decay from the same block. The counterpart rule is that a subject leaving for an unrelated reason is untracked without being sampled: an eviction says something about the mempool, not about whether the transaction would have confirmed.
+
+### Directory-layout record
+A record that keeps its per-item lookup keys and item lengths in fixed-width arrays in front of the items, rather than inline with them. `UtxoRecord` v5 is `txid || output_count || legacy_inline_len || widths || vout_dir || len_dir || payloads`, where each directory entry is the narrowest little-endian width the record needs. The reason is random access: the hot read is `find_output(vout)` — every spent input resolves through `Shard::get`/`get_entry`/`get_meta` — and in a flat variable-length layout each field's length is what locates the next one, so finding output `i` walks the bytes of outputs `0..i`, scripts included. That was measured at 4.4-4.9x slower than the fixed-width v4 it replaced. With directories a lookup scans one dense byte array and sums a second, touching about two bytes per output instead of thirty-five. The two layouts cross over near 64 outputs: below it v5's fixed setup dominates and it is about 3 ns slower at the measured mainnet average of 3.626 outputs per record, above it the scan dominates and v5 wins by up to 1.58x. Storing lengths rather than script lengths is what makes the directory free — the script is whatever remains of its payload, so no length is stored twice. See `docs/benchmarks/utxo-memory.md`.
+
+### Canonical record spelling
+The rule that one logical record has exactly one byte string. Fixed-width fields give this away for free; variable-width encodings must enforce it, and `UtxoRecord` compares and hashes by bytes, so a second spelling makes equal records unequal. v5 needs three rules to keep it: a varint must be minimal, because `[0x80, 0x00]` also decodes to zero; a directory width must be the narrowest that fits, because a wider one describes the same record; and the compact amount form and the escape must be exact complements, so the compact form may encode only amounts the escape refuses and the escape refuses only amounts the compact form covers. The last of these is also a safety rule rather than a tidiness one: `read_varint` hands `decompress_amount` whatever a record contains and `validate_encoded` runs it over every output loaded from a snapshot, and the transform multiplies by up to a billion, so an unbounded input panics a debug build and wraps silently in a release one. `decompress_accepts_exactly_the_encoder_image` states the whole rule as one property over every `u64`.
+
+### Work-count assertion
+Asserting how much of an expensive operation a code path performs, instead of how long it takes. A wall-clock assertion in a test suite is a flake generator, and an assertion that a function merely returns something passes for a stub. `find_output_decompresses_at_most_the_amount_it_returns` counts `decompress_amount` calls behind a `cfg(test)` thread-local and requires one for a hit, none for a miss and none for `max_vout`, at any record size — which is the algorithmic claim the layout rests on, stated deterministically. The counterpart is the case a count cannot make: where the claim really is about elapsed time, the assertion belongs in a paired-arm benchmark, not a test.

DEVIATIONS.md

@@ -255,3 +255,64 @@ serving remains deferred.
 - **G14 empirical validation still deferred.** The `faster than Bitcoin
   Core` claim requires multi-day same-window live mainnet IBD against
   `bitcoin-rs` and `bitcoind`. Operator responsibility.
+
+## §9 — UTXO record payload encoding, and the arena PLAN.md specified
+
+`PLAN.md` design principle 8 specifies a `bumpalo::Bump` arena per shard for
+UTXO record storage. The shipped implementation deviated earlier to one heap
+allocation per record via `ThinRecordBuf`; this section records why the arena
+is now **rejected on measurement** rather than merely deferred, and what
+replaced the record encoding instead.
+
+### The arena is rejected, not pending
+
+The arena's stated purpose was the per-record allocation overhead and the
+fragmentation expected from tens of millions of small allocations. Both were
+measured before any work started (`docs/benchmarks/utxo-memory.md`):
+
+- Allocation header plus slack is **2.2 bytes per output** on a real mainnet
+  chainstate at height 412,732 (55.1 B payload against 57.3 accounted).
+- Fragmentation is **5%** after churning twice the whole set, and the curve is
+  flattening rather than climbing. Uniform small allocations are the case a
+  size-class allocator handles well.
+
+An arena removes an overhead that measurement puts at a few percent, at the cost
+of a self-referential per-shard structure (`self_cell!` over a pinned `Bump`)
+plus the round-robin `defrag_one_shard` PLAN.md Task 5 Step 7 also specifies.
+Do not start it without new evidence; the two numbers above are the evidence
+against it.
+
+### The record payload is v5, not the v4 layout
+
+The same measurement found the UTXO set is **77.4% of process RSS**, which is
+where the encoding work went instead. Per-output metadata was a fixed 19 bytes
+(`vout(4) || value(8) || height(4) || coinbase(1) || script_len(2)`); it is now
+Core's `CTxOutCompressor` amount transform, `height` and `coinbase` packed into
+one varint, and two fixed-width directories in front of the payloads. Measured
+saving **11.75 bytes per output, 21.7% of the payload**, about 1.97 GiB at tip.
+
+Three things about this are deviations worth naming:
+
+- **A flat varint layout was built first and rejected.** It hit the size target
+  and lost 4.4-4.9x on `find_output`, the hot read. See the *Directory-layout
+  record* concept.
+- **`height` is not hoisted into the record header**, which would save three
+  bytes more. It needs "every output of a record shares one height" to hold, and
+  BIP30's duplicate coinbase txids are exactly where it might not.
+- **The snapshot disk format is unchanged.** `PLAN.md`'s successor step called
+  for a v5 file format; disk size is not a G14 budget item (the budgets are tip
+  RSS and Electrum p95), so the invariant that step really protects was covered
+  instead by a golden vector generated from a v4 build —
+  `crates/utxo/tests/snapshot_v4_golden.rs`. `hash_serialized_3` and the MuHash
+  trailer are computed over decoded consensus values, never over the in-memory
+  encoding, and that is asserted in both directions plus as a load/store fixed
+  point.
+
+### Revert criterion
+
+v5 costs about 3 ns per lookup at the measured mainnet average and 3-21% on
+block commit p95, against budgets with roughly twenty times the headroom. It
+buys 12 points of the 16 GiB tip-RSS budget. **If G14 tip RSS measures well
+under budget — say below 10 GiB — this complexity is not earning its keep and
+reverting is the right call.** v4 remains in the tree as the equivalence oracle
+and the benchmark's `before` arm, so a revert is a revert, not a rewrite.

docs/benchmarks/utxo-memory.md

@@ -129,6 +129,28 @@ worth having, but it does not by itself settle the gate.
 **Step 2.2 is justified and done. Step 2.4 is not**: fragmentation measured 5%
 after churning twice the whole set.
 
+### When to revert v5
+
+Stated now, while the numbers are in front of us, so that whoever reads this
+later does not have to reconstruct the trade.
+
+v5 costs about **3 ns per lookup** at the measured mainnet average and **3-21%
+on block commit p95**, against budgets with roughly twenty times the headroom.
+It buys **12 points of the 16 GiB tip-RSS budget**. That is a good trade only
+while tip RSS is actually near the budget — and the budget has never been
+measured. The 13.83 GiB figure everything here is projected from comes from
+*excluded* evidence at height 645,804, on a run that never made the tip.
+
+**If G14 tip RSS measures well under budget — say below 10 GiB — this
+complexity is not earning its keep and reverting is the right call.** v4 is
+retained in the tree as the equivalence oracle and the benchmark's `before`
+arm, so a revert is a revert rather than a rewrite.
+
+The second thing that would change the answer is outputs per record. The
+projection holds it at 3.626; it has not converged (2.296 at height 183k, 4.056
+at 390k), and it is the number the result is most sensitive to, because the
+32-byte txid amortizes over it directly.
+
 The projection holds outputs per record at 3.626, which has not converged and
 remains the number the result is most sensitive to.
 

docs/solutions/best-practices/benchmark-the-operation-the-workload-performs-not-the-one-the-api-exposes.md

@@ -0,0 +1,115 @@
+---
+title: Benchmark the operation the workload performs, not the one the API exposes — a codec that won on encode/decode lost 4.9x on the call the node actually makes
+date: 2026-08-17
+category: docs/solutions/best-practices
+module: performance measurement / UTXO record codec (crates/utxo)
+problem_type: best_practice
+component: tooling
+severity: high
+applies_when:
+  - "Benchmarking a data-structure or codec change behind an accessor API"
+  - "Choosing between a fixed-width and a variable-length in-memory layout"
+  - "Quoting a speedup ratio measured on one fixture size"
+related_components:
+  - development_workflow
+  - testing_framework
+tags:
+  - benchmark
+  - paired-arm
+  - codec
+  - measurement-discipline
+---
+
+## What happened
+
+The UTXO record payload was re-encoded to save memory. The refactor set required
+a paired benchmark, and it had one: `encode` and `decode` of a whole record, v4
+against v5, both arms in one Criterion group over one fixture. v5 was slower on
+both, by 1.9-3.2x, and two rounds of optimization went into closing that gap.
+
+Both the benchmark and the optimization were aimed at the wrong thing.
+
+The operation the node performs is **`find_output(vout)`** — every spent input
+resolves one output by index through `Shard::get`, `get_entry` or `get_meta`,
+and all three land there. Whole-record decode is the snapshot and rescan path,
+which is rare by comparison. Reshaped around `find_output`, the same v5 codec
+measured **4.4-4.9x slower**, not 1.9x. The change was far worse than the
+harness had been reporting, and no amount of tuning the encode path would have
+found it.
+
+## Why the wrong benchmark looked reasonable
+
+`encode` and `decode` are what the codec module exposes. They are the natural
+unit to benchmark if you are looking at the codec. `find_output` lives one layer
+up, in the record type, and reads like a convenience wrapper over the decoder —
+which is exactly what it was, and exactly why it was slow: it decoded every
+output it rejected.
+
+The tell was available and was not read: the codec is `pub(crate)`, and the only
+callers outside its own tests were three `Shard` methods, all doing the same
+by-index lookup. **The call sites were the benchmark specification.**
+
+## The second-order finding
+
+Once `find_output` was benchmarked, the cause was not what it appeared:
+
+> Every v4 field sits at a constant offset, so when only `vout` is read the
+> optimizer deletes the loads for the rest. v4 got lazy field skipping for free
+> from LLVM, without anyone designing it. A variable-length layout cannot be
+> given the same treatment, because each field's length is what locates the
+> next, so the reads are a serial dependency chain no optimizer can remove.
+
+The fixed-width arm was not merely faster — it was benefiting from an
+optimization the variable-length arm is structurally unable to receive. That is
+a property of the layout choice, not of the implementation, and it is invisible
+in a benchmark that consumes every field.
+
+The fix was a layout change (fixed-width directories in front of the payloads),
+not a tuning pass.
+
+## One fixture size hides a crossover in both directions
+
+The corrected harness ran 1, 4 and 16 outputs per record. The real-workload
+benchmark, `utxo_commit`, used a fixture with **256**. They disagreed, and both
+were quoted as if general:
+
+| outputs | `find_output/miss` v4 | v5 | ratio |
+|---:|---:|---:|---:|
+| 1 | 2.6 ns | 7.6 ns | 0.35x |
+| 3.626 (measured mainnet average) | 3.9 ns | 6.9 ns | 0.57x |
+| 16 | 16.6 ns | 20.7 ns | 0.80x |
+| 64 | 108.4 ns | 78.7 ns | **1.38x** |
+| 256 | 462.1 ns | 294.1 ns | **1.57x** |
+
+Benchmarking only small sizes hid the crossover one way; quoting only the large
+fixture hid it the other. The claim shipped as "makes lookups faster than v4",
+which was **true of the fixture and false of the workload** — the measured
+mainnet average is 3.626 outputs per record, where v5 is slower.
+
+## Rules
+
+1. **Enumerate the call sites before writing the harness.** For a `pub(crate)`
+   API that is a grep, and it is the specification. Benchmark what calls it, not
+   what it exports.
+2. **Bracket the workload's real parameter, and put the workload's own value in
+   the table.** One fixture size cannot show a crossover, and a crossover is the
+   normal outcome when a change trades fixed cost for per-item cost.
+3. **Suspect the arm that looks impossibly fast.** v4's best case measured 2.1 ns
+   and did not move with record size; that was the optimizer eliminating work,
+   which is real but tells you the arms are not doing comparable work.
+4. **When two harnesses disagree, publish the conservative one and say why.**
+   `utxo_commit` reported 2.35x and the microbenchmark 1.58x on the same shape,
+   because the microbenchmark reimplements the `before` arm as a direct loop the
+   optimizer handles better — so its v4 arm is *faster than the shipped v4*. The
+   smaller number is the one that survives scrutiny.
+
+## Related
+
+- `docs/solutions/best-practices/small-window-benchmarks-do-not-predict-at-scale-throughput.md`
+  — the same failure in the size dimension rather than the operation dimension.
+- `docs/solutions/best-practices/criterion-bench-trust-rebuild-drift-baselines-allocator.md`
+  — why both arms belong in one group in one run.
+- `docs/benchmarks/utxo-memory.md` — the campaign this came from, including the
+  correction notice.
+- The *Directory-layout record* and *Work-count assertion* entries in
+  `CONCEPTS.md`.