Attribute every measured product hot path exhaustively

[metaphorics]

Question

Across all three #45 product workloads and all 36 required cells, what is the overlap-aware whole-workload wall-time contribution of every measured hot path and cost center, and what is the evidence-backed disposition of every repeatable contribution above the predeclared noise floor?

Scope

This ticket covers every measurable product hot path in all 36 cells, including cells that already exceed 2.0x. SIMD, allocators, and mechanisms named in advance are examples, not limits. Measure and reconcile:

• exclusive and inclusive CPU cost;
• allocation count, bytes, lifetime, fragmentation, and retained memory;
• storage reads, writes, compaction, flush, durability, and page-cache behavior;
• I/O wait, syscalls, copying, parsing, serialization, and network transfer;
• lock contention, atomics, queueing, scheduler delay, backpressure, and idle gaps;
• branch, instruction-cache, data-cache, TLB, vectorization, and memory-bandwidth behavior;
• validation, hashing, Merkle, MuHash, script, UTXO, reorg-readiness, and body-persistence work;
• backend-specific and architecture-specific costs.

Do not stop after finding the first hotspot. Continue until the overlap-aware measured components reconcile with wall time and the unclassified residual is within the predeclared run-to-run noise floor. When profiler evidence is ambiguous, disable or neutralize one stage and measure the whole-workload delta. Treat a mechanism as a campaign candidate only when the attributed cost is large enough to affect a product-cell result.

Deliverables

• A coverage ledger for every measured product hot path across every applicable workload and cell. Each row records applicability, raw custody, baseline wall contribution, disable or neutralize delta on total product wall time, overlap or concurrency with other stages, affected cells, and disposition.
• A cross-cell ledger that separates shared hot paths from backend-, corpus-, domain-, and architecture-specific paths.
• One ordered candidate list based on whole-product impact, dependency order, reversibility, and correctness risk.
• Explicit dispositions for every repeatable above-noise product bottleneck: optimize with whole-workload remeasurement, already bounded, blocked with evidence, or rejected with evidence.

Acceptance

• Freeze the attribution method, trial count, and per-workload and per-host noise rule before inspecting attribution results. Do not choose a materiality threshold after seeing the data.
• Every one of the 36 cells has an attributed wall-time account whose explicit residual is no larger than its predeclared run-to-run noise floor.
• Every repeatable above-noise CPU, allocation, I/O, synchronization, scheduler, cache, and retained-memory cost has a disposition. No above-noise cost may be hidden in other, omitted because another cost is larger, or dismissed because its cell already exceeds 2.0x.
• Every optimization links to correctness evidence and fresh whole-workload remeasurement of every affected cell. No microbenchmark or isolated-stage result substitutes for a product-domain measurement.
• One dominant cost, one lever per failing cell, or coverage of only below-target cells is insufficient. Attribution completeness does not alter the fixed 36-cell speed denominator.
• No candidate relies on weaker consensus, validation, durability, recovery, body availability, cache posture, or index posture.
• This ticket closes only when the ledger has no missing applicable path, no unexplained above-noise residual, and no unresolved repeatable above-noise product bottleneck.

Blocked by

• Capture fresh 36-cell baseline matrix

[metaphorics]

Static hot-path inventory complete; timing attribution remains open

This pass mapped the current production paths for all three frozen domains. It does not assign wall time and does not satisfy this ticket's 36-cell residual gate. The map is the prerequisite for placing exact probes before fresh baselines.

Fixed applicability

Every row will name applicable coordinates from the fixed product denominator only:

• domains: offline full-validation chainstate construction, controlled loopback P2P initial sync, full UTXO-set MuHash RPC query;
• corpora: C150 and the empirically selected Cmodern;
• native architectures: x86-64 and ARM64;
• backends: fjall, RocksDB, and redb.

Allocator variants, scalar/AVX2 variants, source forms, test networks, pruning experiments, microbenchmarks, and mdbx remain diagnostics. They are not additional cells and cannot replace any of the 36 cells. For the MuHash query, the backend coordinate identifies how the identical committed state was constructed, checkpointed, reopened, and served; the live scan itself traverses the in-memory UtxoSet.

Deduplicated source-path skeleton

Shared apply path

1. header acceptance and contextual softfork/MTP lookup;
2. block wire decode and transaction parse/plan construction;
3. window prevout overlay and script-check preparation;
4. structural, Merkle, and witness validation;
5. script execution in the dedicated verification pool;
6. contextual consensus checks;
7. UTXO change construction and sharded commit;
8. undo encoding and persistence;
9. body serialization/append/index persistence;
10. the frozen optional index posture;
11. block-tree and applied-tip publication;
12. clean checkpoint, shutdown drain, reopen, recovery, and body/undo/index readiness.

Primary seams are crates/node/src/apply.rs, crates/node/src/window_overlay.rs, crates/consensus/src/verify_tx.rs, crates/consensus/src/sha256d64.rs, crates/utxo/src/set.rs, crates/utxo/src/undo_codec.rs, crates/storage/src/block_file.rs, crates/node/src/checkpoint.rs, crates/node/src/state.rs, and the fjall/RocksDB/redb implementations.

Offline-only path

• Core-frame read, buffering, page-cache behavior, frame/hash custody, consensus decode, replay window formation, production apply handoff, final checkpoint, durable exit, and untimed reopen validation in crates/node/examples/mainnet_prefix_replay.rs and the corpus reader/manifest path.

P2P-only path

• socket read/write, Bitcoin wire header/checksum/decode, per-message allocation/copy;
• inbound/outbound channels and wake coalescing;
• header acceptance;
• peer selection, getheaders/getdata construction, queue-to-writer delay;
• pending/download-window state, out-of-order staging, contiguous-prefix drain;
• staller, timeout, hedge, retry, cooldown, and disconnect behavior;
• event-loop wake/tick delay, scheduler idle, backpressure, apply handoff, shutdown drain, and reopen.

Primary seams are crates/p2p/src/listener.rs, crates/p2p/src/wire.rs, crates/p2p/src/peer.rs, crates/node/src/event_loop.rs, crates/node/src/sync.rs, crates/node/src/sync/window.rs, and crates/node/src/sync/stage.rs.

MuHash-RPC-only path

• TCP accept/auth, HTTP body read, UTF-8/JSON parse, method dispatch;
• stable-view lock acquisition and hold;
• 256-shard record traversal and zero-copy output decode;
• CoinStats preimage formatting;
• SHA-256 plus ChaCha20 element expansion;
• 3072-bit reduction, multiplication, combine, inversion, and finalization;
• response construction, JSON serialization, and socket write.

Primary seams are crates/rpc/src/server.rs, crates/rpc/src/handlers/chain.rs, crates/utxo/src/set.rs, crates/utxo/src/record.rs, crates/coinstats/src/stats.rs, and crates/coinstats/src/muhash3072.rs.

Cross-cutting classes

• allocation count/bytes/lifetime/fragmentation and retained memory;
• lock wait/hold, atomics, queue age/depth, backpressure, runnable/off-CPU/idle time;
• syscalls, copies, file/network bytes, I/O wait, page-cache behavior;
• backend WAL/durability/flush/compaction/cache behavior;
• branch, instruction-cache, data-cache, TLB, vectorization, and memory-bandwidth behavior;
• corpus-, backend-, domain-, and native-architecture-conditioned costs.

Unsafe attribution shortcuts rejected

The ledger will not call these product-safe: bypassing scripts through assume-valid, skipping Merkle/witness checks, using a dummy UTXO mutator, suppressing undo/body writes, changing durability, removing stable-view locking, replacing UTXO decoding or MuHash arithmetic with constants, or suppressing checkpoint/recovery work. Those mutations weaken the frozen product contract and cannot supply a comparator result.

Where a source-level cost needs a disabling experiment, the experiment must preserve output and the full product posture. Otherwise the row is blocked pending safe probe. Any mutation trial still requires unchanged full-validation, durable checkpoint, reopen, body/undo/index, state-commitment, and reorg-readiness evidence.

Probe prerequisites before baseline attribution

1. Freeze interval IDs, exact trial count, and host/workload noise rules before inspecting results.
2. Capture whole-run wall time, process and per-thread CPU, runnable/off-CPU time, and explicit idle intervals.
3. Capture allocation count/bytes/lifetime/fragmentation, retained-memory snapshots, and peak RSS.
4. Capture syscall counts, read/write/network bytes, copying/parsing, I/O wait, page faults, and page-cache state.
5. Add backend operation latency and WAL/durability/flush/compaction/cache evidence for fjall, RocksDB, and redb.
6. Add lock wait/hold, queue age/depth, atomic-contention, and backpressure evidence.
7. Capture branch, instruction/data-cache, TLB, vectorization, and memory-bandwidth counters on both native architectures.
8. Add exclusive leaf timing for validation, Merkle/witness, script, UTXO, undo, body, checkpoint/reopen, and MuHash phases.
9. Give every observation a parent interval and concurrency group. Never sum nested child wall time into its parent or sum parallel worker wall time as elapsed time.
10. Run unchanged correctness and durability gates for every affected cell.
11. Reconcile the overlap-aware account to wall time and retain an explicit unclassified residual; never hide it as other.

The current stage histograms are useful but nested. apply_window encloses prove_window and per-block commits; script totals enclose serial/parallel variants; event-loop ticks enclose scheduling, apply, and synchronous waits. They cannot be added directly. Fresh 36-cell baselines and measured attribution remain blocked by the comparator/corpus/controller prerequisites.

[rabbitson87]

A partial contribution to the coverage ledger, for the UTXO-set memory cost center.

Measured on a pruned mainnet sync at height 412,732 (38,145,360 outputs, 10,519,335 records), taken from the checkpoint path so sync has stopped and subsystems have drained:

Layer	Bytes/output	Total
Record payload	55.1	1.96 GiB
...plus allocation header and slack	57.3
...plus hash-table backing store	61.2	2.18 GiB
Process RSS	79.2	2.81 GiB

The UTXO set is 77.4% of process RSS at a quiet moment. The 0.64 GiB residual is not yet attributed.

Two method notes that may save someone else the same detours:

1. RSS sampled during IBD cannot be used for this. It swung between 1.1 and 3.2 GB on a run whose UTXO set only grows monotonically. Research: P2P staging-window and database-cache optimization headroom #51 explains why: staging retains both the decoded bitcoin::Block and the original bytes::Bytes. An earlier pass of this attribution used an in-flight sample and concluded the UTXO set was ~44% of RSS; the quiet measurement says 77.4%.
2. dbcache_mb is not a component you can subtract. It parses and defaults to 450 MiB but never reaches a backend constructor, exactly as Research: P2P staging-window and database-cache optimization headroom #51 reports — crates/storage contains no reference to it and RocksDB uses a fixed 256 MiB block cache. I had attributed 450 MB of the residual to it before checking.

The dominant single item inside the payload is the 32-byte txid, stored once per record. It amortizes over live outputs per record, which is the number the whole result is most sensitive to and which is not converged: 2.296 at height 183k, 3.427 at 302k, 4.056 at 390k (the 2015 UTXO-spam era), 3.626 at 412k. An earlier revision assumed 1.5 and overstated per-output cost by ~46%.

Harness: crates/utxo/examples/utxo_memory_attribution.rs plus UtxoSetView::memory_report(), validated against the real chainstate to within 1.3%. Full method in docs/benchmarks/utxo-memory.md (in progress).

[rabbitson87]

Cross-reference from the read-path / UTXO-memory campaign (#80, #83, #84). It produced attribution in roughly the disposition shape this ticket asks for, over a narrower scope: the Electrum/index read path and the UTXO set, not all 36 cells.

Attributed, with a whole-workload measurement rather than a profiler reading. On a real pruned mainnet chainstate at height 412,732 (10,519,335 records, 38,145,360 outputs), sampled from the checkpoint path so the subsystems have drained:

Layer	Bytes/output	Share
Record payload	55.1
...plus allocation header and slack	57.3
...plus hash-table backing store	61.2
Process RSS	79.2	UTXO set = 77.4%

The remaining 0.64 GiB is fjall, CoinStats, the block-record log and the runtime, and is still unattributed — partly because dbcache_mb reaches no backend (#51), so there is no knob to turn and re-measure.

Two dispositions are "rejected with evidence", which this ticket explicitly wants recorded:

• Allocator fragmentation across tens of millions of small records — the leading hypothesis before measurement. Tested by spending the oldest tenth of the set and refilling, repeatedly, at constant live count: 5% more RSS after churning twice the whole set, with the curve flattening rather than climbing. Refuted at this scale on this allocator. The bumpalo arena PLAN.md specifies is therefore rejected rather than deferred (DEVIATIONS.md §9).
• A decoded-block cache in front of NodeBlockSource — built, measured, reverted. The hit case was 1.027x slower than decoding from scratch, because BlockSource::block_at_height returns an owned Block, so a hit still deep-clones 2,200 transactions. Caching a value the API forces you to copy saves nothing.

One methodological result that may be worth folding into this ticket's rules. The codec benchmark first measured encode/decode because that is what the module exposes, while the node calls find_output. Reshaped around the real call, the same change measured 4.4–4.9x slower rather than 1.9x — and the cause was that a fixed-width layout gets lazy field skipping free from the optimizer while a variable-length one structurally cannot. Written up as docs/solutions/best-practices/benchmark-the-operation-the-workload-performs-not-the-one-the-api-exposes.md.

#84 is a survey of what was never looked at — the crates with no benchmarks at all (rpc, mempool, p2p, chain, filters) — with six candidates, their evidence, and explicitly which carry measured per-unit costs versus which are structural findings whose magnitude is unknown. It also records negative results so they are not re-derived.