Research: P2P staging-window and database-cache optimization headroom

[gosunuts]

Problem

The local-file processing path is already competitive with Bitcoin Core, but the loopback P2P path still trails Core by approximately 1.37x in the historical matched 0→150k measurement. The current implementation also exposes storage-cache configuration that does not appear to reach the selected backend.

This issue tracks two concrete sources of remaining headroom:

1. P2P download/staging depth and memory representation.
2. Effective database-cache configuration and measured storage behavior.

This is a scoped investigation under #33 and #39. The controlled P2P fixture in #35 should be used for acceptance measurements.

Current observations

Fixed 128-block P2P window

crates/node/src/sync.rs currently couples:

• PENDING_BUDGET = 128
• RECEIVED_BLOCK_BUDGET = 128
• a 2 MiB estimated block size
• an approximately 256 MiB pending/staging byte budget
• 16 in-flight blocks per peer while fan-out is active

The count bound is reasonable for modern 1–2 MiB blocks, but it binds far before the byte budget for small historical blocks. This limits both request depth and the script-verification windows formed by P2P sync. The README already identifies the 128-block staging depth as the reason peer sync does not reach local-replay throughput.

A constant-only increase is not acceptable. The count is coupled to staller arming, pending/staged parity, inline capacities, channel drain sizing, peer fan-out, and memory assertions.

Staging retains decoded and serialized bodies

crates/node/src/sync/stage.rs retains both:

• a decoded bitcoin::Block
• the original bytes::Bytes P2P payload

The byte budget accounts for the serialized payload, while the decoded block adds substantial unaccounted memory. A fully out-of-order window can therefore retain roughly twice the nominal staged bytes.

dbcache_mb is not plumbed into backend construction

Config::dbcache_mb is parsed and defaults to 450 MiB, but NodeStorage::open calls backend constructors without passing it.

Current backend behavior appears to be:

• Fjall: builder defaults
• RocksDB: fixed 256 MiB block cache
• redb/MDBX: backend defaults

The configured cache target therefore does not currently define a process-level database cache posture. This also makes cache-parity claims difficult to interpret.

Investigation plan

1. Add attribution before tuning

Measure at least:

• time apply is idle waiting for blocks
• time download/staging is blocked by apply
• per-peer block RTT and bytes/sec EWMA
• inbound queue wait and block decode time
• pending/staged block and byte high-water marks
• actual retained memory for decoded plus serialized bodies
• retry count, duplicate delivery, staller disconnects, and pending timeouts
• backend write, flush, compaction, and cache behavior

Do not select a mechanism until the controlled P2P baseline attributes the gap.

2. Evaluate a dynamic count-and-byte window

Evaluate a larger count ceiling while preserving the existing byte ceiling:

allowed blocks = min(count ceiling, byte headroom / observed block-size EWMA)

The intended behavior is:

• small historical blocks: hundreds or thousands of blocks may be in flight/staged
• modern large blocks: the byte ceiling keeps the effective depth near the current range
• one oversized valid block remains processable
• no eviction/re-download churn is introduced
• staller detection continues to blame only a peer-caused blocked frontier

All count-coupled invariants and tests must be made explicit rather than mechanically changing 128.

3. Evaluate raw-first staging

Evaluate retaining only the checksum-validated wire payload plus the minimum identity metadata needed by the scheduler, deferring full bitcoin::Block decode until the block enters an apply window.

This candidate should be judged primarily as a way to:

• make the byte budget reflect actual retained block data more closely
• permit a deeper small-block window under the same RSS limit
• remove full decode work from peer reader threads

It must not add another parse or serialization pass to the apply path.

4. Evaluate measured peer-quality scheduling

Only if the baseline shows peer heterogeneity is material, evaluate request allocation using recent per-peer throughput, completion latency, and frontier-blocking history.

Any policy must retain peer diversity and bounded exploration rather than assigning all work permanently to the currently fastest peer. Existing staller disconnect, cooldown, single-peer fallback, and no-blame backpressure behavior must remain intact.

5. Make dbcache_mb effective

Introduce backend constructors that accept a cache budget and define how one process-level budget is divided among chainstate and optional indexes.

Requirements:

• Fjall cache capacity is configured explicitly.
• RocksDB no longer ignores the configured value in favor of an unconditional 256 MiB cache.
• Enabling txindex and blockfilterindex does not multiply the total configured budget once per database.
• Effective capacities are observable in logs or metrics.
• Zero, minimum, overflow, and unsupported-backend behavior is defined.

Measure cold start, warm replay, modern-corpus apply, and indexed workloads. The RAM-resident UTXO path means an early-chain replay alone is insufficient evidence for a cache change.

6. Keep lower-ceiling storage work secondary

Window-level block-file/index batching and incremental checkpoints may be investigated only after attribution shows they are material.

Historical 0→150k decomposition bounded block-body persistence at approximately 4.66 seconds total. Do not weaken body availability, reorg readiness, pruning ordering, or crash durability to remove that bounded cost.

Acceptance criteria

• Baselines use the controlled comparator from Build controlled loopback P2P comparator #35 and the measurement rules from Attribute every measured product hot path exhaustively #39/Capture fresh 36-cell baseline matrix #43.
• Wall time, CPU time, peak RSS, actual staged memory, retry/churn counters, and correctness commitments are captured together.
• Candidate and control runs are interleaved; isolated microbenchmarks are diagnostic only.
• A dynamic-window candidate preserves all staller, backpressure, single-peer fallback, and memory-bound invariants.
• A raw-first candidate demonstrates that no additional full parse or serialization pass was introduced.
• dbcache_mb produces observable backend capacities without multiplying the process budget across optional indexes.
• Reopen, interrupted-write, branch-switch, pruning, and checkpoint tests preserve the existing durability and body-availability contract.
• Every candidate receives an evidence-backed disposition: land, reject, or defer.
• No result is generalized to public-WAN full-tip IBD without a separate live high-height validation.

Non-goals

• Weakening consensus or script verification.
• Trust shortcuts or broader assume-valid use.
• Increasing constants without updating their coupled safety invariants.
• Moving synchronous storage I/O onto the event loop.
• Claiming public-network IBD leadership from a loopback result.

[coderabbitai]

🔗 Related PRs

#7 - Parallelize block input script checks for performance improvement [merged]
#14 - feat: header timestamp consensus rules, index reorg rollback, mempool policy, fuzz targets [merged]
#29 - chore(perf): calibrate replay allocator parity [merged]
#30 - perf(consensus): batch Merkle hashing with AVX2 [merged]
#49 - rpc: Core-compatible REST gateway so a remote bip300301_enforcer can use bitcoin-rs as its mainchain node [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Instrument first, tune never-before-measured. Phase 1 adds the missing attribution (apply-idle, per-peer throughput, storage stats, retained-memory high-water marks); Phase 2 produces the controlled interleaved baseline that gates every later phase.
• Fix the dead dbcache_mb wire (Phase 3) independently of the sync verdict, sharing or partitioning one budget so optional indexes do not multiply it.
• Evaluate the dynamic count-and-byte window (Phase 4) and raw-first staging (Phase 5) as measured candidates, making every count-coupled invariant explicit and proving no extra parse pass, never as blind constant bumps.
• Gate peer-quality scheduling and lower-ceiling storage work (Phase 6) on measured need, and give every candidate an evidence-backed disposition: land, reject, or defer.

Design Choices

Design Choice 1: Division of one process-level dbcache_mb budget across chainstate and optional indexes

Options Considered:

1. Share a single backend-native cache object (for example one RocksDB Cache) across all instances of the same backend, so all column families draw from one pool.
2. Partition the budget by fixed fractions per subsystem (for example chainstate gets the majority, each enabled index gets a smaller slice), passing a computed per-instance cache size into each open().

Chosen Option: 1

Rationale: Use Option 1 where the backend supports a shared cache handle (RocksDB). Use Option 2 as the portable fallback for backends without a shareable cache. Sharing a native cache is the most honest interpretation of "one process-level budget" and avoids per-index multiplication directly. Partitioning is the only tool available for backends that cannot share a cache object.

Design Choice 2: Whether raw-first staging replaces the decoded bitcoin::Block in BlockStager, or keeps it as an optional cached field

Options Considered:

1. Store only the checksum-validated wire payload plus minimal identity metadata. Decode lazily when a block enters an apply window. Hand the decoded block plus the same Bytes to apply_block_with_serialized so the apply path parses exactly once.
2. Keep the decoded block but drop it under memory pressure and re-decode on demand.

Chosen Option: 1

Rationale: Option 1 makes the byte budget reflect actual retained data, removes decode from peer reader threads, and keeps a single parse on the apply path. Option 2 keeps the double-memory footprint in the common case and defeats the purpose.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Attribution instrumentation

Nobody tunes anything until the baseline shows where the time goes. Right now the code cannot see apply-idle time or per-peer throughput, so any "optimization" is a guess. This phase adds the missing metrics and nothing else. It changes no scheduling or storage behavior.

Task 1: Add sync-path timing and high-water attribution

Instrument where the download/apply pipeline actually spends and wastes time.

• Extend crates/node/src/sync.rs and crates/node/src/sync/window.rs to record: apply-idle time (the gap where apply has no ready block and waits on download), download-blocked-by-apply time (the no-blame apply_side_busy condition duration), inbound-queue wait, and block decode time.
• Add high-water-mark gauges for pending blocks, staged blocks, pending bytes, and staged bytes. The test-only peak_staged_bytes idea already exists; promote the concept to a production gauge.
• Add a duplicate-delivery counter. credit_duplicate_delivery already runs silently; make it emit. Reuse the existing retry_count, staller_disconnects, and pending_timeout_disconnects counters; do not duplicate them.
• Follow the existing metrics::counter!/gauge!/histogram! call-site pattern. Do not invent a new metrics abstraction.

Task 2: Add per-peer throughput and latency instrumentation

The scheduler has no idea which peer is fast. Measure it before deciding whether peer heterogeneity even matters.

• Add per-SocketAddr state in DownloadWindow (parallel to peer_inflight and recent_stallers) that records per-block completion latency (from PendingBlock.requested_at to delivery) and a bytes/sec EWMA.
• Do not thread new fields into PeerInfo; it is handshake-only and gets rebuilt each connection. Keep the measurement in the window, keyed by address.
• Emit per-peer gauges/histograms for block RTT and bytes/sec EWMA. This data is diagnostic in this phase. It does not feed selection yet.

Task 3: Add storage-backend instrumentation

Storage has zero metrics today. A cache change cannot be judged blind.

• Add write, flush, and cache-behavior instrumentation to the backends in crates/storage/, starting with rocksdb_impl.rs (attach a RocksDB Statistics collector and expose block-cache hit/miss and compaction counters) and fjall_impl.rs.
• Time KvStore::flush() at its call sites or inside each impl. Keep the metric names backend-neutral where possible.
• If the bitcoin-rs-storage crate cannot depend on the metrics crate cleanly, expose counters through a small stats accessor on each store and record them from crates/node. Prefer the least-invasive option that keeps the trait layer clean.

Task 4: Surface attribution into the measurement artifact

Metrics that never reach the benchmark JSON are useless for A/B work.

• Extend the artifact emitted by crates/node/examples/mainnet_prefix_replay.rs (and the controlled-fixture harness from #35) to snapshot the new metrics alongside the existing per-stage apply timers, peak RSS, and elapsed time.
• Record actual retained staged memory (decoded plus serialized) as a distinct field, so the "roughly double the nominal staged bytes" claim becomes a measured number, not an assertion.

🤖 Prompt for AI agents

Implement Phase 1: add attribution instrumentation to the sync pipeline and
storage layer. Do not change any scheduling or storage behavior in this phase;
only add measurement.

1. Sync-path timing and high-water attribution:
- In `crates/node/src/sync.rs` and `crates/node/src/sync/window.rs`, record
apply-idle time, download-blocked-by-apply time (the `apply_side_busy` no-blame
condition duration), inbound-queue wait time, and block decode time.
- Add high-water-mark gauges for pending blocks, staged blocks, pending bytes,
and staged bytes. Promote the existing test-only `peak_staged_bytes` concept to
a production gauge.
- Make `credit_duplicate_delivery` emit a duplicate-delivery counter. Reuse
`retry_count`, `staller_disconnects`, and `pending_timeout_disconnects`; do not
duplicate these counters.
- Follow the existing `metrics::counter!/gauge!/histogram!` call-site pattern;
do not introduce a new metrics abstraction.

2. Per-peer throughput and latency instrumentation:
- Add per-`SocketAddr` state in `DownloadWindow`, parallel to `peer_inflight`
and `recent_stallers`, recording per-block completion latency (from
`PendingBlock.requested_at` to delivery) and a bytes/sec EWMA.
- Do not add new fields to `PeerInfo` (it is handshake-only and rebuilt each
connection); keep this state in the window, keyed by address.
- Emit per-peer gauges/histograms for block RTT and bytes/sec EWMA. Treat this
data as diagnostic only in this phase; it must not feed peer selection yet.

3. Storage-backend instrumentation:
- Add write, flush, and cache-behavior instrumentation to backends in
`crates/storage/`, starting with `rocksdb_impl.rs` (attach a RocksDB
`Statistics` collector, expose block-cache hit/miss and compaction counters) and
`fjall_impl.rs`.
- Time `KvStore::flush()` at its call sites or inside each impl. Use
backend-neutral metric names where possible.
- If `bitcoin-rs-storage` cannot cleanly depend on the `metrics` crate, expose
counters via a small stats accessor on each store and record them from
`crates/node`. Choose the least invasive option that keeps the trait layer
clean.

4. Surface attribution into the measurement artifact:
- Extend the artifact emitted by `crates/node/examples/mainnet_prefix_replay.rs`
and the controlled-fixture harness from `#35` to snapshot all new metrics
alongside existing per-stage apply timers, peak RSS, and elapsed time.
- Add a distinct field for actual retained staged memory (decoded plus
serialized) so the "roughly double" claim becomes measured, not asserted.

Phase 2: Controlled baseline

Establish the number before touching any mechanism. The prior campaign proved that peer-death-contaminated runs invalidate the whole candidate ranking. Do not repeat that mistake.

Produce an interleaved controlled baseline with attribution

Attribute the 1.37x loopback gap before proposing a fix.

• Use the controlled P2P fixture from #35 and the measurement rules from #39/Capture fresh 36-cell baseline matrix #43: binary-identity verification, fresh datadir, stop-hash byte-identity check at a fixed height, interleaved candidate/control runs, wall and CPU measured together.
• Capture the Phase 1 attribution outputs for the baseline run. Identify where the gap lives: apply-idle-on-network, decode on reader threads, staging-depth-limited verification windows, or storage.
• Record the standing comparator numbers as frozen values. Do not re-derive them.
• Output: an attribution report that names the dominant cost and gates which candidates in Phases 3–6 are worth running. Any expected gain below the noise floor (~15s per prior campaign) is rejected at planning time.

🤖 Prompt for AI agents

Implement Phase 2: produce a controlled interleaved baseline with attribution
before any mechanism changes.

- Use the controlled P2P fixture from `#35` together with the measurement rules
from `#39/`#43: verify binary identity, use a fresh datadir, verify stop-hash byte
identity at a fixed height, interleave candidate and control runs, and measure
wall time and CPU time together.
- Capture the Phase 1 attribution outputs for this baseline run. Identify where
the gap lives: apply-idle-on-network, decode on reader threads,
staging-depth-limited verification windows, or storage.
- Record the standing comparator numbers as frozen values; do not re-derive
them.
- Produce an attribution report that names the dominant cost. Use this report to
gate which candidates in Phases 3–6 are worth running. Reject at planning time
any expected gain below the noise floor (~15s per the prior campaign).

Phase 3: Make dbcache_mb effective

This is a dead config wire. The value is parsed, merged from three sources, printed in Debug, and then thrown away. RocksDB ignores it for a hardcoded 256 MiB. That is embarrassing and independently fixable, so do it regardless of the sync-window verdict.

Task 1: Add cache-budget-aware backend constructors

Let the backends accept the budget they were always supposed to honor.

• Add cache-size-aware constructors to the backends in crates/storage/ (for example open_with_cache(path, cache_bytes) or an options struct), keeping the existing open(path) as a default-backed wrapper.
• RocksDB: replace the hardcoded BLOCK_CACHE_SIZE with the passed budget. Fjall: configure cache capacity explicitly. redb/MDBX: apply the budget where the library exposes a cache knob; where it does not, document that the value is a no-op for that backend.
• Define zero, minimum, and overflow behavior explicitly (clamp to a sane minimum, cap at a sane maximum). Define unsupported-backend behavior explicitly.

Task 2: Divide one budget across chainstate and optional indexes

Do not multiply the configured budget once per database.

• Update NodeStorage::open, open_tx_index, and open_filter_index in crates/node/src/state.rs to read config.dbcache_mb and pass a cache budget into each store.
• Where the backend supports a shareable native cache handle (RocksDB Cache), construct one cache from the total budget and share it across the chainstate, txindex, and filters instances (Assumption 1).
• Where the backend cannot share a cache object, partition the budget by fixed fractions across the enabled subsystems so the total never exceeds dbcache_mb.
• Log the effective per-instance capacity at startup and expose it as a gauge. The capacity must be observable.

Task 3: Measure the cache change across representative workloads

An early-chain replay does not exercise the cache. The UTXO set is RAM-resident there.

• Measure cold start, warm replay, modern-corpus apply, and an indexed (txindex + blockfilterindex enabled) workload, interleaved with control.
• Capture the Phase 1.3 storage metrics (cache hit/miss, flush time) for each run.
• Produce a disposition: land, reject, or defer, backed by the measured deltas. Do not generalize a loopback result to public-WAN full-tip IBD.

🤖 Prompt for AI agents

Implement Phase 3: make `dbcache_mb` an effective, non-multiplied configuration
value.

1. Cache-budget-aware backend constructors:
- Add cache-size-aware constructors to backends in `crates/storage/` (for
example `open_with_cache(path, cache_bytes)` or an options struct). Keep the
existing `open(path)` as a default-backed wrapper.
- RocksDB: replace the hardcoded `BLOCK_CACHE_SIZE` with the passed budget.
Fjall: configure cache capacity explicitly. redb/MDBX: apply the budget where
the library exposes a cache knob; where it does not, document explicitly that
the value is a no-op for that backend.
- Explicitly define behavior for zero, minimum, and overflow cache sizes (clamp
to a sane minimum, cap at a sane maximum), and explicitly define
unsupported-backend behavior.

2. Divide one budget across chainstate and optional indexes:
- Update `NodeStorage::open`, `open_tx_index`, and `open_filter_index` in
`crates/node/src/state.rs` to read `config.dbcache_mb` and pass a cache budget
into each store.
- Where the backend supports a shareable native cache handle (RocksDB `Cache`),
construct one cache from the total budget and share it across chainstate,
txindex, and filters instances, per Assumption 1.
- Where the backend cannot share a cache object, partition the budget by fixed
fractions across enabled subsystems so the total never exceeds `dbcache_mb`.
- Log the effective per-instance capacity at startup and expose it as a gauge so
the capacity is observable.

3. Measure the cache change:
- Measure cold start, warm replay, modern-corpus apply, and an indexed workload
(txindex + blockfilterindex enabled), interleaved with control runs.
- Capture the Phase 1.3 storage metrics (cache hit/miss, flush time) for each
run.
- Produce a disposition (land, reject, or defer) backed by measured deltas. Do
not generalize a loopback result to public-WAN full-tip IBD.

Phase 4: Dynamic count-and-byte window candidate

Run this phase only if Phase 2 attributes material gap to staging depth. A constant-only bump is explicitly rejected. The count is coupled to staller arming, pending/staged parity, inline capacities, channel drain sizing, fan-out, and memory. Make every coupling explicit before changing anything.

Task 1: Make the count-coupled invariants explicit

Convert the compile-time const products into runtime-derived relationships that hold for any window size.

• In crates/node/src/sync.rs, replace the two const _: () = assert!(...) couplings with relationships that a variable window can satisfy: the staller-arming reachability invariant (byte budget must admit half the staged count at max block size) and the PENDING_BUDGET == RECEIVED_BLOCK_BUDGET inline-capacity guarantee.
• Preserve the inline SmallVec capacities (ExpectedBlockHashes, prefix-probe and cold-front vectors) or size them to a defined maximum so a deeper window does not silently spill to the heap without accounting.
• Re-derive INBOUND_BLOCK_STAGE_CHUNK, INBOUND_BLOCK_CHANNEL_LIMIT, and MIN_PEERS_FOR_FANOUT from the new window parameters rather than the literal 128.

Task 2: Implement the min(count, byte-headroom / EWMA) window

Let small historical blocks fill a deep window while the byte ceiling caps modern blocks.

• In crates/node/src/sync/window.rs, compute the effective allowed depth as min(count_ceiling, byte_headroom / observed_block_size_ewma), reusing the existing ewma_block_bytes estimate.
• Guarantee one oversized valid block remains processable at the byte ceiling. The existing BlockStager::insert already admits the next-expected block at exhaustion; preserve that no-deadlock rule.
• Introduce no eviction or re-download churn. The byte ceiling must gate admission before staging overflows.

Task 3: Preserve staller and fan-out behavior under a variable window

The staller detector must still blame only a peer-caused blocked frontier.

• Keep the three-term stall predicate (front in flight at the apply frontier, staged successor present, staged-count fraction of the window). The fraction term must track the runtime window size, not the literal 64.
• Keep the apply_side_busy no-blame guard, the adaptive timeout, the cooldown, and the single-peer fallback intact.
• Update the unit tests in crates/node/src/sync/window.rs and crates/node/src/sync.rs that hardcode 128/16/8 to express the relationship as an invariant that holds for any window size, not a fixed triple. Update the benches in sync_pipeline.rs and sync_apply_metrics.rs similarly.

Task 4: Measure and dispose the window candidate

Interleaved control, or it did not happen.

• Measure wall, CPU, peak RSS, actual staged memory, and retry/churn counters together, interleaved with control, using the #35 fixture.
• Confirm no invariant regression: staller firing behavior, backpressure, single-peer fallback, and the memory bound.
• Produce a disposition: land, reject, or defer, with the measured evidence.

🤖 Prompt for AI agents

Implement Phase 4 only if Phase 2 attributes material gap to staging depth.
Reject a constant-only bump. Make every count-coupled invariant explicit before
changing behavior.

1. Make count-coupled invariants explicit:
- In `crates/node/src/sync.rs`, replace the two `const _: () = assert!(...)`
couplings with runtime-derived relationships valid for any window size: the
staller-arming reachability invariant (byte budget must admit half the staged
count at max block size) and the `PENDING_BUDGET == RECEIVED_BLOCK_BUDGET`
inline-capacity guarantee.
- Preserve the inline `SmallVec` capacities (`ExpectedBlockHashes`,
prefix-probe, cold-front vectors), or size them to a defined maximum, so a
deeper window does not silently spill to the heap without accounting.
- Re-derive `INBOUND_BLOCK_STAGE_CHUNK`, `INBOUND_BLOCK_CHANNEL_LIMIT`, and
`MIN_PEERS_FOR_FANOUT` from the new window parameters instead of the literal
128.

2. Implement the min(count, byte-headroom / EWMA) window:
- In `crates/node/src/sync/window.rs`, compute the effective allowed depth as
`min(count_ceiling, byte_headroom / observed_block_size_ewma)`, reusing the
existing `ewma_block_bytes` estimate.
- Guarantee that one oversized valid block remains processable at the byte
ceiling; preserve the existing `BlockStager::insert` no-deadlock rule that
admits the next-expected block at exhaustion.
- Do not introduce eviction or re-download churn. The byte ceiling must gate
admission before staging overflows.

3. Preserve staller and fan-out behavior:
- Keep the three-term stall predicate (front in flight at the apply frontier,
staged successor present, staged-count fraction of the window), with the
fraction term tracking the runtime window size instead of the literal 64.
- Keep the `apply_side_busy` no-blame guard, the adaptive timeout, the cooldown,
and the single-peer fallback intact.
- Update unit tests in `crates/node/src/sync/window.rs` and
`crates/node/src/sync.rs` that hardcode `128`/`16`/`8` to express the
relationship as an invariant valid for any window size. Update the benches in
`sync_pipeline.rs` and `sync_apply_metrics.rs` the same way.

4. Measure and dispose:
- Measure wall time, CPU time, peak RSS, actual staged memory, and retry/churn
counters together, interleaved with control, using the `#35` fixture.
- Confirm no invariant regression in staller firing behavior, backpressure,
single-peer fallback, and the memory bound.
- Produce a disposition (land, reject, or defer) backed by the measured
evidence.

Phase 5: Raw-first staging candidate

Run this phase to make the byte budget honest and to remove decode from reader threads. It pairs naturally with Phase 4 because a smaller per-entry footprint permits a deeper small-block window under the same RSS limit. The hard rule: no second parse or serialization pass on the apply path.

Task 1: Defer full decode in the stager

Store the wire payload plus minimal scheduler metadata, not a decoded block.

• Change ReceivedBlock in crates/node/src/sync/stage.rs to retain the checksum-validated bytes::Bytes and only the identity metadata the scheduler needs (block hash, previous-block hash, byte length). Drop the eager bitcoin::Block field (Assumption 2).
• Move full bitcoin::Block decode out of the peer reader / inbound-drain path and into the point where a block enters an apply window.
• Update the byte accounting so received_bytes reflects the actual retained payload, which now matches the byte budget instead of roughly doubling it.

Task 2: Prove a single parse on the apply path

The apply path already accepts a serialized payload. Use it.

• When a staged block enters the apply window, decode once and hand both the decoded block and the same bytes::Bytes to apply_block_with_serialized, so the existing path parses exactly once.
• Confirm by inspection and by a decode-time metric that no additional full parse or serialization pass was added relative to the current pipeline. The acceptance criterion demands this proof.
• Preserve drain_expected_prefix, restore_many, and the next-expected admission rule.

Task 3: Measure and dispose the raw-first candidate

• Measure retained staged memory (now expected to be roughly halved), reader-thread CPU, wall, and peak RSS, interleaved with control.
• Produce a disposition: land, reject, or defer.

🤖 Prompt for AI agents

Implement Phase 5: make the byte budget honest by deferring block decode,
without adding a second parse or serialization pass on the apply path.

1. Defer full decode in the stager:
- Change `ReceivedBlock` in `crates/node/src/sync/stage.rs` to retain the
checksum-validated `bytes::Bytes` and only minimal identity metadata (block
hash, previous-block hash, byte length). Drop the eager `bitcoin::Block` field,
per Assumption 2.
- Move full `bitcoin::Block` decode out of the peer reader / inbound-drain path,
to the point where a block enters an apply window.
- Update byte accounting so `received_bytes` reflects the actual retained
payload, matching the byte budget instead of roughly doubling it.

2. Prove a single parse on the apply path:
- When a staged block enters the apply window, decode once, then hand both the
decoded block and the same `bytes::Bytes` to `apply_block_with_serialized` so
the existing path parses exactly once.
- Confirm by inspection and by a decode-time metric that no additional full
parse or serialization pass was added relative to the current pipeline; this
proof is required by the acceptance criterion.
- Preserve `drain_expected_prefix`, `restore_many`, and the next-expected
admission rule.

3. Measure and dispose:
- Measure retained staged memory (expected roughly halved), reader-thread CPU,
wall time, and peak RSS, interleaved with control.
- Produce a disposition: land, reject, or defer.

Phase 6: Conditional peer-quality scheduling and deferred storage work

Run this phase only if Phase 2 shows peer heterogeneity is material. If every peer performs alike on the loopback fixture, this is wasted work; reject it and say so.

Task 1: Layer a peer-quality ranking as a secondary key

Do not hand all work to the fastest peer, and do not resurrect a cooling-down staller.

• Feed the Phase 1.2 per-peer throughput/latency/frontier-blocking history into the ordering inside sync_peer_selection() and the filter in statically_fanout_eligible() in crates/node/src/sync.rs.
• Use the quality score as a secondary ranking key only. The soft_blocked exclusion (expired pending / staller cooldown), the apply_side_busy no-blame guard, and the single-peer last-resort fallback must all remain authoritative over the score.
• Retain peer diversity and bounded exploration. Never assign all work permanently to one peer.
• Measure, then dispose: land, reject, or defer.

Task 2: Record lower-ceiling storage work as deferred

Do not touch block-body persistence to shave the bounded 4.66s cost.

• Investigate window-level block-file/index batching and incremental checkpoints only if Phase 2 attribution shows they are material. Otherwise defer them explicitly with a written reason.
• Do not weaken body availability, reorg readiness, pruning ordering, or crash durability. Confirm reopen, interrupted-write, branch-switch, pruning, and checkpoint tests still preserve the durability and body-availability contract.

🤖 Prompt for AI agents

Implement Phase 6 only if Phase 2 shows peer heterogeneity is material. If every
peer performs alike on the loopback fixture, reject this work explicitly and
record the reason.

1. Layer a peer-quality ranking as a secondary key:
- Feed the Phase 1.2 per-peer throughput/latency/frontier-blocking history into
the ordering inside `sync_peer_selection()` and the filter in
`statically_fanout_eligible()` in `crates/node/src/sync.rs`.
- Use the quality score only as a secondary ranking key. Keep the `soft_blocked`
exclusion (expired pending / staller cooldown), the `apply_side_busy` no-blame
guard, and the single-peer last-resort fallback fully authoritative over the
score.
- Retain peer diversity and bounded exploration; never assign all work
permanently to one peer.
- Measure, then produce a disposition: land, reject, or defer.

2. Record lower-ceiling storage work as deferred:
- Investigate window-level block-file/index batching and incremental checkpoints
only if Phase 2 attribution shows they are material; otherwise defer them
explicitly with a written reason.
- Do not weaken body availability, reorg readiness, pruning ordering, or crash
durability.
- Confirm reopen, interrupted-write, branch-switch, pruning, and checkpoint
tests still preserve the durability and body-availability contract.

Research

This is a research ticket. Two headroom targets exist: the P2P staging window and the database cache. The work adds attribution instrumentation first, measures a controlled baseline, then evaluates four candidates (dynamic count-and-byte window, raw-first staging, peer-quality scheduling, effective dbcache_mb). Each candidate must earn an evidence-backed disposition: land, reject, or defer.

Codebase Summary

The sync window lives in crates/node/src/sync.rs with two submodules: sync/window.rs (DownloadWindow: in-flight/pending tracking, fan-out, staller detection) and sync/stage.rs (BlockStager: out-of-order staging). The window is pinned to PENDING_BUDGET = RECEIVED_BLOCK_BUDGET = 128, a 2 MiB per-block estimate, and a ~256 MiB byte budget. Two const _: () = assert!(...) checks couple these counts to bytes at compile time. BlockStager keeps both a decoded bitcoin::Block and the raw bytes::Bytes per entry, so a full out-of-order window costs roughly double the accounted bytes.

Storage lives in crates/storage/. NodeStorage::open (in crates/node/src/state.rs) dispatches to per-backend open(path) constructors. None of them accept a cache size. Config::dbcache_mb (default 450) is parsed and merged through CLI/env/bitcoin.conf, then read by nobody. RocksDB hardcodes a 256 MiB block cache. state.rs opens up to three separate backend instances (chainstate, txindex, filters), each with the full column-family set, so any naive cache wiring multiplies the budget per database.

Metrics live in crates/node/src/metrics.rs: an in-memory recorder, snapshot-only, no exporter. The apply path is well instrumented. The sync window has stall/retry/disconnect counters. There is no apply-idle metric, no per-peer RTT or bytes/sec, no inbound-queue-wait, no decode-time, no high-water marks, no retained-memory gauge, and zero storage-backend instrumentation. The controlled comparator and measurement discipline are documented in prior campaign plans and the loopback-fixture methodology doc.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: add attribution instrumentation to the sync pipeline and
storage layer. Do not change any scheduling or storage behavior in this phase;
only add measurement.

1. Sync-path timing and high-water attribution:
- In `crates/node/src/sync.rs` and `crates/node/src/sync/window.rs`, record
apply-idle time, download-blocked-by-apply time (the `apply_side_busy` no-blame
condition duration), inbound-queue wait time, and block decode time.
- Add high-water-mark gauges for pending blocks, staged blocks, pending bytes,
and staged bytes. Promote the existing test-only `peak_staged_bytes` concept to
a production gauge.
- Make `credit_duplicate_delivery` emit a duplicate-delivery counter. Reuse
`retry_count`, `staller_disconnects`, and `pending_timeout_disconnects`; do not
duplicate these counters.
- Follow the existing `metrics::counter!/gauge!/histogram!` call-site pattern;
do not introduce a new metrics abstraction.

2. Per-peer throughput and latency instrumentation:
- Add per-`SocketAddr` state in `DownloadWindow`, parallel to `peer_inflight`
and `recent_stallers`, recording per-block completion latency (from
`PendingBlock.requested_at` to delivery) and a bytes/sec EWMA.
- Do not add new fields to `PeerInfo` (it is handshake-only and rebuilt each
connection); keep this state in the window, keyed by address.
- Emit per-peer gauges/histograms for block RTT and bytes/sec EWMA. Treat this
data as diagnostic only in this phase; it must not feed peer selection yet.

3. Storage-backend instrumentation:
- Add write, flush, and cache-behavior instrumentation to backends in
`crates/storage/`, starting with `rocksdb_impl.rs` (attach a RocksDB
`Statistics` collector, expose block-cache hit/miss and compaction counters) and
`fjall_impl.rs`.
- Time `KvStore::flush()` at its call sites or inside each impl. Use
backend-neutral metric names where possible.
- If `bitcoin-rs-storage` cannot cleanly depend on the `metrics` crate, expose
counters via a small stats accessor on each store and record them from
`crates/node`. Choose the least invasive option that keeps the trait layer
clean.

4. Surface attribution into the measurement artifact:
- Extend the artifact emitted by `crates/node/examples/mainnet_prefix_replay.rs`
and the controlled-fixture harness from `#35` to snapshot all new metrics
alongside existing per-stage apply timers, peak RSS, and elapsed time.
- Add a distinct field for actual retained staged memory (decoded plus
serialized) so the "roughly double" claim becomes measured, not asserted.
===============================================================================

Task: 2

Implement Phase 2: produce a controlled interleaved baseline with attribution
before any mechanism changes.

- Use the controlled P2P fixture from `#35` together with the measurement rules
from `#39/`#43: verify binary identity, use a fresh datadir, verify stop-hash byte
identity at a fixed height, interleave candidate and control runs, and measure
wall time and CPU time together.
- Capture the Phase 1 attribution outputs for this baseline run. Identify where
the gap lives: apply-idle-on-network, decode on reader threads,
staging-depth-limited verification windows, or storage.
- Record the standing comparator numbers as frozen values; do not re-derive
them.
- Produce an attribution report that names the dominant cost. Use this report to
gate which candidates in Phases 3–6 are worth running. Reject at planning time
any expected gain below the noise floor (~15s per the prior campaign).
===============================================================================

Task: 3

Implement Phase 3: make `dbcache_mb` an effective, non-multiplied configuration
value.

1. Cache-budget-aware backend constructors:
- Add cache-size-aware constructors to backends in `crates/storage/` (for
example `open_with_cache(path, cache_bytes)` or an options struct). Keep the
existing `open(path)` as a default-backed wrapper.
- RocksDB: replace the hardcoded `BLOCK_CACHE_SIZE` with the passed budget.
Fjall: configure cache capacity explicitly. redb/MDBX: apply the budget where
the library exposes a cache knob; where it does not, document explicitly that
the value is a no-op for that backend.
- Explicitly define behavior for zero, minimum, and overflow cache sizes (clamp
to a sane minimum, cap at a sane maximum), and explicitly define
unsupported-backend behavior.

2. Divide one budget across chainstate and optional indexes:
- Update `NodeStorage::open`, `open_tx_index`, and `open_filter_index` in
`crates/node/src/state.rs` to read `config.dbcache_mb` and pass a cache budget
into each store.
- Where the backend supports a shareable native cache handle (RocksDB `Cache`),
construct one cache from the total budget and share it across chainstate,
txindex, and filters instances, per Assumption 1.
- Where the backend cannot share a cache object, partition the budget by fixed
fractions across enabled subsystems so the total never exceeds `dbcache_mb`.
- Log the effective per-instance capacity at startup and expose it as a gauge so
the capacity is observable.

3. Measure the cache change:
- Measure cold start, warm replay, modern-corpus apply, and an indexed workload
(txindex + blockfilterindex enabled), interleaved with control runs.
- Capture the Phase 1.3 storage metrics (cache hit/miss, flush time) for each
run.
- Produce a disposition (land, reject, or defer) backed by measured deltas. Do
not generalize a loopback result to public-WAN full-tip IBD.
===============================================================================

Task: 4

Implement Phase 4 only if Phase 2 attributes material gap to staging depth.
Reject a constant-only bump. Make every count-coupled invariant explicit before
changing behavior.

1. Make count-coupled invariants explicit:
- In `crates/node/src/sync.rs`, replace the two `const _: () = assert!(...)`
couplings with runtime-derived relationships valid for any window size: the
staller-arming reachability invariant (byte budget must admit half the staged
count at max block size) and the `PENDING_BUDGET == RECEIVED_BLOCK_BUDGET`
inline-capacity guarantee.
- Preserve the inline `SmallVec` capacities (`ExpectedBlockHashes`,
prefix-probe, cold-front vectors), or size them to a defined maximum, so a
deeper window does not silently spill to the heap without accounting.
- Re-derive `INBOUND_BLOCK_STAGE_CHUNK`, `INBOUND_BLOCK_CHANNEL_LIMIT`, and
`MIN_PEERS_FOR_FANOUT` from the new window parameters instead of the literal
128.

2. Implement the min(count, byte-headroom / EWMA) window:
- In `crates/node/src/sync/window.rs`, compute the effective allowed depth as
`min(count_ceiling, byte_headroom / observed_block_size_ewma)`, reusing the
existing `ewma_block_bytes` estimate.
- Guarantee that one oversized valid block remains processable at the byte
ceiling; preserve the existing `BlockStager::insert` no-deadlock rule that
admits the next-expected block at exhaustion.
- Do not introduce eviction or re-download churn. The byte ceiling must gate
admission before staging overflows.

3. Preserve staller and fan-out behavior:
- Keep the three-term stall predicate (front in flight at the apply frontier,
staged successor present, staged-count fraction of the window), with the
fraction term tracking the runtime window size instead of the literal 64.
- Keep the `apply_side_busy` no-blame guard, the adaptive timeout, the cooldown,
and the single-peer fallback intact.
- Update unit tests in `crates/node/src/sync/window.rs` and
`crates/node/src/sync.rs` that hardcode `128`/`16`/`8` to express the
relationship as an invariant valid for any window size. Update the benches in
`sync_pipeline.rs` and `sync_apply_metrics.rs` the same way.

4. Measure and dispose:
- Measure wall time, CPU time, peak RSS, actual staged memory, and retry/churn
counters together, interleaved with control, using the `#35` fixture.
- Confirm no invariant regression in staller firing behavior, backpressure,
single-peer fallback, and the memory bound.
- Produce a disposition (land, reject, or defer) backed by the measured
evidence.
===============================================================================

Task: 5

Implement Phase 5: make the byte budget honest by deferring block decode,
without adding a second parse or serialization pass on the apply path.

1. Defer full decode in the stager:
- Change `ReceivedBlock` in `crates/node/src/sync/stage.rs` to retain the
checksum-validated `bytes::Bytes` and only minimal identity metadata (block
hash, previous-block hash, byte length). Drop the eager `bitcoin::Block` field,
per Assumption 2.
- Move full `bitcoin::Block` decode out of the peer reader / inbound-drain path,
to the point where a block enters an apply window.
- Update byte accounting so `received_bytes` reflects the actual retained
payload, matching the byte budget instead of roughly doubling it.

2. Prove a single parse on the apply path:
- When a staged block enters the apply window, decode once, then hand both the
decoded block and the same `bytes::Bytes` to `apply_block_with_serialized` so
the existing path parses exactly once.
- Confirm by inspection and by a decode-time metric that no additional full
parse or serialization pass was added relative to the current pipeline; this
proof is required by the acceptance criterion.
- Preserve `drain_expected_prefix`, `restore_many`, and the next-expected
admission rule.

3. Measure and dispose:
- Measure retained staged memory (expected roughly halved), reader-thread CPU,
wall time, and peak RSS, interleaved with control.
- Produce a disposition: land, reject, or defer.
===============================================================================

Task: 6

Implement Phase 6 only if Phase 2 shows peer heterogeneity is material. If every
peer performs alike on the loopback fixture, reject this work explicitly and
record the reason.

1. Layer a peer-quality ranking as a secondary key:
- Feed the Phase 1.2 per-peer throughput/latency/frontier-blocking history into
the ordering inside `sync_peer_selection()` and the filter in
`statically_fanout_eligible()` in `crates/node/src/sync.rs`.
- Use the quality score only as a secondary ranking key. Keep the `soft_blocked`
exclusion (expired pending / staller cooldown), the `apply_side_busy` no-blame
guard, and the single-peer last-resort fallback fully authoritative over the
score.
- Retain peer diversity and bounded exploration; never assign all work
permanently to one peer.
- Measure, then produce a disposition: land, reject, or defer.

2. Record lower-ceiling storage work as deferred:
- Investigate window-level block-file/index batching and incremental checkpoints
only if Phase 2 attribution shows they are material; otherwise defer them
explicitly with a written reason.
- Do not weaken body availability, reorg readiness, pruning ordering, or crash
durability.
- Confirm reopen, interrupted-write, branch-switch, pruning, and checkpoint
tests still preserve the durability and body-availability contract.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[rabbitson87]

Cross-reference from the read-path / UTXO-memory campaign (#80, #83) and its follow-up survey (#84). Two of this ticket's observations were independently reproduced, and one of them has since cost a published claim.

dbcache_mb is not plumbed — confirmed, zero references. crates/storage contains no occurrence of dbcache at all. It is parsed at crates/node/src/config.rs:162 and crates/node/src/bitcoin_conf_compat.rs:64, layered at config.rs:574, and reaches nothing.

This is not only a config lie, it actively blocked attribution work. docs/benchmarks/utxo-memory.md had to retract a claim that 450 MB of residual RSS was the configured dbcache, because the setting reaches no constructor. With no lever between config and the backends there is no way to tune the one G14 budget that is anywhere near binding (tip RSS ≤ 16 GiB) and re-measure, so the non-UTXO residual — 0.64 GiB at height 412,732 — stays unattributed.

Staging double-retention has a sibling worth measuring at the same time. This ticket notes crates/node/src/sync/stage.rs retaining both a decoded bitcoin::Block and the original bytes::Bytes. The survey in #84 found the same shape elsewhere: crates/rpc/src/context.rs:534 pushes one BlockRecord per applied block and nothing ever removes one. Pruning (crates/node/src/state.rs:566) blanks block_hex but keeps the record; the only removal is a single-entry reorg undo. At 264 B/record — 160 of which is an 80-byte header stored as a hex String — that is 103.9 MiB at height 412,732 and 241.1 MiB at tip, roughly a sixth of the unattributed residual above.

One measurement that may be useful here. The UTXO set was attributed against process RSS on a real pruned mainnet chainstate at height 412,732 (38,145,360 outputs): it is 77.4% of process RSS, not the ~44% an in-flight sample suggests — in-flight samples swing between 1.1 and 3.2 GB and are measuring block staging, which is this ticket's subject. Anyone measuring staging memory should sample from the checkpoint path, where the subsystems have drained, and treat in-flight numbers as staging rather than steady state.

crates/utxo/examples/snapshot_memory.rs (in #83) loads a real utxo-v4.dat and reports the set's cost against process RSS, if a baseline is useful.