Architectural boundary: durable chain events and independent derived-state consumers

[gosunuts]

Problem

Optional products currently perform work directly in synchronous block apply: txindex ingestion, BIP157/158 filters, mempool eviction, coinstats updates, and ZMQ publication. Electrum depends on txindex storage; Utreexo has a separate state model; the BIP300/301 enforcer consumes post-apply interfaces. Each new product requirement can therefore add locks, persistence ordering, and reorg handling to consensus-critical code.

Goal

Keep full-node chainstate transitions authoritative and small, while non-consensus derived state becomes independently recoverable consumers of one durable, ordered chain-event boundary.

P2P -> headers/bodies -> consensus -> UTXO + body/undo + applied tip
                                                   |
                                      durable ChainEvent source
                                                   |
    txindex | Electrum | filter index | enforcer adapter | Utreexo mode

The default full-node path must not wait for, trust, or become unavailable because a derived consumer is slow, disabled, failed, or rebuilding.

Requirements

Authoritative core

The core apply/disconnect path owns consensus validation, default full-UTXO mutation, required body/undo persistence, the authoritative applied tip, and durable emission of ordered state-transition records. Optional indexes, Electrum serving, external notifications, and experimental Utreexo state are not consensus prerequisites.

Durable event contract

Define a versioned, ordered record containing at least:

• ConnectedBlock: height, hash, parent hash, body locator, and data needed for derived state.
• DisconnectedBlock: height, hash, prior/replacement context, and a durable undo/body locator as needed.
• A monotonic event cursor independent of the ZMQ transport sequence.
• Schema/version and retention guarantees.

The contract must allow a consumer to restart from a stored cursor without relying on best-effort ZMQ. It may be an append-only log or an authoritative-storage journal with equivalent replay guarantees.

Mempool acceptance/removal should later use the same abstraction, but must not block the block-event boundary.

Consumer lifecycle

Every optional consumer owns:

• an independent durable cursor expressed as event sequence plus height/hash;
• state: disabled, building, caught-up, paused, failed, or stale;
• backfill from a named active-chain anchor;
• idempotent connect/disconnect processing;
• interruption/event-retention/reorg recovery; and
• an explicit query-readiness contract.

A consumer failure must not invalidate chainstate. Its API becomes unavailable or explicitly partial until it catches up.

Ordering, reorgs, and external notifications

• Events reflect only committed authoritative transitions.
• Reorgs preserve tip-first disconnect then replacement-branch connect ordering.
• Consumer backfill first establishes a cursor/subscription, scans its historical anchor, then drains subsequent events.
• Required body, undo, and event retention must be reconciled with pruning and each consumer's rebuild procedure.
• ZMQ remains a low-latency best-effort projection of committed events, not a recovery log.
• The BIP300/301 enforcer remains an external sidecar and must not hold core apply locks or gate ordinary Bitcoin block application.

Migration plan

1. Specify and test the connect/disconnect event, cursor, and recovery contract.
2. Extract txindex and filter-index work from inline apply into cursor-owning consumers; implement Add resumable post-IBD backfill for txindex, Electrum, and block filters #75 on this boundary.
3. Attach Electrum readiness and notifications to txindex consumer state.
4. Add mempool consumer semantics and A/R events; see Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74.
5. Use the boundary for experimental Utreexo state progression without making it a default-consensus dependency; see Implement an experimental Utreexo compact-validator node mode #76.
6. Retire direct optional side effects from apply.rs after parity/reorg/recovery coverage passes.

Acceptance criteria

• The event schema, ordering, durability, cursor semantics, and retention policy are versioned and documented.
• A consumer can stop at any event, restart, and converge to the same state as uninterrupted processing.
• Connect/disconnect ordering is proven across ordinary and invalidation-triggered reorgs plus restart boundaries.
• Default full-node apply remains correct and available when all optional consumers are disabled, paused, slow, or failed.
• Txindex/filter inline work moves to independently catch-up-capable consumers without changing final index rows.
• Electrum exposes source-index readiness and does not claim complete history while rebuilding.
• ZMQ event loss has a defined consumer recovery path.
• Enforcer integration remains asynchronous and external I/O occurs outside consensus/apply locks.
• Utreexo stays isolated behind its experimental mode and consumes the declared transition contract.
• Default IBD has no optional-index work on its critical path.

Related issues

• Unify UTXO crates under one state model #6: parent UTXO state-model unification.
• Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74: mempool A/R pubsequence events.
• Add resumable post-IBD backfill for txindex, Electrum, and block filters #75: post-IBD txindex, Electrum, and filter backfill.
• Implement an experimental Utreexo compact-validator node mode #76: experimental Utreexo compact validator.
• Plan #1: original chain-event-bus direction.

Non-goals

• Changing Bitcoin consensus or weakening default full-UTXO validation.
• Requiring optional consumers for normal node operation.
• Treating ZMQ as a durable queue.
• Making the external enforcer an inline consensus oracle.
• Defining Utreexo proof protocol details, which Implement an experimental Utreexo compact-validator node mode #76 owns.

[coderabbitai]

🔗 Related PRs

#14 - feat: header timestamp consensus rules, index reorg rollback, mempool policy, fuzz targets [merged]
#30 - perf(consensus): batch Merkle hashing with AVX2 [merged]
#49 - rpc: Core-compatible REST gateway so a remote bip300301_enforcer can use bitcoin-rs as its mainchain node [merged]
#52 - node: publish Core's sequence ZMQ topic for block connects and disconnects [merged]
#73 - Feature/drynet sync [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[gosunuts]

I think there is a simpler way to satisfy the recovery goal without putting a per-block durable WAL or fsync on the apply path.

The important distinction is between historical event replay and derived-state convergence. Txindex, filters, and Electrum need to converge to the current authoritative chain; they do not need an audit log containing every transient transition that ever occurred.

I would therefore make the boundary:

• authoritative active-chain view + retained block bodies = durable recovery source;
• in-memory ChainEvent delivery = low-latency, best-effort work hint;
• consumer-owned (height, hash) = durable recovery anchor;
• (epoch, sequence) = restart/gap/order detection, not the source of truth.

Conceptually:

struct ConsumerCursor {
    epoch: u64,
    sequence: u64,
    height: u32,
    hash: BlockHash,
}

struct ChainSnapshot {
    epoch: u64,
    sequence: u64,
    tip: TipSnapshot,
}

The epoch is allocated and flushed once when the node starts. Sequence is monotonic within that process. A restart always changes the epoch, so a cursor is never silently reused. Per-block apply then only needs an atomic increment, one coherent snapshot publication, and non-blocking try_send notifications; no event DB write or fsync belongs on the IBD critical path.

A consumer subscribes first, captures a coherent ChainSnapshot, reconciles its durable height/hash anchor to that snapshot, and then drains live events. If the epoch changes, a sequence gap appears, or the bounded queue overflows, it reconciles again:

1. If the consumer anchor is still on the active chain, connect anchor + 1 .. current tip.
2. Otherwise find the common ancestor, disconnect consumer state tip-first to it, then connect the current branch.
3. Commit each derived-state mutation and the consumer cursor in the same consumer-owned DB batch.
4. Report caught-up only when the consumer anchor equals a fresh core snapshot.

This also gives a clean pruning rule: core guarantees a declared body retention window, but a stale consumer must not hold core sync or pruning indefinitely. If required bodies fall outside that window, only that consumer becomes stale and rebuilds.

One wording change may be needed in this issue: “durable event boundary” should mean a durable reconciliation contract with equivalent convergence guarantees, not necessarily an append-only audit log of every historical transition. If exact historical transition replay is a hard requirement, then a WAL, group commit, startup replay, and body/undo durability ordering are unavoidable. For the derived consumers named here, that seems like complexity and I/O cost without a corresponding correctness benefit.

This keeps the safety property that optional consumers cannot block or fail core apply, while still proving restart, missed-event, and reorg convergence.

[gosunuts]

Final design clarification after checking this proposal against bitcoin-rs's current checkpoint and reorg model:

The intended boundary should be reconciliation-based, not a separately persisted event journal.

Stable contract

Core owns

• the authoritative active-chain snapshot;
• active-chain ancestry lookup;
• retained block-body lookup for forward backfill; and
• low-latency, bounded, non-blocking live ChainEvent hints.

Live events are an optimization, not a recovery source. Core apply must use non-blocking delivery and must remain correct when a consumer is disabled, slow, failed, rebuilding, or has missed every notification.

Each consumer owns

• a durable (height, hash) cursor;
• its derived rows;
• block-identified inverse/undo metadata sufficient to roll its own state back;
• its schema, migration, retention, readiness, stale, and rebuild policy.

A consumer must commit the derived-state mutation, its inverse metadata, and the new cursor in one consumer-owned DB batch.

Reconciliation algorithm

A consumer subscribes to live hints first, then captures one coherent core snapshot containing the current epoch/sequence and applied tip.

1. If the consumer's (height, hash) anchor is on the active chain, connect the active blocks after that anchor through the captured tip.
2. If the anchor is on an old branch, use consumer-owned inverse metadata to disconnect tip-first to the common ancestor.
3. Connect the current branch from the common ancestor through the captured tip.
4. Drain queued live hints.
5. Capture a fresh core snapshot. Only report caught-up when the consumer anchor equals that snapshot. An epoch change, sequence gap, queue overflow, or changed tip simply triggers reconciliation again.

The sequence and process epoch detect missed or reordered hints. They are not durable truth and do not need per-block persistence. A new process epoch prevents a restarted node from silently reusing an old in-memory sequence.

Important reorg constraint

The active-chain view is sufficient for forward backfill, but it cannot by itself reconstruct deletion of old-branch derived state after that branch stops being active. Therefore every incrementally recoverable consumer must retain block-identified inverse information, such as the exact keys/rows created by each processed block.

If the required inverse data or retained forward block bodies are unavailable, only that consumer becomes stale and performs a full rebuild. Core sync and other consumers continue normally.

Rollback must not delete rows by height alone. The current index layout contains families whose keys do not fully identify the producing block, so a same-height competing block can otherwise lose its rows during reorg flip-flop. Deletion must validate the producing block identity, use block-specific inverse keys, or migrate to identity-bearing rows.

Hash-addressed consumers may use a cheaper policy. For example, compact-filter rows keyed by block hash can remain cached for side-chain blocks; only the consumer's active filter-header pointer/cursor needs to rewind before constructing the replacement branch.

Durability and pruning

This design deliberately does not introduce a core event WAL, journal replay, per-block event fsync, or a second source of chain truth. Core defines a finite body-retention guarantee. A consumer lagging beyond that guarantee becomes stale/rebuilding instead of indefinitely blocking pruning.

The resulting invariant is:

Missing any number of live notifications may delay a consumer, but cannot prevent it from converging to the authoritative active chain while its required consumer undo and core-retained forward bodies remain available.

This is the design I would implement before extracting the generic extension lifecycle in #79.

[coderabbitai]

Coding Plan

Summary

• Make the core expose a coherent active-chain snapshot (epoch, sequence, tip), active-chain ancestry/common-ancestor lookup, and retained (height, hash) body/undo lookup, reusing BlockTree and the existing body/undo stores. Do not add a durable per-block event journal.
• Deliver live ChainEvent hints over a bounded, non-blocking try_send channel that only triggers reconciliation. Neither hints nor ZMQ is a source of truth or a recovery log.
• Fix txindex rollback identity first, then build txindex as one concrete reconciliation consumer end-to-end, with a consumer-owned atomic batch (rows plus block-identified inverse metadata plus (height, hash) cursor), an explicit reconcile algorithm, and enumerated tests. Defer any generic consumer framework to a future ticket (Define a static opt-in extension model and workspace extension API #79).
• Keep pruning as a finite, core-controlled policy; a lagging consumer goes Stale and rebuilds. Then migrate filters, attach RPC/Electrum readiness, confirm the enforcer stays external, and hook Utreexo behind its feature, removing the inline optional work from apply.rs only after parity and reorg and restart coverage passes.

Design Choices

Design Choice 1: Source of truth for consumer recovery: a durable per-block event journal, or reconciliation against a coherent snapshot?

Options Considered:

1. Durable per-block ChainEvent journal in a new ChainEvents column family, with a global durable cursor and startup replay. Every committed block writes and fsyncs an event record. This adds write amplification and an fsync to the consensus path, adds a second durable ordering that can disagree with chainstate after a crash, and duplicates ordering that BlockTree already owns.
2. Reconciliation against a coherent active-chain snapshot. The core exposes (epoch, sequence, tip), active-chain ancestry, and retained (height, hash) body/undo lookup. Consumers store their own (epoch, sequence, height, hash) cursor with their own derived state. Live hints are best-effort, bounded, non-blocking triggers. Recovery is a reconcile against a fresh snapshot, not a log replay.

Chosen Option: 2

Rationale: Option 2 keeps the consensus path free of any per-block event write, reuses BlockTree ancestry and the existing (height, hash) body/undo stores as the durable source, and matches the ticket rule that ZMQ (and now the hint channel) is a best-effort projection, not a recovery log.

Design Choice 2: Consumer execution model: cooperative event-loop ticks, or independent OS threads?

Options Considered:

1. Run consumers as cooperative branches inside EventLoop::spin. Keeps one thread, but couples a slow or failed consumer to the consensus tick loop, which the ticket forbids.
2. Run each consumer as an independent OS thread spawned like spawn_electrum_listener, gated by config, joined after spin.

Chosen Option: 2

Rationale: Isolation is the point of this ticket. A consumer that stalls or panics must not stall the event loop or the apply path.

Design Choice 3: Process epoch representation for restart detection.

Options Considered:

1. A startup nonce assigned in memory each NodeState::open. Simple, but not ordered across restarts.
2. A persisted monotonic counter using the checkpoint.rs generation/CURRENT allocate-read pattern, advanced once per process start.

Chosen Option: 2

Rationale: An ordered, persisted epoch lets a consumer reason about restart order and reuses an existing durable counter pattern instead of a fresh mechanism.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Build the core active-chain boundary

The core must expose three read surfaces and one best-effort trigger. No consumer reads them yet. This phase builds the snapshot, the ancestry and body lookup accessors, and the bounded hint channel. There is no ChainEvents column family, no journal writer, no global durable cursor, and no per-block event write. If you find yourself adding an fsync to the apply path here, stop. You have taken the wrong turn.

Task 1: Expose a coherent active-chain snapshot

Give consumers one atomic read of (epoch, sequence, tip).

• Add a snapshot value type carrying epoch: u64, sequence: u64, tip_hash, and tip_height.
• Add an in-memory monotonic sequence counter. Advance it once per committed connect and once per committed disconnect, after applied_tip.store(...) in apply_block_admitted and disconnect_block_admitted in crates/node/src/apply.rs, at the same point SequenceEvent fires today.
• Add a process epoch assigned in NodeState::open in crates/node/src/state.rs, using the checkpoint.rs generation/CURRENT allocate-read pattern.
• Expose NodeState::active_chain_snapshot(). It must return a coherent, non-torn read of the four fields.
• This is a live snapshot value, not a durable event. Do not persist a per-event record.

Task 2: Expose active-chain ancestry and common-ancestor lookup

Let a consumer ask where its cursor sits relative to the active chain.

• Reuse BlockTree in crates/chain/src/tree.rs: active_node_at_height, header_at_active_height, find_common_ancestor, height_of_hash, node_by_hash, iter_active_chain_hashes. Do not reimplement ancestry.
• Add a thread-safe read accessor on NodeState over the existing Arc<RwLock<BlockTree>>.
• Add one helper that answers whether (height, hash) is still on the active chain, by comparing the active_node_at_height(height) hash to hash.
• Reuse plan_reorg (crates/chain/src/reorg.rs) for common-ancestor plus disconnect/connect lists when a consumer must switch branches.

Task 3: Expose retained body and undo lookup by (height, hash)

Give consumers the block data they need to connect and disconnect, verified by identity.

• Reuse PruneBodyStore::load_block_body(height, hash) (crates/node/src/apply.rs) over FlatFileBlockStore::load (crates/storage/src/block_file.rs), which already verifies both height and hash and returns None on mismatch.
• Reuse the (height, hash)-keyed undo store for disconnect data.
• Add a consumer-facing accessor that returns None when the data is pruned. None is the signal that the consumer must go Stale and rebuild. Do not extend retention to avoid it.

Task 4: Add a bounded, non-blocking ChainEvent hint channel

Wake consumers on committed transitions without ever blocking apply.

• Define a ChainEvent hint: kind (Connected/Disconnected), height, hash, epoch, sequence.
• Deliver hints over a bounded channel, following the wake_sync bounded(N) plus try_send-and-drop pattern (crates/p2p/src/listener.rs) and the INBOUND_BLOCK_CHANNEL_LIMIT bound (crates/node/src/state.rs).
• Emit the hint at the commit point with try_send, right after the sequence advance in the snapshot task. On a full queue, drop the hint. A dropped hint is not a bug.
• Hints only wake the consumer. They are not a source of truth and not a recovery log. A dropped hint is recovered by reconciling against the snapshot sequence.

Task 5: Document the boundary contract and retention rule

The acceptance criteria require the boundary, ordering, and retention to be documented.

• Write a design note under docs/solutions/architecture-patterns/ describing the coherent snapshot (epoch, sequence, tip), the ancestry and body/undo lookups, the bounded best-effort hint channel, and the tip-first-disconnect ordering the core already guarantees.
• State plainly that neither the hint channel nor ZMQ is durable or replayable. Recovery is reconciliation, not replay.
• State the retention rule: pruning is a finite, core-controlled policy independent of consumer progress. A consumer that falls behind retained bodies/undo goes Stale and performs a full rebuild. Pruning is never extended or blocked for a lagging consumer.
• Add glossary entries in CONCEPTS.md next to Sequence stream, Commit point, and Undo record.

🤖 Prompt for AI agents

Build the core active-chain boundary: a coherent snapshot, ancestry/body lookup
accessors, and a bounded hint channel. Nothing consumes these yet. There is no
`ChainEvents` column family, no journal writer, no global durable cursor, and no
per-block durable event write. Do not add an fsync to the apply path.

- Add a snapshot value type carrying `epoch: u64`, `sequence: u64`, `tip_hash`,
and `tip_height`. Add an in-memory monotonic `sequence` counter, advanced once
per committed connect and once per committed disconnect, right after
`applied_tip.store(...)` in `apply_block_admitted` and
`disconnect_block_admitted` in `crates/node/src/apply.rs`, at the same point
`SequenceEvent` fires today. Add a process `epoch` assigned in `NodeState::open`
in `crates/node/src/state.rs`, using the `checkpoint.rs` `generation`/`CURRENT`
allocate-read pattern. Expose `NodeState::active_chain_snapshot()` returning a
coherent, non-torn read of all four fields. This is a live value, not a durable
event; do not persist a per-event record.
- Reuse `BlockTree` (`crates/chain/src/tree.rs`) for ancestry:
`active_node_at_height`, `header_at_active_height`, `find_common_ancestor`,
`height_of_hash`, `node_by_hash`, `iter_active_chain_hashes`. Do not reimplement
ancestry. Add a thread-safe read accessor on `NodeState` over the existing
`Arc<RwLock<BlockTree>>`. Add a helper answering whether `(height, hash)` is
still on the active chain. Reuse `plan_reorg` (`crates/chain/src/reorg.rs`) for
common-ancestor plus disconnect/connect lists.
- Reuse `PruneBodyStore::load_block_body(height, hash)`
(`crates/node/src/apply.rs`) over `FlatFileBlockStore::load`
(`crates/storage/src/block_file.rs`), already verifying height and hash and
returning `None` on mismatch. Reuse the `(height, hash)`-keyed undo store for
disconnect data. Add a consumer-facing accessor returning `None` when data is
pruned; that `None` is the signal for the consumer to go `Stale` and rebuild. Do
not extend retention to avoid it.
- Define a `ChainEvent` hint carrying `kind` (`Connected`/`Disconnected`),
`height`, `hash`, `epoch`, `sequence`. Deliver it over a bounded channel
following the `wake_sync` `bounded(N)` plus `try_send`-and-drop pattern
(`crates/p2p/src/listener.rs`) and the `INBOUND_BLOCK_CHANNEL_LIMIT` bound
(`crates/node/src/state.rs`). Emit the hint at the commit point with `try_send`,
right after the sequence advance. On a full queue, drop the hint; that is not a
bug. Hints only wake the consumer; they are not a source of truth and not a
recovery log. A dropped hint recovers via reconciliation against the snapshot
sequence.
- Write a design note under `docs/solutions/architecture-patterns/` describing
the snapshot, ancestry/body/undo lookups, the bounded hint channel, and the
tip-first-disconnect ordering the core already guarantees. State plainly that
neither the hint channel nor ZMQ is durable or replayable; recovery is
reconciliation, not replay. State the retention rule: pruning is a finite,
core-controlled policy independent of consumer progress; a lagging consumer goes
`Stale` and rebuilds; pruning is never extended or blocked for it. Add glossary
entries in `CONCEPTS.md` next to `Sequence stream`, `Commit point`, and `Undo
record`.

Phase 2: Fix rollback correctness before extracting any consumer

A consumer that reconciles reorgs on its own must be able to delete exactly the rows a specific block wrote, keyed by that block's identity. Txindex cannot do that today. Fix this first. Do not defer it to parity testing, where a reorg bug turns into silent data corruption.

Task 1: Audit derived-state row keys for producing-block identity

Confirm which products are reorg-safe and which are not.

• Record the finding for txindex: TxConfirmed/Funding/Spending keys are an 8-byte content prefix plus little-endian height, with no block hash (crates/index/src/types.rs, crates/index/src/index.rs). Rollback is guarded only by header-row presence, not by per-row hash equality. That is not enough for an independent reorg-reconciling consumer.
• Record the finding for filters: FilterIndex rows are keyed by the 32-byte block hash (crates/filters/src/filter_index.rs), so filters are already reorg-safe.
• Conclusion: txindex needs block-identified inverse metadata before Phase 3. Filters do not.

Task 2: Add block-identified inverse metadata for txindex rollback

Make txindex rollback delete only the rows a named (height, hash) block wrote.

• Add per-block inverse metadata keyed by (height, hash) that lists exactly the rows this block wrote. Rollback replays this metadata and deletes only this block's rows, never a replacement block's rows.
• This inverse metadata is the consumer's own undo data. It is written in the consumer-owned batch defined in Phase 3.
• If the row layout must change, treat it as a breaking change. Follow docs/policies/db-migration.md: keep ColumnFamily variants append-only, bump the value version byte, reject unknown versions on read, and require a datadir wipe and resync. There is no in-place migration.

🤖 Prompt for AI agents

Fix txindex rollback correctness before any consumer extraction happens. Do not
defer this to parity testing, where a reorg bug becomes silent data corruption.

- Audit row-key identity across derived-state products. Record that txindex
`TxConfirmed`/`Funding`/`Spending` keys are an 8-byte content prefix plus
little-endian height with no block hash (`crates/index/src/types.rs`,
`crates/index/src/index.rs`), and rollback is guarded only by header-row
presence, not by per-row hash equality; that is not enough for an independent
reorg-reconciling consumer. Record that `FilterIndex` rows are keyed by the
32-byte block hash (`crates/filters/src/filter_index.rs`) and are already
reorg-safe. Conclude that txindex needs block-identified inverse metadata before
Phase 3; filters do not.
- Add per-block inverse metadata keyed by `(height, hash)` listing exactly the
rows that block wrote. Rollback replays this metadata and deletes only that
block's rows, never a replacement block's rows. This inverse metadata is the
consumer's own undo data, written in the consumer-owned batch defined in Phase
3. If the row layout must change, treat it as a breaking change per
`docs/policies/db-migration.md`: keep `ColumnFamily` variants append-only, bump
the value version byte, reject unknown versions on read, and require a datadir
wipe and resync. There is no in-place migration.

Phase 3: Build txindex as the first reconciliation consumer, end-to-end

Prove the whole model with one real consumer before generalizing anything. There is no ChainEventConsumer trait, no registry, and no extension-registration mechanism in this ticket. That generalization is deferred to a future ticket (#79). Build the concrete thing, prove it, then delete the inline path.

Task 1: Define the txindex consumer storage contract

Make progress and derived state commit together, or not at all.

• Each reconciliation step commits ONE atomic consumer-owned DB batch containing: the txindex row mutations, the block-identified inverse metadata from the rollback-fix task, and the consumer cursor (epoch, sequence, height, hash) plus lifecycle state.
• There is no shared framework-level cursor store. The cursor lives with the consumer's own data.
• Define the lifecycle state enum: Disabled, Building, CaughtUp, Paused, Failed, Stale.
• Require idempotent connect and disconnect handling so a replay after interruption converges to the same rows.

Task 2: Implement the reconciliation algorithm

Reconcile the consumer to the current snapshot in explicit, ordered steps.

• Read a fresh coherent snapshot (epoch, sequence, tip).
• Trigger reconcile when any of these hold: epoch changed, a sequence gap is observed, a hint was dropped or the queue overflowed, or the snapshot tip differs from the cursor.
• Test anchor validity: is the cursor (height, hash) still on the active chain?
  • Anchor still active: fast-forward. Connect each block from cursor.height + 1 up to the snapshot tip. Read each body via the body-lookup accessor. Commit each block in one atomic batch.
  • Anchor on a stale branch: find the common ancestor of the cursor hash and the active tip. Roll back tip-first from the cursor down to the common ancestor using the consumer's own inverse metadata. Then connect forward from the common ancestor to the snapshot tip.
• If a required body or inverse record is unavailable because it was pruned (body-lookup returns None): set state Stale, then perform a full rebuild from a named active-chain anchor.
• After reaching the snapshot tip, read a fresh snapshot again. If it is unchanged, set state CaughtUp. Otherwise repeat.

Task 3: Spawn the txindex consumer worker and add its config

Run the consumer as an isolated, config-gated thread.

• Add spawn_txindex_consumer(config, state, shutdown) in crates/node/src/run.rs, following spawn_electrum_listener: config-gated, returns Result<Option<JoinHandle>>, loops while the shared AtomicBool shutdown flag is unset.
• The worker waits on the bounded hint channel and also polls the snapshot on a timer, so a dropped hint cannot stall progress.
• On a consumer error, set state Failed and stop only this worker. Never propagate into apply. Chainstate stays authoritative.
• Reuse begin_batch/end_batch (crates/index/src/index.rs) for backfill throughput.
• Add a txindex-consumer enable flag to Config in crates/node/src/config.rs, following the txindex: bool pattern, with any needed validate() cross-field rule.
• Join the worker in run() after loop_handle.spin returns, alongside the electrum and rpc joins, before shutdown::drain_and_shutdown.

Task 4: Prove txindex correctness with enumerated tests

Itemize the cases. Do not hide them behind a general "parity" label.

• Add tests in crates/node/tests/ using the tempdir plus NodeState::open / drop / reopen harness, and add a gate following the g10_reorg_deep.rs / g11_crash_recovery.rs subprocess-wrapper pattern with the loud-fail stdout guard.
• Test: stop at an arbitrary point, restart, and converge to the same rows as uninterrupted processing.
• Test: recovery after missed live events (dropped hints), driven only by snapshot reconciliation.
• Test: process-epoch change and sequence gap both trigger a correct reconcile.
• Test: bounded queue overflow does not lose correctness.
• Test: ordinary reorg and invalidateblock-triggered reorg both reconcile tip-first disconnect then replacement-branch connect.
• Test: stale detection plus full rebuild when bodies or inverse data are unavailable.
• Test: final txindex rows are identical to the current inline implementation for the same block sequence.

Task 5: Retire inline txindex work from apply

Delete the inline path only after the correctness tests pass.

• Remove the inline tx_index.ingest_* and rollback_* calls from apply_block_admitted and disconnect_block_admitted in crates/node/src/apply.rs.
• Keep consensus validation, UTXO commit, body/undo persistence, applied-tip publication, the sequence advance, and the hint emission. Those are authoritative and stay.
• Do not touch coinstats where checkpoint consistency requires it. Do not break the checkpoint cross-check in crates/node/src/checkpoint.rs.

🤖 Prompt for AI agents

Build txindex as the first concrete reconciliation consumer, end-to-end, then
delete its inline path. Do not build a generic `ChainEventConsumer` trait,
registry, or extension-registration mechanism; that generalization is deferred
to a future ticket (#79).

- Define the txindex consumer storage contract: each reconciliation step commits
ONE atomic consumer-owned DB batch containing the txindex row mutations, the
block-identified inverse metadata from Phase 2, and the consumer cursor `(epoch,
sequence, height, hash)` plus lifecycle state. There is no shared
framework-level cursor store; the cursor lives with the consumer's own data.
Define the lifecycle state enum: `Disabled`, `Building`, `CaughtUp`, `Paused`,
`Failed`, `Stale`. Require idempotent connect and disconnect handling so replay
after interruption converges to the same rows.
- Implement the reconciliation algorithm: read a fresh coherent snapshot
`(epoch, sequence, tip)`. Trigger reconcile on epoch change, sequence gap,
dropped/overflowed hint, or tip mismatch with the cursor. Test whether the
cursor `(height, hash)` is still on the active chain. If yes, fast-forward:
connect each block from `cursor.height + 1` to the snapshot tip, reading each
body via the body-lookup accessor, committing each block in one atomic batch. If
the anchor is on a stale branch, find the common ancestor, roll back tip-first
from the cursor to the common ancestor using the consumer's own inverse
metadata, then connect forward to the snapshot tip. If a required body or
inverse record is unavailable (pruned), set state `Stale` and perform a full
rebuild from a named active-chain anchor. After reaching the snapshot tip,
re-read the snapshot; if unchanged, set state `CaughtUp`, otherwise repeat.
- Add `spawn_txindex_consumer(config, state, shutdown)` in
`crates/node/src/run.rs`, following `spawn_electrum_listener`: config-gated,
returns `Result<Option<JoinHandle>>`, loops while the shared `AtomicBool`
shutdown flag is unset. The worker waits on the bounded hint channel and also
polls the snapshot on a timer so a dropped hint cannot stall progress. On a
consumer error, set state `Failed` and stop only this worker; never propagate
into apply. Reuse `begin_batch`/`end_batch` (`crates/index/src/index.rs`) for
backfill throughput. Add a txindex-consumer enable flag to `Config` in
`crates/node/src/config.rs`, following the `txindex: bool` pattern, with any
needed `validate()` cross-field rule. Join the worker in `run()` after
`loop_handle.spin` returns, alongside the electrum and rpc joins, before
`shutdown::drain_and_shutdown`.
- Add tests in `crates/node/tests/` using the `tempdir` plus `NodeState::open` /
drop / reopen harness, and a gate following the `g10_reorg_deep.rs` /
`g11_crash_recovery.rs` subprocess-wrapper pattern with the loud-fail stdout
guard. Cover each of: stop/restart convergence, dropped-hint recovery via
snapshot reconciliation, epoch change and sequence gap triggering correct
reconcile, bounded queue overflow not losing correctness, ordinary reorg and
`invalidateblock`-triggered reorg reconciling tip-first disconnect then
replacement-branch connect, stale detection plus full rebuild when data is
unavailable, and final txindex rows identical to the inline implementation for
the same block sequence.
- After tests pass, remove the inline `tx_index.ingest_*` and `rollback_*` calls
from `apply_block_admitted` and `disconnect_block_admitted` in
`crates/node/src/apply.rs`. Keep consensus validation, UTXO commit, body/undo
persistence, applied-tip publication, the sequence advance, and hint emission;
those stay authoritative. Do not touch coinstats where checkpoint consistency
requires it, and do not break the checkpoint cross-check in
`crates/node/src/checkpoint.rs`.

Phase 4: Migrate the filter index as the second reconciliation consumer

Reuse the pattern proven with txindex. Still no generic trait. Copy the shape, not an abstraction.

Task 1: Build the filter index reconciliation consumer

Drive FilterIndex from reconciliation and give it the backfill it was documented as owing.

• Use the same storage contract shape as the txindex consumer: a consumer-owned atomic batch holding the filter rows plus the consumer cursor (epoch, sequence, height, hash) and state.
• Filter rows stay hash-addressed and remain cached across disconnects. Do not roll back filter rows. On a disconnect, rewind only the active filter-header pointer/cursor to the parent.
• Maintain the BIP157 header chain from the consumer's own persisted cursor, not the single in-memory filter_header_cache slot.
• Implement backfill so an index enabled late, or after a gap, repairs itself. This is the "backfill can repair" case that apply.rs documents but never implemented.
• Reuse the reconciliation algorithm defined for txindex.

Task 2: Prove filter correctness and retire inline filter work

Itemize the tests, then delete the inline path.

• Test: final filter rows and filter headers are identical to the current inline implementation for the same block sequence.
• Test: stop/restart convergence, dropped-hint recovery, epoch change, sequence gap, and queue overflow, as with the txindex consumer.
• Test: reorg rewinds only the active filter-header pointer and keeps hash-addressed rows.
• Test: stale detection plus full rebuild when data is unavailable.
• After the tests pass, remove the inline compute_basic_filter, put_filter, and filter_header_cache handling, and the inline per-block ZMQ publish, from apply_block_admitted and disconnect_block_admitted in crates/node/src/apply.rs. Reposition ZMQ as a best-effort projection driven off committed transitions, not off inline apply.

🤖 Prompt for AI agents

Migrate the filter index onto the same reconciliation shape proven with txindex.
Do not build a generic trait; copy the pattern, not an abstraction.

- Build the filter index reconciliation consumer using the same storage contract
shape as txindex: a consumer-owned atomic batch holding filter rows plus the
consumer cursor `(epoch, sequence, height, hash)` and state. Keep filter rows
hash-addressed and cached across disconnects; do not roll back filter rows. On
disconnect, rewind only the active filter-header pointer/cursor to the parent.
Maintain the BIP157 header chain from the consumer's own persisted cursor
instead of the single in-memory `filter_header_cache` slot. Implement backfill
so an index enabled late, or after a gap, repairs itself (the "backfill can
repair" case `apply.rs` documents but never implements). Reuse the
reconciliation algorithm defined for the txindex consumer.
- Prove filter correctness with tests: final filter rows and headers identical
to the inline implementation for the same block sequence; stop/restart
convergence, dropped-hint recovery, epoch change, sequence gap, and queue
overflow, as with txindex; reorg rewinds only the active filter-header pointer
while keeping hash-addressed rows; stale detection plus full rebuild when data
is unavailable.
- After tests pass, remove the inline `compute_basic_filter`, `put_filter`, and
`filter_header_cache` handling, and the inline per-block ZMQ publish, from
`apply_block_admitted` and `disconnect_block_admitted` in
`crates/node/src/apply.rs`. Reposition ZMQ as a best-effort projection driven
off committed transitions, not off inline apply.

Phase 5: Attach readiness, keep the enforcer external, hook Utreexo

Now that two real consumers exist and reconcile, expose readiness where clients query it, confirm the enforcer stays external, and let experimental Utreexo consume the same boundary. Mempool A/R events (issue #74) reuse this boundary later and are out of scope here.

Task 1: Expose consumer readiness through RPC

Report lifecycle state so clients do not trust an index that is rebuilding.

• Add a readiness capability to crates/rpc/src/context.rs following the Option<Arc<dyn Trait>> plus with_* builder plus status() getter pattern used for PruneService.
• Surface consumer lifecycle state in the relevant getblockchaininfo-style response fields in crates/rpc/src/handlers/, reusing the existing response-field pattern.

Task 2: Attach Electrum readiness to the txindex consumer state

Electrum must not claim complete history while the source index is Building.

• Add a readiness check at the Electrum dispatch chokepoint (dispatch() in crates/electrum/src/methods.rs), gated on the txindex consumer's lifecycle state.
• When the source index is Building or Stale, return an explicit partial or unavailable response rather than incomplete history. There is no readiness concept in Electrum today; this is net-new.

Task 3: Confirm the enforcer boundary

Keep the BIP300/301 enforcer external.

• Confirm the enforcer remains an external sidecar over JSON-RPC and ZMQ (tools/bip300301-enforcer), and that no apply lock is held during its I/O. Add a test or assertion that the enforcer path uses only the public surfaces. This is a no-op verification, not new coupling.

Task 4: Add the Utreexo consumer behind its feature flag

Let experimental Utreexo consume the boundary without becoming a default dependency.

• Add a Utreexo reconciliation consumer, behind the existing utreexo Cargo feature, that follows the txindex storage contract and reconciliation algorithm, and drives Bridge::ingest_block (crates/utreexo/src/bridge.rs) from active-chain connect transitions using the snapshot and body lookup.
• This is the first real coupling of the utreexo crate. Keep it strictly behind the feature and out of the default consensus path.

🤖 Prompt for AI agents

Expose consumer readiness through RPC and Electrum, verify the BIP300/301
enforcer stays external, and add the Utreexo hook behind its feature flag.
Mempool A/R events (issue #74) are out of scope here; they reuse this boundary
later.

- Add a readiness capability to `crates/rpc/src/context.rs` following the
`Option<Arc<dyn Trait>>` plus `with_*` builder plus `status()` getter pattern
used for `PruneService`. Surface consumer lifecycle state in the relevant
`getblockchaininfo`-style response fields in `crates/rpc/src/handlers/`, reusing
the existing response-field pattern.
- Add a readiness check at the Electrum dispatch chokepoint (`dispatch()` in
`crates/electrum/src/methods.rs`), gated on the txindex consumer's lifecycle
state. When the source index is `Building` or `Stale`, return an explicit
partial or unavailable response instead of incomplete history. This readiness
concept is net-new to Electrum.
- Confirm the enforcer remains an external sidecar over JSON-RPC and ZMQ
(`tools/bip300301-enforcer`), and that no apply lock is held during its I/O. Add
a test or assertion that the enforcer path uses only public surfaces. Treat this
as a no-op verification; do not introduce new coupling.
- Add a Utreexo consumer behind the existing `utreexo` Cargo feature, following
the txindex storage contract and reconciliation algorithm, driving
`Bridge::ingest_block` (`crates/utreexo/src/bridge.rs`) from active-chain
connect transitions using the snapshot and body lookup. Keep it strictly behind
the feature flag and out of the default consensus path.

Phase 6: Prove the cross-cutting acceptance gates

Prove the invariants that are not specific to one consumer.

Task 1: Prove default apply is unaffected by consumer state

The default full-node path must not wait for, trust, or fail because of a consumer.

• Add a gate following the g10/g11 subprocess-wrapper pattern with the loud-fail stdout guard.
• Prove default full-node apply stays correct and available when all consumers are disabled, paused, slow, or failed.
• Prove default IBD runs with no optional-index work on the apply critical path after the Phase 3 and Phase 4 removals.

Task 2: Prove ZMQ loss and reorg recovery at the boundary

Prove the defined recovery path and the ordering guarantee end-to-end.

• Prove that ZMQ event loss has a defined recovery path: the consumer resumes by reconciling against the snapshot, not from ZMQ.
• Prove connect/disconnect ordering across ordinary reorgs, invalidateblock-triggered reorgs, and restart boundaries, observed through consumer convergence rather than a durable event log.

🤖 Prompt for AI agents

Prove the cross-cutting invariants that are not specific to any single consumer.

- Add a gate following the `g10`/`g11` subprocess-wrapper pattern with the
loud-fail stdout guard. Prove default full-node apply stays correct and
available when all consumers are disabled, paused, slow, or failed. Prove
default IBD runs with no optional-index work on the apply critical path after
the Phase 3 and Phase 4 removals.
- Prove that ZMQ event loss has a defined recovery path: the consumer resumes by
reconciling against the snapshot, not from ZMQ. Prove connect/disconnect
ordering across ordinary reorgs, `invalidateblock`-triggered reorgs, and restart
boundaries, observed through consumer convergence rather than a durable event
log.

Research

crates/node/src/apply.rs is a dumping ground. apply_block_admitted and disconnect_block_admitted do consensus validation, UTXO commit, undo/body persistence, and tip publication. That's fine, that's their job. But they also run txindex ingest, filter build/write, mempool eviction, coinstats finish/rewind, and ZMQ publish inline, some fatal, some silently swallowed. That's the garbage this ticket exists to remove.

There is no ChainEvent type today. The only event mechanism is zmq_publisher.rs, fire-and-forget, backed by an in-memory AtomicU32 counter that resets on restart. It is not a recovery log. It must not become one.

The parts we build on already exist, and most of the recovery machinery is already there. We do not need a write-ahead log:

• crates/chain/src/tree.rs BlockTree already exposes active-chain queries: active_node_at_height, header_at_active_height, find_common_ancestor, height_of_hash, node_by_hash, iter_active_chain_hashes, plus tip accessors. crates/chain/src/reorg.rs plan_reorg computes the common ancestor plus disconnect/connect lists. Reuse these. Do not reimplement ancestry.
• Retained body lookup already verifies both height and hash: PruneBodyStore::load_block_body(height, hash) (crates/node/src/apply.rs) over FlatFileBlockStore::load (crates/storage/src/block_file.rs), which returns None on any mismatch. Undo and body keys are (height, hash) composites (crates/pruning/src/*_pruner.rs).
• The commit point is already per-block and strictly ordered. apply_window batches only script verification through WindowOverlay. The real mutation and tip store happen one block at a time in apply_block_admitted. SequenceEvent::Connected fires right after applied_tip.store(...). That is where a snapshot sequence advances and a hint fires.
• Reorg ordering (tip-first disconnect, then connect the replacement branch) is an established, documented decision in node-reorg-execution-design.md and CONCEPTS.md. Do not re-decide it.
• Bounded non-blocking notification is already an idiom: wake_sync uses bounded(1) plus try_send-and-drop (crates/p2p/src/listener.rs); INBOUND_BLOCK_CHANNEL_LIMIT bounds inbound blocks (crates/node/src/state.rs). Follow this pattern. Do not invent a new one.
• There is no process-restart epoch today. crash_recovery.rs and DisconnectMarker compare height/hash only. checkpoint.rs has a persisted generation: u64 with an allocate/read/CURRENT pattern that is reusable for an epoch counter.
• Config uses consistent enable flags (bool, Option<Addr>, Vec<String>) plus validate() cross-checks. Background workers follow spawn_electrum_listener: config-gated spawn, shared AtomicBool shutdown, joined after loop_handle.spin.
• RPC Context uses Option<Arc<dyn Trait>> plus with_* builder plus status() getter (PruneService, ChainControl). Readiness reporting extends this pattern.
• Row-key identity differs by product. FilterIndex keys are the 32-byte block hash, so filters are already reorg-safe. Txindex TxConfirmed/Funding/Spending keys are an 8-byte content prefix plus little-endian height with no block hash; rollback is guarded only by header-row presence, not by per-row hash equality. Txindex needs block-identified inverse metadata before it can reconcile reorgs on its own.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Build the core active-chain boundary: a coherent snapshot, ancestry/body lookup
accessors, and a bounded hint channel. Nothing consumes these yet. There is no
`ChainEvents` column family, no journal writer, no global durable cursor, and no
per-block durable event write. Do not add an fsync to the apply path.

- Add a snapshot value type carrying `epoch: u64`, `sequence: u64`, `tip_hash`,
and `tip_height`. Add an in-memory monotonic `sequence` counter, advanced once
per committed connect and once per committed disconnect, right after
`applied_tip.store(...)` in `apply_block_admitted` and
`disconnect_block_admitted` in `crates/node/src/apply.rs`, at the same point
`SequenceEvent` fires today. Add a process `epoch` assigned in `NodeState::open`
in `crates/node/src/state.rs`, using the `checkpoint.rs` `generation`/`CURRENT`
allocate-read pattern. Expose `NodeState::active_chain_snapshot()` returning a
coherent, non-torn read of all four fields. This is a live value, not a durable
event; do not persist a per-event record.
- Reuse `BlockTree` (`crates/chain/src/tree.rs`) for ancestry:
`active_node_at_height`, `header_at_active_height`, `find_common_ancestor`,
`height_of_hash`, `node_by_hash`, `iter_active_chain_hashes`. Do not reimplement
ancestry. Add a thread-safe read accessor on `NodeState` over the existing
`Arc<RwLock<BlockTree>>`. Add a helper answering whether `(height, hash)` is
still on the active chain. Reuse `plan_reorg` (`crates/chain/src/reorg.rs`) for
common-ancestor plus disconnect/connect lists.
- Reuse `PruneBodyStore::load_block_body(height, hash)`
(`crates/node/src/apply.rs`) over `FlatFileBlockStore::load`
(`crates/storage/src/block_file.rs`), already verifying height and hash and
returning `None` on mismatch. Reuse the `(height, hash)`-keyed undo store for
disconnect data. Add a consumer-facing accessor returning `None` when data is
pruned; that `None` is the signal for the consumer to go `Stale` and rebuild. Do
not extend retention to avoid it.
- Define a `ChainEvent` hint carrying `kind` (`Connected`/`Disconnected`),
`height`, `hash`, `epoch`, `sequence`. Deliver it over a bounded channel
following the `wake_sync` `bounded(N)` plus `try_send`-and-drop pattern
(`crates/p2p/src/listener.rs`) and the `INBOUND_BLOCK_CHANNEL_LIMIT` bound
(`crates/node/src/state.rs`). Emit the hint at the commit point with `try_send`,
right after the sequence advance. On a full queue, drop the hint; that is not a
bug. Hints only wake the consumer; they are not a source of truth and not a
recovery log. A dropped hint recovers via reconciliation against the snapshot
sequence.
- Write a design note under `docs/solutions/architecture-patterns/` describing
the snapshot, ancestry/body/undo lookups, the bounded hint channel, and the
tip-first-disconnect ordering the core already guarantees. State plainly that
neither the hint channel nor ZMQ is durable or replayable; recovery is
reconciliation, not replay. State the retention rule: pruning is a finite,
core-controlled policy independent of consumer progress; a lagging consumer goes
`Stale` and rebuilds; pruning is never extended or blocked for it. Add glossary
entries in `CONCEPTS.md` next to `Sequence stream`, `Commit point`, and `Undo
record`.
===============================================================================

Task: 2

Fix txindex rollback correctness before any consumer extraction happens. Do not
defer this to parity testing, where a reorg bug becomes silent data corruption.

- Audit row-key identity across derived-state products. Record that txindex
`TxConfirmed`/`Funding`/`Spending` keys are an 8-byte content prefix plus
little-endian height with no block hash (`crates/index/src/types.rs`,
`crates/index/src/index.rs`), and rollback is guarded only by header-row
presence, not by per-row hash equality; that is not enough for an independent
reorg-reconciling consumer. Record that `FilterIndex` rows are keyed by the
32-byte block hash (`crates/filters/src/filter_index.rs`) and are already
reorg-safe. Conclude that txindex needs block-identified inverse metadata before
Phase 3; filters do not.
- Add per-block inverse metadata keyed by `(height, hash)` listing exactly the
rows that block wrote. Rollback replays this metadata and deletes only that
block's rows, never a replacement block's rows. This inverse metadata is the
consumer's own undo data, written in the consumer-owned batch defined in Phase
3. If the row layout must change, treat it as a breaking change per
`docs/policies/db-migration.md`: keep `ColumnFamily` variants append-only, bump
the value version byte, reject unknown versions on read, and require a datadir
wipe and resync. There is no in-place migration.
===============================================================================

Task: 3

Build txindex as the first concrete reconciliation consumer, end-to-end, then
delete its inline path. Do not build a generic `ChainEventConsumer` trait,
registry, or extension-registration mechanism; that generalization is deferred
to a future ticket (#79).

- Define the txindex consumer storage contract: each reconciliation step commits
ONE atomic consumer-owned DB batch containing the txindex row mutations, the
block-identified inverse metadata from Phase 2, and the consumer cursor `(epoch,
sequence, height, hash)` plus lifecycle state. There is no shared
framework-level cursor store; the cursor lives with the consumer's own data.
Define the lifecycle state enum: `Disabled`, `Building`, `CaughtUp`, `Paused`,
`Failed`, `Stale`. Require idempotent connect and disconnect handling so replay
after interruption converges to the same rows.
- Implement the reconciliation algorithm: read a fresh coherent snapshot
`(epoch, sequence, tip)`. Trigger reconcile on epoch change, sequence gap,
dropped/overflowed hint, or tip mismatch with the cursor. Test whether the
cursor `(height, hash)` is still on the active chain. If yes, fast-forward:
connect each block from `cursor.height + 1` to the snapshot tip, reading each
body via the body-lookup accessor, committing each block in one atomic batch. If
the anchor is on a stale branch, find the common ancestor, roll back tip-first
from the cursor to the common ancestor using the consumer's own inverse
metadata, then connect forward to the snapshot tip. If a required body or
inverse record is unavailable (pruned), set state `Stale` and perform a full
rebuild from a named active-chain anchor. After reaching the snapshot tip,
re-read the snapshot; if unchanged, set state `CaughtUp`, otherwise repeat.
- Add `spawn_txindex_consumer(config, state, shutdown)` in
`crates/node/src/run.rs`, following `spawn_electrum_listener`: config-gated,
returns `Result<Option<JoinHandle>>`, loops while the shared `AtomicBool`
shutdown flag is unset. The worker waits on the bounded hint channel and also
polls the snapshot on a timer so a dropped hint cannot stall progress. On a
consumer error, set state `Failed` and stop only this worker; never propagate
into apply. Reuse `begin_batch`/`end_batch` (`crates/index/src/index.rs`) for
backfill throughput. Add a txindex-consumer enable flag to `Config` in
`crates/node/src/config.rs`, following the `txindex: bool` pattern, with any
needed `validate()` cross-field rule. Join the worker in `run()` after
`loop_handle.spin` returns, alongside the electrum and rpc joins, before
`shutdown::drain_and_shutdown`.
- Add tests in `crates/node/tests/` using the `tempdir` plus `NodeState::open` /
drop / reopen harness, and a gate following the `g10_reorg_deep.rs` /
`g11_crash_recovery.rs` subprocess-wrapper pattern with the loud-fail stdout
guard. Cover each of: stop/restart convergence, dropped-hint recovery via
snapshot reconciliation, epoch change and sequence gap triggering correct
reconcile, bounded queue overflow not losing correctness, ordinary reorg and
`invalidateblock`-triggered reorg reconciling tip-first disconnect then
replacement-branch connect, stale detection plus full rebuild when data is
unavailable, and final txindex rows identical to the inline implementation for
the same block sequence.
- After tests pass, remove the inline `tx_index.ingest_*` and `rollback_*` calls
from `apply_block_admitted` and `disconnect_block_admitted` in
`crates/node/src/apply.rs`. Keep consensus validation, UTXO commit, body/undo
persistence, applied-tip publication, the sequence advance, and hint emission;
those stay authoritative. Do not touch coinstats where checkpoint consistency
requires it, and do not break the checkpoint cross-check in
`crates/node/src/checkpoint.rs`.
===============================================================================

Task: 4

Migrate the filter index onto the same reconciliation shape proven with txindex.
Do not build a generic trait; copy the pattern, not an abstraction.

- Build the filter index reconciliation consumer using the same storage contract
shape as txindex: a consumer-owned atomic batch holding filter rows plus the
consumer cursor `(epoch, sequence, height, hash)` and state. Keep filter rows
hash-addressed and cached across disconnects; do not roll back filter rows. On
disconnect, rewind only the active filter-header pointer/cursor to the parent.
Maintain the BIP157 header chain from the consumer's own persisted cursor
instead of the single in-memory `filter_header_cache` slot. Implement backfill
so an index enabled late, or after a gap, repairs itself (the "backfill can
repair" case `apply.rs` documents but never implements). Reuse the
reconciliation algorithm defined for the txindex consumer.
- Prove filter correctness with tests: final filter rows and headers identical
to the inline implementation for the same block sequence; stop/restart
convergence, dropped-hint recovery, epoch change, sequence gap, and queue
overflow, as with txindex; reorg rewinds only the active filter-header pointer
while keeping hash-addressed rows; stale detection plus full rebuild when data
is unavailable.
- After tests pass, remove the inline `compute_basic_filter`, `put_filter`, and
`filter_header_cache` handling, and the inline per-block ZMQ publish, from
`apply_block_admitted` and `disconnect_block_admitted` in
`crates/node/src/apply.rs`. Reposition ZMQ as a best-effort projection driven
off committed transitions, not off inline apply.
===============================================================================

Task: 5

Expose consumer readiness through RPC and Electrum, verify the BIP300/301
enforcer stays external, and add the Utreexo hook behind its feature flag.
Mempool A/R events (issue #74) are out of scope here; they reuse this boundary
later.

- Add a readiness capability to `crates/rpc/src/context.rs` following the
`Option<Arc<dyn Trait>>` plus `with_*` builder plus `status()` getter pattern
used for `PruneService`. Surface consumer lifecycle state in the relevant
`getblockchaininfo`-style response fields in `crates/rpc/src/handlers/`, reusing
the existing response-field pattern.
- Add a readiness check at the Electrum dispatch chokepoint (`dispatch()` in
`crates/electrum/src/methods.rs`), gated on the txindex consumer's lifecycle
state. When the source index is `Building` or `Stale`, return an explicit
partial or unavailable response instead of incomplete history. This readiness
concept is net-new to Electrum.
- Confirm the enforcer remains an external sidecar over JSON-RPC and ZMQ
(`tools/bip300301-enforcer`), and that no apply lock is held during its I/O. Add
a test or assertion that the enforcer path uses only public surfaces. Treat this
as a no-op verification; do not introduce new coupling.
- Add a Utreexo consumer behind the existing `utreexo` Cargo feature, following
the txindex storage contract and reconciliation algorithm, driving
`Bridge::ingest_block` (`crates/utreexo/src/bridge.rs`) from active-chain
connect transitions using the snapshot and body lookup. Keep it strictly behind
the feature flag and out of the default consensus path.
===============================================================================

Task: 6

Prove the cross-cutting invariants that are not specific to any single consumer.

- Add a gate following the `g10`/`g11` subprocess-wrapper pattern with the
loud-fail stdout guard. Prove default full-node apply stays correct and
available when all consumers are disabled, paused, slow, or failed. Prove
default IBD runs with no optional-index work on the apply critical path after
the Phase 3 and Phase 4 removals.
- Prove that ZMQ event loss has a defined recovery path: the consumer resumes by
reconciling against the snapshot, not from ZMQ. Prove connect/disconnect
ordering across ordinary reorgs, `invalidateblock`-triggered reorgs, and restart
boundaries, observed through consumer convergence rather than a durable event
log.

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[gosunuts]

@coderabbitai Replan using the final design clarification in the latest maintainer comment as authoritative.

Do not create:

• a durable event journal or journal column family;
• a durable global event-sequence/cursor column family;
• per-block event persistence or fsync;
• startup journal replay; or
• pruning that is indefinitely gated by the slowest consumer.

Use bounded, non-blocking live ChainEvent delivery only as a low-latency hint. Missing events, an epoch change, a sequence gap, or queue overflow must cause reconciliation against a coherent authoritative active-chain snapshot.

The durable recovery anchor is the consumer-owned (height, hash), not the live event sequence. Each consumer must atomically persist, in its own storage boundary:

1. the derived-state mutation;
2. block-identified inverse/undo metadata sufficient to roll back old-branch state; and
3. the updated (height, hash) cursor.

Forward backfill comes from the active-chain view and retained block bodies. Old-branch rollback comes from consumer-owned inverse metadata. If the required inverse metadata or retained forward bodies are unavailable, only that consumer becomes stale and rebuilds; core sync, pruning, and other consumers continue.

Do not claim safe rollback by height alone. The current index contains non-identity-bearing keys, so the plan must first address block identity by validating the producing block, storing exact per-block inverse keys, or migrating the row layout.

Start by extracting exactly one concrete consumer, preferably txindex or the filter index, and prove stop/restart, missed-event recovery, reorg, queue overflow, stale/rebuild, and final row parity. Discover the minimal common lifecycle from that extraction before introducing a generic ChainEventConsumer framework; generic extension registration belongs to #79.

The later goals remain valid: remove optional index work from the apply critical path after parity coverage, expose consumer readiness to RPC/Electrum, keep hash-addressed filter rows across disconnects, keep BIP300/301 external, and ensure a failed/slow/disabled consumer never blocks core apply.

[gosunuts]

For the scope of this issue, assume the block bodies required for reconciliation and backfill remain available.
This issue does not define a new pruning or retention policy. Interaction between consumer backfill/recovery and pruning should be handled separately after the reconciliation boundary is proven.
In particular, #77 should not introduce consumer-driven pruning holds, retention leases, slowest-consumer retention rules, or stale thresholds.
The goal here is only to prove that independent consumers can converge across normal operation, missed notifications, restart, and reorg when their required source data is available.

[coderabbitai]

I've updated the plan above based on your feedback. Ready to proceed when you are!

[gosunuts]

Final scoped design for #77

This comment is the authoritative implementation direction for #77 and supersedes earlier comments or generated plans where they conflict with it.

The purpose of this issue is narrowly:

Prove that one independent derived-state consumer can converge to the authoritative active chain across normal operation, missed notifications, restart, and reorg, without blocking core apply.

#77 does not define a durable event log or complete the migration of every optional product.

Core boundary

Core owns:

• one coherent, immutable active-chain snapshot containing the applied tip and a process-local sequence;
• captured-tip ancestry lookup;
• retained block-body lookup by (height, hash); and
• a bounded, non-blocking wake hint.

The sequence is process-local and is not persisted. A persistent epoch, global durable event cursor, event journal, event column family, per-block event write, and startup journal replay are all explicitly excluded.

The hint is only a latency optimization. Apply publishes the new coherent snapshot after a completed connect or disconnect and calls try_send on a bounded channel. A full queue drops the hint. The consumer periodically compares its cursor with a fresh snapshot, so losing every hint can delay progress but cannot affect correctness.

The architecture requires only that consumer execution never run inline with apply, wait under the chain-transition lock, or propagate its failure into core. A dedicated OS thread is a suitable first implementation, not a permanent architectural requirement.

Durable consumer state

The first concrete consumer is the existing confirmed/electrs-shaped index currently called txindex. It includes the current TxConfirmed, Funding, Spending, and BlockHeaders responsibilities; splitting txid lookup from Electrum-oriented indexing is not part of #77.

Its only durable progress anchor is:

struct IndexCursor {
    height: u32,
    hash: BlockHash,
}

The cursor lives in index-owned metadata in the same txindex database. Process-local sequence or boot identity may be kept in memory for fast-path bookkeeping, but neither is part of durable recovery state.

Every cursor advance or rewind must be committed atomically with the corresponding index row mutations in one index-owned storage batch. The existing Indexer API therefore needs a cursor-aware connect/rollback commit path. Existing begin_batch/end_batch must not be used in a way that flushes rows separately from the cursor.

No new consumer undo

#77 does not add per-block inverse metadata or expose core UTXO undo to the consumer.

The existing index rollback already:

• deterministically reconstructs the txid/funding/spending/header keys from the old block body;
• uses the full 80-byte serialized header row as block identity;
• skips a repeated rollback when that old identity is gone; and
• deletes one block's rows in one atomic write batch.

The existing same-height replacement regression test proves that rolling an old block back again does not delete replacement rows. Under this issue's source-data precondition, both directions use block bodies:

connect:    active-branch block body -> reconstruct and insert rows
disconnect: old-branch block body    -> reconstruct and delete rows

The cursor-aware rollback path should additionally require the stored cursor to identify the block being removed and commit the parent cursor in the same delete batch. If the cursor identifies the block but its header identity row is absent, report index inconsistency rather than silently advancing the cursor.

Reconciliation

A worker starts by loading its durable (height, hash) cursor and capturing core snapshot S.

1. While the cursor is not on the branch ending at S.tip, load the cursor block body, roll its index rows back, and derive the parent cursor from that body's prev_blockhash.
2. Once the cursor is an ancestor of S.tip, obtain the captured target branch from the next height through S.tip and connect those bodies in order.
3. Capture a fresh snapshot. Report CaughtUp only when the durable cursor equals that fresh tip; otherwise repeat.

This intentionally does not require the old cursor tip to remain in the restored BlockTree. For example, if the index cursor was durably advanced to height 105 but a crash restores core to checkpoint height 100, the consumer walks bodies 105 down to 101 using their parent hashes, then converges from the restored authoritative tip. This same body walk handles an ordinary old-branch reorg.

A hint, timeout, startup, sequence gap, or observed tip mismatch can all trigger this same reconciliation procedure. None is a replay cursor.

Source-data precondition

For #77, assume every block body required for reconciliation and historical backfill is available.

This issue does not define:

• pruning or event retention;
• consumer-driven pruning holds or leases;
• a slowest-consumer retention rule;
• stale thresholds; or
• full-rebuild policy when source data is unavailable.

A missing required body may fail only this consumer with a source-unavailable error. Its retention and rebuild policy belongs in a separate follow-up issue.

Minimal readiness safety

After inline indexing is removed, an index that is still building must not return incomplete history as if it were complete. #77 includes only the minimum safety gate:

• CaughtUp: txindex-dependent queries may run;
• Building or Failed: return an explicit index-unavailable result.

Richer RPC status fields, Electrum readiness behavior, and partial-history policy remain follow-up work.

Implementation phases

Phase 1 — Minimal reconciliation boundary

• coherent active-chain snapshot;
• process-local sequence;
• captured-tip ancestry lookup;
• retained block-body lookup;
• bounded non-blocking wake hint;
• ordering and missed-hint race tests;
• design note and reconciled project terminology.

Phase 2 — One concrete index consumer

• durable index-owned (height, hash) cursor;
• cursor-aware atomic connect and rollback batches;
• existing block-body reconstruction and header identity guard;
• startup and periodic reconciliation;
• tests for stop/restart, consumer-ahead-of-restored-core, missed hints, queue overflow, ordinary reorg, invalidateblock reorg, and final-row parity with uninterrupted inline processing.

Phase 3 — Remove inline txindex work

• remove current Indexer ingest from connect apply;
• remove current Indexer rollback from core disconnect;
• add the minimum query-readiness gate;
• prove disabled, slow, or failed index consumption does not affect core apply;
• prove no txindex database work remains on the apply/IBD critical path.

Acceptance criteria

• No durable ChainEvent WAL/journal, global durable event cursor, persistent epoch, or per-block event fsync is introduced.
• Snapshot publication is coherent and hint delivery is bounded and non-blocking.
• The consumer's only durable progress anchor is index-owned (height, hash).
• Index row mutations and cursor movement commit in the same storage batch.
• Connect and rollback both reconstruct rows from retained block bodies; no new consumer inverse store or core-undo API is introduced.
• Stop/restart, a cursor ahead of restored core, missed hints, queue overflow, ordinary reorg, and invalidation-triggered reorg converge to the same final rows as uninterrupted processing.
• Incomplete index state is not served as complete.
• Core apply remains correct and available when this consumer is disabled, slow, or failed.
• Txindex work is absent from the apply/IBD critical path.
• Pruning, retention, missing-source rebuild, and other consumer migrations are not implemented in this issue.

Follow-ups

• Filter backfill/migration and richer Electrum readiness: Add resumable post-IBD backfill for txindex, Electrum, and block filters #75
• Utreexo mode: Implement an experimental Utreexo compact-validator node mode #76
• Generic extension registration and lifecycle: Define a static opt-in extension model and workspace extension API #79
• Mempool A/R semantics: Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74
• Pruning/retention/stale/rebuild interaction: a separate issue
• BIP300/301 enforcer verification, ZMQ repositioning, and physical txindex/Electrum-index separation: separate follow-up work

This keeps #77 as a concrete proof of the reconciliation boundary rather than an event-log subsystem or a multi-product migration epic.