Complete BIP300/301 enforcer integration with mempool A/R pubsequence events

[gosunuts]

Problem

The checked-in BIP300/301 Compose integration currently runs the upstream enforcer in block-only mode. bitcoin-rs publishes Core-compatible pubsequence block connect/disconnect events (C/D), but deliberately omits mempool add/remove events (A/R). As a result, Compose cannot safely enable the enforcer's mempool mode.

The missing work is not just adding two ZMQ labels. Mempool mutations currently occur through multiple direct RwLock<Mempool> call sites, while the pool exposes only a coarse mutation counter and removal paths do not retain a complete, ordered event record with explicit causes. Publishing from selected callers would permit missing, duplicated, or reordered events.

Current state

• crates/node/src/zmq_publisher.rs implements the sequence topic and topic-local u32 transport counter for block C/D events.
• crates/mempool/src/pool.rs maintains a u64 change counter that increments on successful insert/remove operations.
• RPC, wallet, mining, node apply, and other paths can acquire the shared mempool write lock directly.
• Block connection removes confirmed transactions and descendants from the mempool in crates/node/src/apply.rs.
• tools/bip300301-enforcer/docker-compose.yaml deliberately omits --enable-mempool.
• docs/rest-interface.md and CONCEPTS.md document this limitation.

Goal

Provide a single ordered mempool-mutation boundary that can emit Bitcoin Core-compatible pubsequence A/R events without gaps or ordering ambiguity, then enable and validate the upstream enforcer's mempool mode.

Proposed work

1. Freeze compatibility semantics

Before implementation, ground the exact Bitcoin Core and pinned upstream enforcer wire contract:

• payload layout for A and R
• relationship between the ZMQ topic sequence and mempool sequence
• byte order and counter widths
• ordering for parent/descendant removals
• behavior for transactions confirmed in a block
• behavior for conflicts, replacement, expiry, size/fee eviction, explicit removal, clear, reorg, and re-admission
• expected recovery behavior after subscriber or publisher event loss

Record the chosen semantics in tests and project documentation.

2. Introduce a unified mutation result/event model

Represent each successful mutation as an ordered set of transaction-level changes, for example:

• accepted transaction: txid plus assigned mempool sequence
• removed transaction: txid, assigned mempool sequence, and an explicit internal removal reason

Removal reasons should distinguish at least:

• block inclusion
• conflict or replacement
• descendant removal
• fee/size policy eviction
• explicit removal or wholesale reset
• reorg-related removal/re-admission where applicable

The internal reason model may be richer than the ZMQ wire format, but it must be available to other validated-chain consumers and tests.

3. Centralize mempool mutation

Route every production insert/remove/clear path through one mutation gateway rather than publishing opportunistically from individual callers.

Audit at least:

• transaction RPC admission
• P2P transaction admission
• wallet-originated admission
• mining/policy eviction
• block-connect eviction
• reorg reconsideration
• shutdown/reload/clear behavior

State mutation and event ordering must be serialized. ZMQ remains best-effort and must not roll back a valid mempool mutation, but later publishers must not overtake earlier mutations.

4. Extend pubsequence

Extend SequenceEvent and all publisher implementations with Core-compatible transaction A/R payloads.

Preserve:

• existing block C/D behavior
• one monotonically ordered topic-local transport sequence
• reorg ordering: tip-first disconnects before replacement-branch connects
• no external I/O while holding consensus-critical or mempool locks longer than necessary
• no consensus/apply failure caused by notification failure

5. Enable the enforcer mempool mode

After compatibility and end-to-end tests pass:

• add --enable-mempool to the checked-in enforcer Compose command
• validate startup and synchronization against the pinned ENFORCER_REVISION
• update docs/rest-interface.md, README.md, overlapping docs/solutions/ knowledge, and CONCEPTS.md

Acceptance criteria

• The exact Core/enforcer A/R payload and sequencing contract is captured in compatibility tests.
• Every successful production mempool insertion emits exactly one ordered A event.
• Every transaction removed from the mempool emits exactly one ordered R event, including descendants.
• Failed or no-op mutations emit no transaction event and consume no mempool event sequence.
• Concurrent admission/removal cannot reorder publication relative to the committed mutation order.
• Block connection emits the required R events without changing existing block C semantics.
• Reorg tests cover disconnect, transaction reconsideration/re-admission, and replacement-branch connection.
• Conflict/replacement, policy eviction, explicit removal, and clear/reset behavior are tested.
• Topic sequence wraparound behavior is defined and tested.
• ZMQ failure or a disconnected subscriber cannot fail consensus validation or corrupt mempool state.
• The pinned upstream enforcer runs with --enable-mempool and follows an end-to-end transaction lifecycle.
• Existing block-only consumers remain compatible.
• Documentation and durable project terminology are reconciled.

Non-goals

• Making the external enforcer a synchronous prerequisite for ordinary Bitcoin block application.
• Calling enforcer gRPC while holding the chain-transition/apply lock.
• Treating BIP300/301 rejection as native Bitcoin consensus failure.
• Changing the default Bitcoin-only deployment path.
• Implementing a fail-open/fail-closed external consensus policy in this issue.

Follow-up design: optional enforcer-backed block policy

If bitcoin-rs should later refuse a branch based on BIP300/301 validation, track that as a separate architecture issue. It requires a pre-apply candidate-block API, deterministic connect/disconnect state transitions, crash/restart reconciliation, timeout and fail-open/fail-closed policy, explicit header-status semantics, and IBD performance analysis. The current post-apply RPC/REST/ZMQ sidecar topology is not sufficient for an atomic consensus gate.

[coderabbitai]

🔗 Related PRs

#14 - feat: header timestamp consensus rules, index reorg rollback, mempool policy, fuzz targets [merged]
#30 - perf(consensus): batch Merkle hashing with AVX2 [merged]
#49 - rpc: Core-compatible REST gateway so a remote bip300301_enforcer can use bitcoin-rs as its mainchain node [merged]
#52 - node: publish Core's sequence ZMQ topic for block connects and disconnects [merged]
#73 - Feature/drynet sync [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[gosunuts]

Related architecture issue: #77 defines the durable consumer/event model. The A/R pubsequence work here should use that model for recovery semantics; ZMQ remains the best-effort projection rather than the durable source.

[coderabbitai]

Coding Plan

Summary

• Freeze the exact Core A/R wire contract first, then define RemovalReason/MutationResult and a per-event mempool sequence in crates/mempool. No opportunistic publishing.
• Force every production insert/remove/clear through one MempoolGateway that serializes sequence assignment with state mutation and preserves publication order with a nested publish mutex, keeping all ZMQ I/O off the mempool lock.
• Extend SequenceEvent with A/R and drive them from a node-side observer that maps removal reasons to the wire, suppressing R for block inclusion to match Core, and leaving C/D untouched.
• Enable --enable-mempool, validate against the pinned enforcer, reconcile the docs, and prove correctness with byte-level, ordering, reorg, and failure-isolation tests.

Design Choices

Design Choice 1: Does block-inclusion removal emit an R event?

Options Considered:

1. Emit R for every mempool removal during block connect, including block-included transactions.
2. Follow Core: suppress R for the block-inclusion reason, emit R only for conflict/replacement removals triggered during block connect, and preserve C semantics.

Chosen Option: 2

Rationale: The acceptance criteria say block connection emits the required R events, but that contradicts how Bitcoin Core's sequence topic actually works. Core does not emit R for transactions removed because a block included them. The C event already covers that case. The entire point of this ticket is talking to the unmodified pinned enforcer, so match Core's real wire behavior, not the ticket's loose wording. Confirm this against Core source and the pinned ENFORCER_REVISION before any code lands.

Design Choice 2: Where does the mutation gateway live, given crate dependency direction?

Options Considered:

1. Put the gateway in crates/node and hand Arc handles down. This forces rpc/electrum to reach a node type they cannot import.
2. Put the gateway plus a MempoolObserver trait in crates/mempool. node supplies a concrete observer that adapts to ZmqPublisher. rpc/electrum call the gateway and never touch ZMQ.

Chosen Option: 2

Rationale: ZmqPublisher lives in crates/node, but crates/rpc and crates/electrum cannot depend on node because node depends on them. Putting the gateway and observer trait in crates/mempool respects the existing dependency graph and keeps the reason/event model in the crate that owns the state.

Design Choice 3: How much admission wiring is in scope?

Options Considered:

1. Build P2P/wallet transaction relay and admission from scratch inside this ticket.
2. Route only the existing production mutation sites through the gateway, and record the missing admission paths as a documented, gateway-ready gap.

Chosen Option: 2

Rationale: P2P transaction admission and wallet-originated admission do not exist today. Building a full P2P relay admission pipeline is a large separate feature that the codebase explicitly defers. The acceptance criteria are about correct ordered events for real mutations, not about inventing a relay subsystem. That would be scope creep and would not be reviewable. The gateway must be the single entry point so future relay work has exactly one place to plug in.

Design Choice 4: How is publication order guaranteed under concurrent mutations?

Options Considered:

1. Publish inline while holding the mempool write lock (simplest, but violates the rule of no I/O under the mempool lock).
2. Enqueue under lock and drain from a background publisher thread (correct, but adds thread lifecycle and shutdown complexity).
3. Assign sequence under the mempool write lock, acquire a dedicated publish mutex before releasing the write lock, release the write lock, then do ZMQ I/O under the publish mutex only.

Chosen Option: 3

Rationale: Sequence assignment must be serialized with state mutation, but ZMQ I/O must not sit under the mempool RwLock. Acquiring the publish mutex before dropping the write lock forces publish-mutex acquisition order to equal commit order, so publication cannot overtake an earlier mutation. ZMQ I/O never runs under the mempool RwLock, and no extra thread is needed.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Freeze the wire contract and define the mutation/event model

This phase produces the ground truth. Do not write publisher code before the contract is written down and the internal types exist. Every later phase depends on these definitions.

Task 1: Freeze and document the Core/enforcer A/R contract

Write down the exact wire behavior and lock it into tests and docs before touching mutation code.

• Record the payload layout confirmed against Core doc/zmq.md: A body is reversed_txid(32) || 0x41 || mempool_sequence(u64 LE); R body is reversed_txid(32) || 0x52 || mempool_sequence(u64 LE). C/D bodies stay reversed_hash(32) || 0x43|0x44 with no 8-byte field.
• Record that the trailing 4-byte little-endian transport counter is a separate third ZMQ frame shared across all sequence events, distinct from the 8-byte mempool sequence.
• Record the block-inclusion decision from Assumption 1: no R for block-included transactions; C semantics unchanged; conflict/replacement removals during block connect do emit R.
• Record the reorg ordering rule already guaranteed by switch_to_branch: tip-first disconnects, then replacement-branch connects.
• Record recovery expectations: A/R are best-effort; the transport counter lets a subscriber detect a gap; the enforcer reconciles a detected gap through RPC, not through mempool rollback.
• Capture all of the above in a new docs/solutions/architecture-patterns/ note and as named constants/comments in a compatibility test module.

Task 2: Introduce the removal-reason and mutation-result types

Add the internal vocabulary for what changed and why. This is a net-new addition; no such enum exists today.

• Add RemovalReason to crates/mempool/ with variants covering at least: BlockInclusion, Conflict/Replaced, Descendant, PolicyEviction (fee/size), Expiry, Explicit, Clear, and Reorg.
• Add MutationChange (a txid, the assigned mempool sequence, and either "accepted" or a RemovalReason) and MutationResult (an ordered Vec<MutationChange> describing one committed mutation).
• Keep the reason model richer than the wire format. The wire mapping (which reasons emit R, which suppress it) lives in the node observer, not here.

Task 3: Add the per-event mempool sequence counter

Replace the coarse "one bump per call" semantics with Core's per-event sequence, and reconcile existing consumers.

• Add a monotonic mempool-event sequence to Mempool that advances once per emitted change (per accepted transaction, per removed transaction), assigned under the write lock.
• Assign a sequence only for successful, non-empty changes. A failed insert, a no-op remove, or a clear on an empty pool must assign nothing.
• Reconcile the existing sequence_number() getter used by crates/rpc/src/handlers/mempool.rs (getmempoolinfo, getrawmempool) so mempool_sequence reports this same per-event counter, matching Core's meaning of that field.

🤖 Prompt for AI agents

Implement Phase 1: freeze the Core/enforcer A/R wire contract and build the
internal mutation vocabulary. Do this before writing any publisher code — every
later phase depends on it.

Document the contract:
- Confirm the payload layout against Core `doc/zmq.md`: `A` body is
`reversed_txid(32) || 0x41 || mempool_sequence(u64 LE)`; `R` body is
`reversed_txid(32) || 0x52 || mempool_sequence(u64 LE)`. `C`/`D` bodies stay
`reversed_hash(32) || 0x43|0x44` with no 8-byte field.
- Note that the trailing 4-byte little-endian transport counter is a separate
third ZMQ frame shared across all sequence events, distinct from the 8-byte
mempool sequence.
- Lock in the block-inclusion decision: no `R` for block-included transactions,
`C` semantics unchanged, and conflict/replacement removals during block connect
do emit `R`.
- Record the reorg ordering rule guaranteed by `switch_to_branch`: tip-first
disconnects, then replacement-branch connects.
- Record recovery expectations: `A`/`R` are best-effort; the transport counter
lets a subscriber detect a gap; the enforcer reconciles a detected gap through
RPC, not through mempool rollback.
- Write all of this into a new note under
`docs/solutions/architecture-patterns/` and as named constants/comments in a
compatibility test module.

Add the mutation vocabulary in `crates/mempool/`:
- Add `RemovalReason` with variants covering at least `BlockInclusion`,
`Conflict`/`Replaced`, `Descendant`, `PolicyEviction` (fee/size), `Expiry`,
`Explicit`, `Clear`, and `Reorg`.
- Add `MutationChange` (txid, assigned mempool sequence, and either "accepted"
or a `RemovalReason`).
- Add `MutationResult` as an ordered `Vec<MutationChange>` describing one
committed mutation.
- Keep the reason model richer than the wire format; the wire mapping lives in
the node observer later, not in this crate.

Add the per-event sequence counter:
- Add a monotonic mempool-event sequence to `Mempool` that advances once per
emitted change (per accepted transaction, per removed transaction), assigned
under the write lock.
- Assign a sequence only for successful, non-empty changes — no sequence for a
failed insert, a no-op remove, or a `clear` on an empty pool.
- Reconcile `sequence_number()` (used by `crates/rpc/src/handlers/mempool.rs`
for `getmempoolinfo`/`getrawmempool`) so `mempool_sequence` reports this same
per-event counter.

Phase 2: Centralize mutation behind one gateway

This phase removes the scattered write locks. After it, no production code outside the gateway mutates the pool. This is the structural core of the ticket.

Task 1: Make mutation methods return MutationResult

Stop returning bare Vec<EntryId>/Vec<Txid>. Every mutation must report its ordered changes with reasons.

• Update insert_entry, remove_entry_and_descendants, remove_by_txid, enforce_size_limit, evict_below_fee_rate, and clear in crates/mempool/src/pool.rs to build and return a MutationResult.
• Update replace_transaction/check_replacement in crates/mempool/src/rbf.rs to tag direct conflicts as Replaced and swept descendants as Descendant, instead of discarding the eviction return value.
• For insert_entry, the returned MutationResult must include the accepted transaction and any transactions the automatic post-insert enforce_size_limit evicts, in commit order, each with PolicyEviction.
• Ensure a package removal reports the parent removal before its descendants, so the emitted order is deterministic.

Task 2: Build the MempoolGateway and MempoolObserver trait

Introduce the single boundary and its ordering discipline.

• In crates/mempool/, add MempoolObserver (a trait with one method that receives a committed MutationResult) and MempoolGateway, which owns Arc<RwLock<Mempool>>, an optional Arc<dyn MempoolObserver>, and a dedicated publish mutex.
• Implement the ordering discipline from Assumption 4 for every mutating gateway method: take the mempool write lock, mutate, assign sequences, acquire the publish mutex, drop the write lock, hand the MutationResult to the observer under the publish mutex, then release the publish mutex.
• Expose read access (a read guard or with_read helper) so lookup callers keep working without touching the raw lock.
• Keep the observer call best-effort by contract. The gateway must never let an observer error affect the already-committed mutation.

Task 3: Route all production mutation sites through the gateway

Replace every direct write lock with a gateway call and audit the rest.

• Update crates/node/src/apply.rs block-connect removal (the remove_by_txid loop) to call the gateway with RemovalReason::BlockInclusion. Keep it after the mempool lock scope, consistent with the existing publish placement.
• Update crates/rpc/src/handlers/mempool.rs clearmempool and crates/rpc/src/handlers/mining.rs prioritisetransaction to go through the gateway. prioritise changes priority only, so it produces no A/R change.
• Update crates/electrum/src/methods.rs MempoolHandle so blockchain.transaction.broadcast inserts through the gateway instead of pool.write().insert_entry.
• Give crates/rpc/src/context.rs, crates/node/src/apply.rs ApplyHandles, and crates/electrum the shared Arc<MempoolGateway> in place of the raw Arc<RwLock<Mempool>>.
• Document, in code comments and the Phase 1 note, that P2P Message::Tx admission and wallet-originated admission do not exist yet and must enter through this gateway when they are built. Do not build them here.

🤖 Prompt for AI agents

Implement Phase 2: eliminate every scattered raw write lock and force all
production mutation through one `MempoolGateway`. After this phase, nothing
outside the gateway may mutate the pool.

Change mutation methods to report ordered changes:
- Update `insert_entry`, `remove_entry_and_descendants`, `remove_by_txid`,
`enforce_size_limit`, `evict_below_fee_rate`, and `clear` in
`crates/mempool/src/pool.rs` to build and return `MutationResult` instead of
bare `Vec<EntryId>`/`Vec<Txid>`.
- Update `replace_transaction`/`check_replacement` in
`crates/mempool/src/rbf.rs` to tag direct conflicts as `Replaced` and swept
descendants as `Descendant`.
- For `insert_entry`, include in the returned `MutationResult` both the accepted
transaction and any transactions the automatic post-insert `enforce_size_limit`
evicts, in commit order, each tagged `PolicyEviction`.
- Guarantee package removal reports the parent removal before its descendants,
so emitted order is deterministic.

Build the gateway and observer trait in `crates/mempool/`:
- Add `MempoolObserver`, a trait with one method that receives a committed
`MutationResult`.
- Add `MempoolGateway`, owning `Arc<RwLock<Mempool>>`, an optional `Arc<dyn
MempoolObserver>`, and a dedicated `publish` mutex.
- For every mutating gateway method, follow this exact ordering: take the
mempool write lock, mutate, assign sequences, acquire the `publish` mutex, drop
the write lock, hand the `MutationResult` to the observer under the `publish`
mutex, then release the `publish` mutex.
- Expose a read guard or `with_read` helper for lookup callers, so they never
touch the raw lock.
- Make the observer call best-effort by contract: an observer error must never
affect an already-committed mutation.

Route every production mutation site through the gateway:
- `crates/node/src/apply.rs`: route the block-connect removal (`remove_by_txid`
loop) through the gateway with `RemovalReason::BlockInclusion`, keeping it after
the mempool lock scope, consistent with the existing publish placement.
- `crates/rpc/src/handlers/mempool.rs` `clearmempool` and
`crates/rpc/src/handlers/mining.rs` `prioritisetransaction`: route through the
gateway. Note that `prioritise` only changes priority, so it must produce no
`A`/`R` change.
- `crates/electrum/src/methods.rs` `MempoolHandle`: route
`blockchain.transaction.broadcast` inserts through the gateway instead of
`pool.write().insert_entry`.
- Replace the raw `Arc<RwLock<Mempool>>` with a shared `Arc<MempoolGateway>` in
`crates/rpc/src/context.rs`, `crates/node/src/apply.rs` `ApplyHandles`, and
`crates/electrum`.
- Add code comments (and update the Phase 1 note) stating that P2P `Message::Tx`
admission and wallet-originated admission do not exist yet, and must enter
through this gateway when built. Do not build them in this phase.

Phase 3: Extend pubsequence and wire the ZMQ observer

This phase turns committed mutations into wire events. Block behavior must not change.

Task 1: Extend SequenceEvent with A/R

Add the two transaction variants and their payload builder.

• Extend SequenceEvent in crates/node/src/zmq_publisher.rs with Added(Txid, u64) and Removed(Txid, u64), with labels 0x41/0x52.
• Add a payload builder that appends the 8-byte little-endian mempool sequence for A/R, and leaves C/D payloads byte-for-byte unchanged.
• Reuse the existing ZmqTopic::Sequence transport counter for A/R; do not add a new counter slot. The 8-byte mempool sequence is separate from that transport counter.

Task 2: Implement the node-side ZMQ observer

Adapt MutationResult into ordered publisher calls.

• Add a MempoolObserver implementation in crates/node that holds the Arc<dyn ZmqPublisher> and, for each MutationChange in order, calls publish_sequence with Added or Removed.
• Apply the wire mapping: RemovalReason::BlockInclusion produces no R (per Assumption 1); all other removal reasons produce R. Accepted changes always produce A.
• Wire this observer into the gateway during node startup in crates/node/src/state.rs/run.rs, next to the existing SocketZmqPublisher/NoOpZmqPublisher selection. When no sequence endpoint is configured, install a no-op observer.

Task 3: Preserve ordering, lock, and failure guarantees

Verify the non-negotiable invariants hold across apply and reorg.

• Confirm reorg emits per-block D then C in the existing switch_to_branch order, and that any R events from conflict removal interleave correctly relative to the block events.
• Confirm no ZMQ I/O runs while the mempool RwLock is held; the observer runs only under the publish mutex after the write lock is dropped.
• Confirm a ZMQ send failure or absent subscriber cannot return an error into apply, reorg, or the gateway commit path. Notification failure stays best-effort.

🤖 Prompt for AI agents

Implement Phase 3: turn committed `MutationResult`s into wire events. Existing
block-event behavior (`C`/`D`) must not change one bit.

Extend the wire model in `crates/node/src/zmq_publisher.rs`:
- Add `Added(Txid, u64)` and `Removed(Txid, u64)` variants to `SequenceEvent`,
with labels `0x41`/`0x52`.
- Add a payload builder that appends the 8-byte little-endian mempool sequence
for `A`/`R`, and leaves `C`/`D` payloads byte-for-byte unchanged.
- Reuse the existing `ZmqTopic::Sequence` transport counter for `A`/`R`. Do not
add a new counter slot. The 8-byte mempool sequence is a separate value from
that transport counter.

Implement the node-side observer in `crates/node`:
- Add a `MempoolObserver` implementation holding `Arc<dyn ZmqPublisher>`. For
each `MutationChange` in order, call `publish_sequence` with `Added` or
`Removed`.
- Apply the wire mapping: `RemovalReason::BlockInclusion` produces no `R` event;
every other removal reason produces `R`. Accepted changes always produce `A`.
- Wire this observer into the gateway during node startup in
`crates/node/src/state.rs`/`run.rs`, next to the existing
`SocketZmqPublisher`/`NoOpZmqPublisher` selection. Install a no-op observer when
no `sequence` endpoint is configured.

Verify the invariants hold:
- Confirm reorg still emits per-block `D` then `C` in the existing
`switch_to_branch` order, and that any `R` events from conflict removal
interleave correctly relative to the block events.
- Confirm no ZMQ I/O runs while the mempool `RwLock` is held. The observer must
run only under the `publish` mutex, after the write lock is dropped.
- Confirm a ZMQ send failure or an absent subscriber cannot propagate an error
into apply, reorg, or the gateway commit path. Notification failure must stay
best-effort, always.

Phase 4: Enable the enforcer mempool mode and reconcile documentation

This phase flips the switch and fixes the docs that currently say the switch is off on purpose.

Task 1: Enable --enable-mempool in Compose

Turn on mempool mode and validate against the pinned revision.

• Add --enable-mempool to the enforcer command: in tools/bip300301-enforcer/docker-compose.yaml, keeping the existing --node-zmq-addr-sequence wiring and the pinned ENFORCER_REVISION.
• Validate enforcer startup and synchronization against the pinned revision. If the pinned revision cannot consume the frozen contract, record the required revision bump rather than silently changing wire behavior.

Task 2: Reconcile documentation and terminology

Update every doc that documents the old limitation.

• Update docs/rest-interface.md to state that pubsequence now publishes mempool A/R and that the enforcer runs with --enable-mempool. Remove the "block-only mode" and "deliberately omits --enable-mempool" language.
• Update CONCEPTS.md "Sequence stream" and reorg sections to describe per-transaction sequence assignment and explicit removal reasons.
• Update README.md and the overlapping docs/solutions/architecture-patterns/ notes (including the reorg-execution design's "Open" list) to move mempool sequence events and reconsideration status.

🤖 Prompt for AI agents

Implement Phase 4: flip the enforcer's mempool mode on, and fix every doc that
still claims it is deliberately off.

Enable mempool mode in Compose:
- Add `--enable-mempool` to the enforcer `command:` in
`tools/bip300301-enforcer/docker-compose.yaml`, keeping the existing
`--node-zmq-addr-sequence` wiring and the pinned `ENFORCER_REVISION` untouched.
- Validate enforcer startup and synchronization against the pinned revision. If
the pinned revision cannot consume the frozen contract, record the required
revision bump instead of silently altering wire behavior.

Reconcile the documentation:
- Update `docs/rest-interface.md`: state that `pubsequence` now publishes
mempool `A`/`R`, and that the enforcer runs with `--enable-mempool`. Remove the
"block-only mode" and "deliberately omits `--enable-mempool`" language.
- Update `CONCEPTS.md`'s "Sequence stream" and reorg sections to describe
per-transaction sequence assignment and explicit removal reasons.
- Update `README.md` and the overlapping `docs/solutions/architecture-patterns/`
notes (including the reorg-execution design's "Open" list) to move mempool
sequence events and reconsideration status out of the open/pending category.

Phase 5: Compatibility and behavior tests

This phase is the acceptance evidence. Follow the existing RecordingSequencePublisher pattern in crates/node/src/apply.rs tests.

Task 1: Payload and sequence compatibility tests

Assert exact bytes.

• Add byte-level tests for A and R payloads (32-byte reversed txid, correct label, 8-byte LE mempool sequence), alongside the existing sequence_payload tests.
• Assert C/D payloads are unchanged.
• Add a defined and tested behavior for transport-counter u32 wraparound.

Task 2: Ordering and reason tests

Assert one event per change, in commit order, for every removal cause.

• Add a recording MempoolObserver (a Vec of captured (id, label, sequence, reason) tuples) mirroring RecordingSequencePublisher.
• Add tests for: single accept emits one A; package removal emits R for parent then each descendant; conflict/replacement emits R for direct conflicts and descendants; fee/size eviction emits R; clear emits R for all entries; block connect emits no R for block-included transactions but does for conflicts.
• Add reorg tests covering disconnect, transaction reconsideration/re-admission where applicable, and replacement-branch connect ordering.
• Assert failed inserts and no-op removes emit nothing and consume no sequence.

Task 3: Failure isolation and end-to-end gate

Prove notification failure is harmless and document the live validation.

• Add a test where the observer/publisher fails and assert the mempool mutation still commits and apply/reorg still succeed.
• Add or extend a gate under bin/bitcoin-rs/tests/gates/ following the existing loud-placeholder convention, documenting the end-to-end enforcer --enable-mempool transaction-lifecycle validation against the pinned revision.

🤖 Prompt for AI agents

Implement Phase 5: build the acceptance evidence for everything done in Phases 1
through 4. Follow the existing `RecordingSequencePublisher` pattern already used
in `crates/node/src/apply.rs` tests.

Payload and sequence compatibility tests:
- Add byte-level tests for `A` and `R` payloads: 32-byte reversed txid, correct
label, 8-byte LE mempool sequence. Place these alongside the existing
`sequence_payload` tests.
- Assert `C`/`D` payloads remain byte-for-byte unchanged.
- Add a defined and tested behavior for transport-counter `u32` wraparound.

Ordering and reason tests:
- Add a recording `MempoolObserver` that captures `(id, label, sequence,
reason)` tuples in a `Vec`, mirroring `RecordingSequencePublisher`.
- Cover: single accept emits exactly one `A`; package removal emits `R` for the
parent then each descendant, in that order; conflict/replacement emits `R` for
direct conflicts and descendants; fee/size eviction emits `R`; `clear` emits `R`
for every entry; block connect emits no `R` for block-included transactions but
does emit `R` for conflicts.
- Add reorg tests covering disconnect, transaction reconsideration/re-admission
where applicable, and replacement-branch connect ordering.
- Assert failed inserts and no-op removes emit nothing and consume no sequence
number.

Failure isolation and end-to-end gate:
- Add a test where the observer/publisher fails, and assert the mempool mutation
still commits, and apply/reorg still succeed regardless.
- Add or extend a gate under `bin/bitcoin-rs/tests/gates/`, following the
existing loud-placeholder convention, documenting the end-to-end enforcer
`--enable-mempool` transaction-lifecycle validation against the pinned revision.

Research

This is a Rust workspace, and frankly the mempool code is a mess of scattered locks. crates/mempool owns the real Mempool (entries, ancestor/descendant sets, eviction, RBF, orphan pool) and a coarse sequence: AtomicU64 that gets bumped once per mutating call — that's a lazy counter, not a real event log.

crates/node owns apply.rs (block connect/disconnect, the only production mempool removal via remove_by_txid), zmq_publisher.rs (the SequenceEvent::{Connected,Disconnected} model, C/D labels, a topic-local AtomicU32 transport counter, and the ZmqPublisher trait), and reorg.rs (switch_to_branch with tip-first disconnect ordering under ChainTransition).

crates/rpc and crates/electrum hold their own Arc<RwLock<Mempool>> clones and mutate directly — no gateway, no discipline, everybody just grabs the lock and does whatever they want.

tools/bip300301-enforcer/docker-compose.yaml runs the pinned enforcer in block-only mode. docs/rest-interface.md and CONCEPTS.md document the deliberate absence of A/R.

Two facts drive the design, and both are ugly:

First, there is no lock-wrapper or gateway anywhere. Every crate grabs the raw RwLock. That is exactly the kind of thing that turns into a debugging nightmare the moment you add a second writer.

Second, there is no production transaction-admission pipeline at all. P2P Message::Tx is decoded and dropped on the floor, RPC sendrawtransaction only writes a lookup cache, and the single real insert path is Electrum blockchain.transaction.broadcast. The codebase itself admits that relay and reconsideration are "owed once transaction relay exists." Fine. We are not building that here. We are building the one gateway so that when someone finally does build it, there's a door to walk through instead of another back-alley lock grab.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement Phase 1: freeze the Core/enforcer A/R wire contract and build the
internal mutation vocabulary. Do this before writing any publisher code — every
later phase depends on it.

Document the contract:
- Confirm the payload layout against Core `doc/zmq.md`: `A` body is
`reversed_txid(32) || 0x41 || mempool_sequence(u64 LE)`; `R` body is
`reversed_txid(32) || 0x52 || mempool_sequence(u64 LE)`. `C`/`D` bodies stay
`reversed_hash(32) || 0x43|0x44` with no 8-byte field.
- Note that the trailing 4-byte little-endian transport counter is a separate
third ZMQ frame shared across all sequence events, distinct from the 8-byte
mempool sequence.
- Lock in the block-inclusion decision: no `R` for block-included transactions,
`C` semantics unchanged, and conflict/replacement removals during block connect
do emit `R`.
- Record the reorg ordering rule guaranteed by `switch_to_branch`: tip-first
disconnects, then replacement-branch connects.
- Record recovery expectations: `A`/`R` are best-effort; the transport counter
lets a subscriber detect a gap; the enforcer reconciles a detected gap through
RPC, not through mempool rollback.
- Write all of this into a new note under
`docs/solutions/architecture-patterns/` and as named constants/comments in a
compatibility test module.

Add the mutation vocabulary in `crates/mempool/`:
- Add `RemovalReason` with variants covering at least `BlockInclusion`,
`Conflict`/`Replaced`, `Descendant`, `PolicyEviction` (fee/size), `Expiry`,
`Explicit`, `Clear`, and `Reorg`.
- Add `MutationChange` (txid, assigned mempool sequence, and either "accepted"
or a `RemovalReason`).
- Add `MutationResult` as an ordered `Vec<MutationChange>` describing one
committed mutation.
- Keep the reason model richer than the wire format; the wire mapping lives in
the node observer later, not in this crate.

Add the per-event sequence counter:
- Add a monotonic mempool-event sequence to `Mempool` that advances once per
emitted change (per accepted transaction, per removed transaction), assigned
under the write lock.
- Assign a sequence only for successful, non-empty changes — no sequence for a
failed insert, a no-op remove, or a `clear` on an empty pool.
- Reconcile `sequence_number()` (used by `crates/rpc/src/handlers/mempool.rs`
for `getmempoolinfo`/`getrawmempool`) so `mempool_sequence` reports this same
per-event counter.
===============================================================================

Task: 2

Implement Phase 2: eliminate every scattered raw write lock and force all
production mutation through one `MempoolGateway`. After this phase, nothing
outside the gateway may mutate the pool.

Change mutation methods to report ordered changes:
- Update `insert_entry`, `remove_entry_and_descendants`, `remove_by_txid`,
`enforce_size_limit`, `evict_below_fee_rate`, and `clear` in
`crates/mempool/src/pool.rs` to build and return `MutationResult` instead of
bare `Vec<EntryId>`/`Vec<Txid>`.
- Update `replace_transaction`/`check_replacement` in
`crates/mempool/src/rbf.rs` to tag direct conflicts as `Replaced` and swept
descendants as `Descendant`.
- For `insert_entry`, include in the returned `MutationResult` both the accepted
transaction and any transactions the automatic post-insert `enforce_size_limit`
evicts, in commit order, each tagged `PolicyEviction`.
- Guarantee package removal reports the parent removal before its descendants,
so emitted order is deterministic.

Build the gateway and observer trait in `crates/mempool/`:
- Add `MempoolObserver`, a trait with one method that receives a committed
`MutationResult`.
- Add `MempoolGateway`, owning `Arc<RwLock<Mempool>>`, an optional `Arc<dyn
MempoolObserver>`, and a dedicated `publish` mutex.
- For every mutating gateway method, follow this exact ordering: take the
mempool write lock, mutate, assign sequences, acquire the `publish` mutex, drop
the write lock, hand the `MutationResult` to the observer under the `publish`
mutex, then release the `publish` mutex.
- Expose a read guard or `with_read` helper for lookup callers, so they never
touch the raw lock.
- Make the observer call best-effort by contract: an observer error must never
affect an already-committed mutation.

Route every production mutation site through the gateway:
- `crates/node/src/apply.rs`: route the block-connect removal (`remove_by_txid`
loop) through the gateway with `RemovalReason::BlockInclusion`, keeping it after
the mempool lock scope, consistent with the existing publish placement.
- `crates/rpc/src/handlers/mempool.rs` `clearmempool` and
`crates/rpc/src/handlers/mining.rs` `prioritisetransaction`: route through the
gateway. Note that `prioritise` only changes priority, so it must produce no
`A`/`R` change.
- `crates/electrum/src/methods.rs` `MempoolHandle`: route
`blockchain.transaction.broadcast` inserts through the gateway instead of
`pool.write().insert_entry`.
- Replace the raw `Arc<RwLock<Mempool>>` with a shared `Arc<MempoolGateway>` in
`crates/rpc/src/context.rs`, `crates/node/src/apply.rs` `ApplyHandles`, and
`crates/electrum`.
- Add code comments (and update the Phase 1 note) stating that P2P `Message::Tx`
admission and wallet-originated admission do not exist yet, and must enter
through this gateway when built. Do not build them in this phase.
===============================================================================

Task: 3

Implement Phase 3: turn committed `MutationResult`s into wire events. Existing
block-event behavior (`C`/`D`) must not change one bit.

Extend the wire model in `crates/node/src/zmq_publisher.rs`:
- Add `Added(Txid, u64)` and `Removed(Txid, u64)` variants to `SequenceEvent`,
with labels `0x41`/`0x52`.
- Add a payload builder that appends the 8-byte little-endian mempool sequence
for `A`/`R`, and leaves `C`/`D` payloads byte-for-byte unchanged.
- Reuse the existing `ZmqTopic::Sequence` transport counter for `A`/`R`. Do not
add a new counter slot. The 8-byte mempool sequence is a separate value from
that transport counter.

Implement the node-side observer in `crates/node`:
- Add a `MempoolObserver` implementation holding `Arc<dyn ZmqPublisher>`. For
each `MutationChange` in order, call `publish_sequence` with `Added` or
`Removed`.
- Apply the wire mapping: `RemovalReason::BlockInclusion` produces no `R` event;
every other removal reason produces `R`. Accepted changes always produce `A`.
- Wire this observer into the gateway during node startup in
`crates/node/src/state.rs`/`run.rs`, next to the existing
`SocketZmqPublisher`/`NoOpZmqPublisher` selection. Install a no-op observer when
no `sequence` endpoint is configured.

Verify the invariants hold:
- Confirm reorg still emits per-block `D` then `C` in the existing
`switch_to_branch` order, and that any `R` events from conflict removal
interleave correctly relative to the block events.
- Confirm no ZMQ I/O runs while the mempool `RwLock` is held. The observer must
run only under the `publish` mutex, after the write lock is dropped.
- Confirm a ZMQ send failure or an absent subscriber cannot propagate an error
into apply, reorg, or the gateway commit path. Notification failure must stay
best-effort, always.
===============================================================================

Task: 4

Implement Phase 4: flip the enforcer's mempool mode on, and fix every doc that
still claims it is deliberately off.

Enable mempool mode in Compose:
- Add `--enable-mempool` to the enforcer `command:` in
`tools/bip300301-enforcer/docker-compose.yaml`, keeping the existing
`--node-zmq-addr-sequence` wiring and the pinned `ENFORCER_REVISION` untouched.
- Validate enforcer startup and synchronization against the pinned revision. If
the pinned revision cannot consume the frozen contract, record the required
revision bump instead of silently altering wire behavior.

Reconcile the documentation:
- Update `docs/rest-interface.md`: state that `pubsequence` now publishes
mempool `A`/`R`, and that the enforcer runs with `--enable-mempool`. Remove the
"block-only mode" and "deliberately omits `--enable-mempool`" language.
- Update `CONCEPTS.md`'s "Sequence stream" and reorg sections to describe
per-transaction sequence assignment and explicit removal reasons.
- Update `README.md` and the overlapping `docs/solutions/architecture-patterns/`
notes (including the reorg-execution design's "Open" list) to move mempool
sequence events and reconsideration status out of the open/pending category.
===============================================================================

Task: 5

Implement Phase 5: build the acceptance evidence for everything done in Phases 1
through 4. Follow the existing `RecordingSequencePublisher` pattern already used
in `crates/node/src/apply.rs` tests.

Payload and sequence compatibility tests:
- Add byte-level tests for `A` and `R` payloads: 32-byte reversed txid, correct
label, 8-byte LE mempool sequence. Place these alongside the existing
`sequence_payload` tests.
- Assert `C`/`D` payloads remain byte-for-byte unchanged.
- Add a defined and tested behavior for transport-counter `u32` wraparound.

Ordering and reason tests:
- Add a recording `MempoolObserver` that captures `(id, label, sequence,
reason)` tuples in a `Vec`, mirroring `RecordingSequencePublisher`.
- Cover: single accept emits exactly one `A`; package removal emits `R` for the
parent then each descendant, in that order; conflict/replacement emits `R` for
direct conflicts and descendants; fee/size eviction emits `R`; `clear` emits `R`
for every entry; block connect emits no `R` for block-included transactions but
does emit `R` for conflicts.
- Add reorg tests covering disconnect, transaction reconsideration/re-admission
where applicable, and replacement-branch connect ordering.
- Assert failed inserts and no-op removes emit nothing and consume no sequence
number.

Failure isolation and end-to-end gate:
- Add a test where the observer/publisher fails, and assert the mempool mutation
still commits, and apply/reorg still succeed regardless.
- Add or extend a gate under `bin/bitcoin-rs/tests/gates/`, following the
existing loud-placeholder convention, documenting the end-to-end enforcer
`--enable-mempool` transaction-lifecycle validation against the pinned revision.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!