Define and differentially verify the Bitcoin Core-compatible RPC, REST, and ZMQ contract

[gosunuts]

Problem

The project exposes a useful Bitcoin Core-shaped interface:

• synchronous HTTP JSON-RPC with Core method names;
• a deliberately small REST gateway for remote validators;
• ZMQ hash/raw topics and pubsequence block connect/disconnect notifications;
• Core-style configuration and getzmqnotifications metadata.

The implementation has handler smoke tests, selected response-schema tests, REST edge-case tests, and ZMQ transport/wiring tests. Those are a solid foundation but do not define a complete supported API contract or establish drop-in compatibility with Bitcoin Core.

Current gaps include:

• no machine-readable inventory of supported, partial, disabled, and unsupported methods;
• selected JSON result-key tests rather than broad behavioral/error differential coverage;
• no versioned Core reference target for API compatibility;
• REST intentionally implements only chaininfo and headers endpoints;
• pubsequence omits mempool A/R events (Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74);
• ZMQ is best-effort and has no durable recovery contract (Architectural boundary: durable chain events and independent derived-state consumers #77);
• some wallet methods are intentionally disabled because this node has no private keys;
• documented gaps remain in reorg/mempool and index readiness semantics.

Goal

Publish and enforce a narrow, explicit, versioned Bitcoin Core compatibility contract for RPC, REST, and ZMQ. Every claimed supported surface must be verified against a pinned Bitcoin Core reference through deterministic fixtures and differential tests. Unsupported and intentionally partial surfaces must fail or report capability status predictably.

This issue does not require implementing every Core API method.

Scope

1. Freeze the compatibility target and surface manifest

• Pin the Bitcoin Core major/minor release used for each compatibility claim.
• Add a machine-readable manifest for every exposed RPC, REST route, and ZMQ topic:
  • supported;
  • supported with documented deviations;
  • disabled by product policy;
  • not implemented.
• State parameter forms, prerequisites, required indexes, pruning behavior, auth policy, and readiness behavior.
• Generate or validate human API reference material from the manifest.

2. JSON-RPC differential harness

For each supported method, compare bitcoin-rs and the pinned Core reference using deterministic regtest and fixture chains.

Cover:

• success JSON shape, types, optional fields, ordering where wire-visible, and numeric semantics;
• positional and named parameters where applicable;
• invalid params, unknown objects, pruned/missing data, disabled index, and auth failures;
• RPC error code and stable error-category/message behavior;
• batch, notification, malformed JSON, HTTP status, content type, and keep-alive behavior;
• state transitions: initial sync, caught-up, reorg, index building/stale, pruning, and shutdown.

Do not normalize meaningful incompatibilities away. Record an explicit deviation only after it is reviewed.

3. REST contract and tests

• Keep the REST surface intentionally small unless a concrete consumer needs expansion.
• Differentially verify implemented chaininfo/header semantics, status codes, content types, binary/hex encoding, malformed values, side branches, header-only ranges, and disabled REST behavior.
• Document that REST is unauthenticated and define safe listener/exposure guidance.
• Add any enforcer-specific fixture necessary to prove the expected discovery and rescan behavior.

4. ZMQ contract and tests

• Verify endpoint parsing, high-water mark behavior, multipart frame layout, topic names, byte order, and sequence counter behavior against Core fixtures for each supported topic.
• Make connect/disconnect ordering part of the public pubsequence contract.
• Complete A/R semantics under Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74 before declaring full pubsequence parity.
• State clearly that ZMQ is a low-latency projection, not a durable replay mechanism; connect it to the durable event/cursor recovery model in Architectural boundary: durable chain events and independent derived-state consumers #77.
• Test slow/disconnected subscribers, socket failure, reconnection, and sequence wraparound without allowing notification failures to affect consensus.

5. Capability/readiness reporting

Expose enough status for clients to distinguish:

• node applied tip versus header tip;
• txindex/filter index absent, building, caught-up, failed, or stale;
• pruning/body availability;
• ZMQ configured topics/endpoints;
• feature-disabled methods.

Avoid pretending a partial index or partial node state is complete Core-equivalent data.

Acceptance criteria

• A pinned Core version and a machine-readable supported/deviation/disabled/not-implemented manifest exist.
• Every method/route/topic advertised as supported has a deterministic differential test or a documented fixture-based reason it cannot be compared.
• JSON-RPC happy path, invalid parameters, method-not-found, auth, malformed request, batch, and HTTP framing are covered.
• Result schemas and semantic values are compared, not merely method names or selected key sets.
• REST endpoint/status/encoding behavior is tested for the complete documented subset.
• ZMQ frame layout, endianness, sequence behavior, C/D ordering, and configured-topic discovery are tested.
• Mempool A/R pubsequence parity remains explicitly unavailable until Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74 is complete.
• ZMQ delivery failure cannot affect apply/consensus, and event-loss recovery points to Architectural boundary: durable chain events and independent derived-state consumers #77's durable source.
• Wallet methods without private-key support remain explicitly disabled with stable errors rather than partial imitation.
• API docs describe prerequisites and incomplete-index/pruning/readiness behavior.
• CI executes the portable deterministic compatibility suite; live Core integration runs are documented and reproducible.

Related issues

• Complete BIP300/301 enforcer integration with mempool A/R pubsequence events #74: complete A/R pubsequence events for enforcer mempool mode.
• Add resumable post-IBD backfill for txindex, Electrum, and block filters #75: backfill-derived indexes and readiness states that APIs must report accurately.
• Architectural boundary: durable chain events and independent derived-state consumers #77: durable chain events underpin consumer recovery; ZMQ is a projection of that state.
• Implement an experimental Utreexo compact-validator node mode #76: experimental Utreexo mode must expose explicit capability/readiness rather than changing default API meaning.
• Plan #1: original full-node API and Electrum scope.

Non-goals

• Implementing every Bitcoin Core RPC or REST endpoint.
• Adding private-key wallet management.
• Treating a Core method name as proof of behavioral compatibility.
• Making ZMQ a durable queue or a consensus dependency.
• Blocking default full-node work on optional API products.

[coderabbitai]

🔗 Related PRs

#14 - feat: header timestamp consensus rules, index reorg rollback, mempool policy, fuzz targets [merged]
#49 - rpc: Core-compatible REST gateway so a remote bip300301_enforcer can use bitcoin-rs as its mainchain node [merged]
#50 - rpc: report real difficulty, time and mediantime from getblockchaininfo [merged]
#52 - node: publish Core's sequence ZMQ topic for block connects and disconnects [merged]
#73 - Feature/drynet sync [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

✅ Successfully generated implementation plan.

💡 Use @coderabbitai <feedback> to replan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Make the manifest the single source of truth, then force the dispatch table, the API reference, and every differential test to agree with it. No surface can drift silently.
• Split the differential harness into a portable golden-fixture path that runs in CI and an #[ignore]d live path that regenerates fixtures against a pinned bitcoind. This matches the repo's existing gate convention.
• Compare values and error codes by decoded semantics. Never compare by JSON text spelling. Follow the recorded float-parity lesson.
• Document, do not implement, the out-of-scope gaps. Mempool A/R stays blocked on #74. Durable recovery stays pointed at #77. Disabled wallet methods keep their stable policy error.

Design Choices

Design Choice 1: What format and location does the compatibility manifest use?

Options Considered:

1. A checked-in TOML file under a new crates/rpc/compat/ (or docs/compat/) directory, parsed with serde, validated by a test that cross-checks it against the live dispatch table.
2. A JSON file with the same validation.
3. Rust const tables compiled into the crate.

Chosen Option: 1

Rationale: TOML is the most reviewable in pull requests, parses cleanly with the existing serde stack, and keeps the contract as data rather than code so the API reference and tests both consume one artifact. The manifest must be machine-readable, reviewable in diffs, and the single source of truth for the RPC dispatch table, REST routes, and ZMQ topics.

Design Choice 2: How does the differential harness run in CI without a live `bitcoind`?

Options Considered:

1. Capture golden JSON/ZMQ fixtures once from the pinned Core release, check them in, compare bitcoin-rs output against those fixtures in CI, and add a separate #[ignore]d live harness that regenerates the fixtures against a real pinned bitcoind.
2. Require a live bitcoind for the whole differential suite.

Chosen Option: 1

Rationale: This satisfies the acceptance criterion that CI runs a portable deterministic suite while live Core runs stay documented and reproducible, and it matches the repo's #[ignore] plus evidence-contract convention. CI never spawns external binaries; the established convention is #[ignore] plus a strict env-var evidence contract plus a documented manual command.

Design Choice 3: How are identical fixture chains built on both bitcoin-rs and Core?

Options Considered:

1. Check in deterministic regtest fixture blocks and load them into both bitcoin-rs and Core through submitblock (both implement it).
2. Add a generatetoaddress/generate RPC to bitcoin-rs to match Core's mining entry point.

Chosen Option: 1

Rationale: submitblock already exists on both sides and gives byte-identical chains without adding a new mining RPC, which is out of scope for this ticket. Differential comparison needs byte-identical chain state on both nodes, and there is no shared chain-building utility today.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Freeze the compatibility target and publish the manifest

This phase defines the contract before any test enforces it. It pins the Core reference version, writes the machine-readable manifest for every RPC method, REST route, and ZMQ topic, records intentional deviations, and generates the human API reference from the manifest. Nothing here compares against Core yet; it establishes the source of truth.

Task 1: Pin the Core reference version and write the compatibility policy

Create the versioned contract document and pin the exact Core release each claim targets.

• Add a new policy document docs/policies/core-compatibility.md. Mirror the structure of docs/policies/source-compatibility.md: a Scope and Authority header, numbered rules, and an explicit anti-shim / immediate-cutover posture.
• State the pinned Bitcoin Core major.minor release used for every compatibility claim in a settings table. Reuse the version referenced in existing benchmarks/docs unless a newer pin is justified.
• Cross-reference docs/rest-interface.md, CONCEPTS.md (sequence stream), and docs/policies/source-compatibility.md §5.2.
• State the version-bump trigger rules the same way §4.1 of source-compatibility does.

Task 2: Author the machine-readable surface manifest

Define the single source of truth for every exposed surface and its status.

• Create crates/rpc/compat/manifest.toml. Enumerate every RPC method, REST route, and ZMQ topic.
• For each entry record: a status of supported, supported_with_deviations, disabled_by_policy, or not_implemented; the parameter forms (positional and named); prerequisites; required indexes; pruning behavior; auth policy; and readiness behavior.
• Populate RPC entries from the dispatch table in crates/rpc/src/handlers.rs and the per-domain modules. Mark the private-key wallet methods (signrawtransactionwithkey, dumpprivkey, and the rest of the disabled list) as disabled_by_policy with the stable NO_PRIVATE_KEYS reason.
• Populate REST entries from crates/rpc/src/rest.rs (/rest/chaininfo.json, /rest/headers/*).
• Populate ZMQ entries from the ZmqTopic enum in crates/node/src/zmq_publisher.rs. Mark sequence as supported_with_deviations because mempool A/R events are absent (link #74).

Task 3: Validate the manifest against the live surface

Prevent the manifest from drifting from the code. A manifest that lies is worse than no manifest.

• Add a manifest-parsing module in crates/rpc/src/ that deserializes manifest.toml with serde and exposes typed lookups.
• Add a test (extend or replace crates/rpc/tests/core_compat.rs) that asserts every dispatchable method in Handler::dispatch appears in the manifest with a non-not_implemented status, and every supported/disabled_by_policy manifest entry maps to a real dispatch arm.
• Assert the same bidirectional coverage for REST routes and ZMQ topics.

Task 4: Generate the API reference from the manifest

Produce human-readable reference material that cannot contradict the manifest.

• Add a generator (a small test-invoked routine or an xtask-style helper under crates/rpc/ or tools/) that renders docs/rpc-reference.md from manifest.toml, listing prerequisites, required indexes, pruning behavior, auth, and readiness per surface.
• Add a test that fails if the checked-in generated document is stale relative to the manifest.
• Record the getnetworkinfo version/subversion behavior and the mempool A/R omission as dated, sourced entries in DEVIATIONS.md, using the existing table + Consequence format, cross-referencing the manifest and #74.

🤖 Prompt for AI agents

Goal: freeze the Bitcoin Core compatibility contract as data, before any
enforcement code exists.

1. Write `docs/policies/core-compatibility.md` following the structure of
`docs/policies/source-compatibility.md`: Scope and Authority header, numbered
rules, anti-shim / immediate-cutover posture, a settings table pinning the exact
Core major.minor release, cross-references to `docs/rest-interface.md`,
`CONCEPTS.md` sequence stream section, and
`docs/policies/source-compatibility.md` §5.2, and version-bump trigger rules
matching §4.1 of source-compatibility.

2. Create `crates/rpc/compat/manifest.toml` enumerating every RPC method, REST
route, and ZMQ topic. For each entry record status (`supported`,
`supported_with_deviations`, `disabled_by_policy`, `not_implemented`), parameter
forms (positional and named), prerequisites, required indexes, pruning behavior,
auth policy, and readiness behavior. Source RPC entries from
`crates/rpc/src/handlers.rs` and per-domain modules; mark disabled private-key
wallet methods (`signrawtransactionwithkey`, `dumpprivkey`, etc.) as
`disabled_by_policy` with reason `NO_PRIVATE_KEYS`. Source REST entries from
`crates/rpc/src/rest.rs`. Source ZMQ entries from the `ZmqTopic` enum in
`crates/node/src/zmq_publisher.rs`; mark `sequence` as
`supported_with_deviations` referencing the missing mempool A/R events (`#74`).

3. Add a manifest-parsing module in `crates/rpc/src/` that deserializes
`manifest.toml` with serde and exposes typed lookups. Replace or extend
`crates/rpc/tests/core_compat.rs` with a bidirectional coverage test: every
dispatchable method in `Handler::dispatch` must appear in the manifest with a
non-`not_implemented` status, and every `supported`/`disabled_by_policy`
manifest entry must map to a real dispatch arm. Apply the same bidirectional
check to REST routes and ZMQ topics.

4. Add a generator (test-invoked routine or `xtask`-style helper) that renders
`docs/rpc-reference.md` from `manifest.toml`, listing prerequisites, required
indexes, pruning behavior, auth, and readiness per surface. Add a test that
fails if the checked-in generated document is stale relative to the manifest.
Record the `getnetworkinfo` version/subversion behavior and the mempool A/R
omission in `DEVIATIONS.md`, using the existing table + Consequence format,
cross-referencing the manifest and `#74`.

Phase 2: Build the differential harness foundation

This phase builds the reusable test infrastructure that later phases consume. It extracts a shared regtest chain builder, spins a real RpcServer over a socket backed by applied chain state, defines the checked-in fixture format, and implements a comparison engine that respects numeric semantics. This phase produces no method-specific coverage yet; it produces the tools.

Task 1: Extract a shared regtest chain-and-server harness

Stop copy-pasting nonce loops into every test file. Build one harness and reuse it.

• Create a new test-support module (a dev-dependency support crate under crates/rpc/tests/support/ or a small internal crate) that mines deterministic regtest blocks by brute-forcing the nonce against the declared target, reusing the pattern in crates/node/tests/sync_smoke.rs and crates/node/src/state.rs.
• Provide a helper that opens a real NodeState on Network::Regtest, applies N fixture blocks, builds a Context through Context::from_handles, and starts a real RpcServer bound to 127.0.0.1:0, following the socket pattern in crates/rpc/tests/auth.rs.
• Provide the same fixture blocks as checked-in raw hex so both bitcoin-rs and a live Core node ingest identical chains through submitblock.

Task 2: Define the fixture format and capture path

Define how Core reference output is captured, versioned, and loaded.

• Create a crates/rpc/tests/fixtures/ directory. Store per-method golden captures (request, response body, HTTP status, content type) from the pinned Core release as checked-in files.
• Record the pinned Core version inside each fixture set so a version bump forces a visible refresh.
• Add a #[ignore]d live harness that spawns a pinned bitcoind (model the lifecycle on scripts/run-g14-bitcoin-core-mainnet-ibd.sh), loads the fixture chain through submitblock, captures Core output, and regenerates the golden fixtures. Gate it on an env-var evidence contract as other gates do.

Task 3: Implement the numeric-safe comparison engine

Comparison must respect the float-parity lesson. Text equality is not value equality, and value equality is not text equality.

• Add a comparison module in the test-support crate that compares a bitcoin-rs response against a Core fixture by JSON structure, key presence (required vs optional), value types, and wire-visible ordering.
• Compare numeric values by decoded value, not by text spelling. For values computed by Rust before serialization, compare by f64::to_bits(); for values parsed from JSON text, compare with a documented tolerance or canonicalized representation. Follow docs/solutions/logic-errors/core-float-parity-is-value-parity-not-json-text-parity.md.
• Compare JSON-RPC error responses by code, error category, and stable message, not only by presence of an error.

🤖 Prompt for AI agents

Goal: build the reusable differential-testing infrastructure that Phases 3-5
will consume. Produce no method-specific test coverage in this phase; produce
only the harness, fixture format, and comparison engine.

1. Create a test-support module (a `dev-dependency` support crate under
`crates/rpc/tests/support/` or a small internal crate). Implement deterministic
regtest block mining by brute-forcing the nonce against the declared target,
reusing the pattern in `crates/node/tests/sync_smoke.rs` and
`crates/node/src/state.rs`. Provide a helper that opens a real `NodeState` on
`Network::Regtest`, applies N fixture blocks, builds a `Context` through
`Context::from_handles`, and starts a real `RpcServer` bound to `127.0.0.1:0`,
following the socket pattern in `crates/rpc/tests/auth.rs`. Check in the same
fixture blocks as raw hex so both bitcoin-rs and a live Core node ingest
identical chains through `submitblock`.

2. Create a `crates/rpc/tests/fixtures/` directory. Store per-method golden
captures (request, response body, HTTP status, content type) from the pinned
Core release as checked-in files, each recording the pinned Core version so a
version bump forces a visible refresh. Add a `#[ignore]`d live harness that
spawns a pinned `bitcoind` (model the lifecycle on
`scripts/run-g14-bitcoin-core-mainnet-ibd.sh`), loads the fixture chain through
`submitblock`, captures Core output, and regenerates the golden fixtures. Gate
this harness on an env-var evidence contract, matching the repo's existing gate
convention.

3. Add a comparison module in the test-support crate. Compare a bitcoin-rs
response against a Core fixture by JSON structure, key presence (required vs
optional), value types, and wire-visible ordering. Compare numeric values by
decoded value, never by text spelling: use `f64::to_bits()` for values computed
by Rust before serialization, and a documented tolerance or canonicalized
representation for values parsed from JSON text. Follow the guidance in
`docs/solutions/logic-errors/core-float-parity-is-value-parity-not-json-text-parity.md`.
Compare JSON-RPC error responses by `code`, error category, and stable message,
not merely by presence of an error.

Phase 3: JSON-RPC differential coverage

This phase uses the Phase 2 harness to differentially verify every RPC method the manifest marks supported or supported_with_deviations. It covers success shapes, parameter forms, error paths, HTTP framing, and state transitions. It also closes the error-code gap for the methods under test.

Task 1: Cover success shapes and parameter forms

Verify each supported method returns Core-equivalent output.

• Add differential tests, driven by the manifest, that for each supported method compare bitcoin-rs output against the golden fixture: JSON shape, types, optional fields, wire-visible ordering, and numeric semantics through the Phase 2.3 engine.
• Cover both positional and named parameters where the manifest declares them.
• For any method that cannot be compared, record a documented fixture-based reason in the manifest instead of silently skipping it.

Task 2: Cover error paths and auth failures

Verify predictable failures, not just happy paths.

• Add tests for invalid params, unknown objects, pruned/missing data, disabled index, and method-not-found, comparing error code and stable message against Core fixtures.
• Extend crates/rpc/src/error.rs to map the additional Core RPC_* codes that the covered methods actually require (for example RPC_INVALID_ADDRESS_OR_KEY, RPC_IN_WARMUP, RPC_CLIENT_IN_INITIAL_DOWNLOAD). Do not add codes no covered method uses.
• Verify HTTP 401 on auth failure for JSON-RPC and unauthenticated access for REST, using the socket harness.
• Verify the disabled wallet methods return the stable NO_PRIVATE_KEYS policy error, not a partial imitation.

Task 3: Cover HTTP framing and batch behavior

Verify the transport contract, not only the payload.

• Extend the socket-level tests (pattern from crates/rpc/tests/auth.rs) to differentially check batch requests, v2 notifications, malformed JSON, HTTP status codes, content type, and keep-alive against Core's framing behavior.
• Verify JSON-RPC v1 vs v2 status-code and envelope differences match Core.

Task 4: Cover node state transitions

Verify readiness-sensitive output across node states.

• Add tests that drive the harness through initial sync, caught-up, reorg (through invalidateblock), index building/stale, pruning, and shutdown, and assert the state-dependent RPC output (for example initialblockdownload, verificationprogress, getindexinfo.synced) matches Core semantics.

🤖 Prompt for AI agents

Goal: differentially verify every RPC method the manifest marks `supported` or
`supported_with_deviations`, using the Phase 2 harness and comparison engine,
and close the error-code mapping gap for the methods under test.

1. Add manifest-driven differential tests: for each `supported` method, compare
bitcoin-rs output against the golden fixture on JSON shape, types, optional
fields, wire-visible ordering, and numeric semantics through the Phase 2
comparison engine. Cover both positional and named parameters where the manifest
declares them. If a method cannot be compared, record a documented fixture-based
reason in the manifest instead of silently skipping it.

2. Add tests for error paths: invalid params, unknown objects, pruned/missing
data, disabled index, and method-not-found, comparing error `code` and stable
message against Core fixtures. Extend `crates/rpc/src/error.rs` to map only the
additional Core `RPC_*` codes that the covered methods actually require (for
example `RPC_INVALID_ADDRESS_OR_KEY`, `RPC_IN_WARMUP`,
`RPC_CLIENT_IN_INITIAL_DOWNLOAD`); do not add codes that no covered method uses.
Verify HTTP 401 on auth failure for JSON-RPC and unauthenticated access for REST
using the socket harness. Verify disabled wallet methods return the stable
`NO_PRIVATE_KEYS` policy error rather than a partial imitation.

3. Extend the socket-level tests (pattern from `crates/rpc/tests/auth.rs`) to
differentially check batch requests, v2 notifications, malformed JSON, HTTP
status codes, content type, and keep-alive against Core's framing behavior.
Verify JSON-RPC v1 vs v2 status-code and envelope differences match Core.

4. Add tests that drive the harness through initial sync, caught-up, reorg (via
`invalidateblock`), index building/stale, pruning, and shutdown states, and
assert state-dependent RPC output (for example `initialblockdownload`,
`verificationprogress`, `getindexinfo.synced`) matches Core semantics.

Phase 4: REST contract and tests

This phase locks the intentionally small REST surface and verifies it differentially. It keeps the surface unchanged in size, documents the unauthenticated exposure, and adds the enforcer fixture.

Task 1: Differentially verify the REST subset

Verify chaininfo and headers against Core, including edge cases.

• Add differential tests over the socket harness for /rest/chaininfo.json and /rest/headers/* covering status codes, content types, .json/.hex/.bin encodings, malformed hashes, out-of-range count, side branches, header-only ranges, and the disabled-REST 404.
• Preserve the load-bearing 404-vs-400 distinction described in docs/rest-interface.md; the enforcer relies on it.

Task 2: Document exposure guidance and add the enforcer fixture

Make the unauthenticated posture explicit and prove enforcer discovery.

• Update docs/rest-interface.md to state REST is unauthenticated and to define safe listener/exposure guidance.
• Add an enforcer-oriented fixture that proves the expected discovery and rescan behavior, reusing the wiring in tools/bip300301-enforcer/.
• Keep the REST surface size unchanged; the manifest marks unimplemented Core REST routes as not_implemented.

🤖 Prompt for AI agents

Goal: lock the small, intentional REST surface (`chaininfo`, `headers`) and
verify it differentially against Core, without growing the surface.

1. Add differential tests over the socket harness for `/rest/chaininfo.json` and
`/rest/headers/*`, covering status codes, content types, `.json`/`.hex`/`.bin`
encodings, malformed hashes, out-of-range `count`, side branches, header-only
ranges, and the disabled-REST 404. Preserve the load-bearing 404-vs-400
distinction described in `docs/rest-interface.md`, because the enforcer depends
on it.

2. Update `docs/rest-interface.md` to state that REST is unauthenticated and to
define safe listener/exposure guidance. Add an enforcer-oriented fixture that
proves expected discovery and rescan behavior, reusing the wiring in
`tools/bip300301-enforcer/`. Keep the REST surface size unchanged: mark every
unimplemented Core REST route as `not_implemented` in the manifest, not as a new
route.

Phase 5: ZMQ contract and tests

This phase makes the ZMQ frame layout a public contract, verifies it against Core fixtures, and tests subscriber failure modes without allowing any notification failure to reach consensus. It documents the projection posture and keeps A/R events explicitly out of scope.

Task 1: Verify frame layout against Core fixtures

Replace self-referential encode/decode checks with fixture parity.

• Add fixture-based tests in crates/node/tests/ (or extend the zmq_publisher.rs test module) that verify, per supported topic, the multipart frame layout, topic names, byte order, and the trailing little-endian u32 sequence counter against captured Core bytes.
• Make connect/disconnect (C/D) ordering part of the documented pubsequence contract, consistent with the exact CONCEPTS.md wording (disconnects tip-first before connects).
• Verify configured-topic discovery through getzmqnotifications matches the manifest and the running endpoints.

Task 2: Test subscriber failure modes and consensus isolation

Prove ZMQ failures never touch apply/consensus.

• Add tests for slow/disconnected subscribers, high-water-mark drop, socket failure, reconnection, and sequence-counter wraparound.
• Assert that block apply and disconnect in crates/node/src/apply.rs complete regardless of ZMQ delivery outcome, so a notification failure cannot affect consensus.

Task 3: Document the projection posture and open gaps

State clearly what ZMQ is and is not.

• Update CONCEPTS.md and docs/rest-interface.md so ZMQ is described as a low-latency projection, not a durable replay mechanism, and point event-loss recovery at #77's durable event/cursor model.
• State that mempool A/R pubsequence parity remains explicitly unavailable until #74, and mark it as such in the manifest and DEVIATIONS.md.

🤖 Prompt for AI agents

Goal: make the ZMQ frame layout an enforced public contract against Core
fixtures, prove subscriber failures cannot touch consensus, and document the
projection posture honestly, keeping mempool A/R events explicitly out of scope.

1. Add fixture-based tests in `crates/node/tests/` (or extend the
`zmq_publisher.rs` test module) verifying, per supported topic, the multipart
frame layout, topic names, byte order, and the trailing little-endian `u32`
sequence counter against captured Core bytes. Make connect/disconnect (`C`/`D`)
ordering part of the documented `pubsequence` contract, matching the exact
`CONCEPTS.md` wording that disconnects go tip-first before connects. Verify
configured-topic discovery through `getzmqnotifications` matches the manifest
and the running endpoints.

2. Add tests for slow/disconnected subscribers, high-water-mark drop, socket
failure, reconnection, and sequence-counter wraparound. Assert that block apply
and disconnect in `crates/node/src/apply.rs` complete regardless of ZMQ delivery
outcome, so a notification failure never affects consensus.

3. Update `CONCEPTS.md` and `docs/rest-interface.md` to describe ZMQ as a
low-latency projection, not a durable replay mechanism, and to point event-loss
recovery at `#77`'s durable event/cursor model. State that mempool A/R
`pubsequence` parity remains unavailable until `#74`, and mark this in both the
manifest and `DEVIATIONS.md`.

Phase 6: Capability and readiness reporting

This phase gives clients enough status to distinguish real Core-equivalent data from partial node/index state. It introduces a readiness vocabulary and exposes it through existing RPC surfaces consistent with the manifest.

Task 1: Introduce a readiness state vocabulary

Replace the single synced boolean with a state that can express partial data.

• Add a readiness state enum (absent, building, caught_up, stale, failed) in the crate that owns index/node readiness. Do not overload crates/index/src/status.rs; that file is Electrum status hashing and unrelated.
• Derive the state from existing signals: header tip vs applied tip, index applied height, and failure/stall conditions.

Task 2: Expose readiness through RPC consistent with the manifest

Report status so clients never mistake partial data for complete data.

• Extend getindexinfo and getblockchaininfo in crates/rpc/src/handlers/chain.rs to expose applied tip versus header tip, per-index readiness state, pruning/body availability, configured ZMQ topics/endpoints, and feature-disabled methods, staying within the manifest's declared readiness behavior.
• Pin the new readiness output in the differential suite (Phase 3.4) so it is verified, not merely present.
• Keep the readiness reporting truthful: a partial index or partial node state must report its state, not claim Core-equivalent completeness.

🤖 Prompt for AI agents

Goal: give clients a readiness vocabulary that distinguishes real
Core-equivalent data from partial node/index state, and expose it through
existing RPC surfaces consistent with the manifest.

1. Add a readiness state enum (`absent`, `building`, `caught_up`, `stale`,
`failed`) in the crate that owns index/node readiness. Do not put this in
`crates/index/src/status.rs`; that file handles Electrum status hashing and is
unrelated. Derive the state from existing signals: header tip versus applied
tip, index applied height, and failure/stall conditions.

2. Extend `getindexinfo` and `getblockchaininfo` in
`crates/rpc/src/handlers/chain.rs` to expose applied tip versus header tip,
per-index readiness state, pruning/body availability, configured ZMQ
topics/endpoints, and feature-disabled methods, staying within the manifest's
declared readiness behavior. Pin this new readiness output in the Phase 3.4
differential suite so it is verified, not merely present. Keep the readiness
reporting truthful: a partial index or partial node state must report its actual
state, never claim Core-equivalent completeness.

Research

The RPC crate (crates/rpc) dispatches Core-named methods through handlers.rs into per-domain handler modules, serves a blocking HTTP/1.1 server (server.rs) with JSON-RPC v1/v2 framing, HTTP Basic/cookie auth (auth.rs), and a small REST surface (rest.rs — only chaininfo and headers). Private-key wallet methods are disabled with a stable policy error. ZMQ notifications live in crates/node/src/zmq_publisher.rs (topics hashblock, hashtx, rawblock, rawtx, sequence with C/D labels). Readiness is reported today only as booleans through getindexinfo and getblockchaininfo.

Now the blunt part. The existing crates/rpc/tests/core_compat.rs is misnamed. It compares response key sets against hand-typed literal lists inside an in-process Handler with an empty Context. It never talks to a real bitcoind, never checks a value, and never checks an error code. It is a schema-shape smoke test wearing a "compat" hat. There is no pinned Core version, no manifest, no reusable regtest-chain-plus-live-RpcServer harness, and error codes map only 3 of ~20 Core RPC_* values. This ticket replaces the pretense with a real, enforced contract.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Goal: freeze the Bitcoin Core compatibility contract as data, before any
enforcement code exists.

1. Write `docs/policies/core-compatibility.md` following the structure of
`docs/policies/source-compatibility.md`: Scope and Authority header, numbered
rules, anti-shim / immediate-cutover posture, a settings table pinning the exact
Core major.minor release, cross-references to `docs/rest-interface.md`,
`CONCEPTS.md` sequence stream section, and
`docs/policies/source-compatibility.md` §5.2, and version-bump trigger rules
matching §4.1 of source-compatibility.

2. Create `crates/rpc/compat/manifest.toml` enumerating every RPC method, REST
route, and ZMQ topic. For each entry record status (`supported`,
`supported_with_deviations`, `disabled_by_policy`, `not_implemented`), parameter
forms (positional and named), prerequisites, required indexes, pruning behavior,
auth policy, and readiness behavior. Source RPC entries from
`crates/rpc/src/handlers.rs` and per-domain modules; mark disabled private-key
wallet methods (`signrawtransactionwithkey`, `dumpprivkey`, etc.) as
`disabled_by_policy` with reason `NO_PRIVATE_KEYS`. Source REST entries from
`crates/rpc/src/rest.rs`. Source ZMQ entries from the `ZmqTopic` enum in
`crates/node/src/zmq_publisher.rs`; mark `sequence` as
`supported_with_deviations` referencing the missing mempool A/R events (`#74`).

3. Add a manifest-parsing module in `crates/rpc/src/` that deserializes
`manifest.toml` with serde and exposes typed lookups. Replace or extend
`crates/rpc/tests/core_compat.rs` with a bidirectional coverage test: every
dispatchable method in `Handler::dispatch` must appear in the manifest with a
non-`not_implemented` status, and every `supported`/`disabled_by_policy`
manifest entry must map to a real dispatch arm. Apply the same bidirectional
check to REST routes and ZMQ topics.

4. Add a generator (test-invoked routine or `xtask`-style helper) that renders
`docs/rpc-reference.md` from `manifest.toml`, listing prerequisites, required
indexes, pruning behavior, auth, and readiness per surface. Add a test that
fails if the checked-in generated document is stale relative to the manifest.
Record the `getnetworkinfo` version/subversion behavior and the mempool A/R
omission in `DEVIATIONS.md`, using the existing table + Consequence format,
cross-referencing the manifest and `#74`.
===============================================================================

Task: 2

Goal: build the reusable differential-testing infrastructure that Phases 3-5
will consume. Produce no method-specific test coverage in this phase; produce
only the harness, fixture format, and comparison engine.

1. Create a test-support module (a `dev-dependency` support crate under
`crates/rpc/tests/support/` or a small internal crate). Implement deterministic
regtest block mining by brute-forcing the nonce against the declared target,
reusing the pattern in `crates/node/tests/sync_smoke.rs` and
`crates/node/src/state.rs`. Provide a helper that opens a real `NodeState` on
`Network::Regtest`, applies N fixture blocks, builds a `Context` through
`Context::from_handles`, and starts a real `RpcServer` bound to `127.0.0.1:0`,
following the socket pattern in `crates/rpc/tests/auth.rs`. Check in the same
fixture blocks as raw hex so both bitcoin-rs and a live Core node ingest
identical chains through `submitblock`.

2. Create a `crates/rpc/tests/fixtures/` directory. Store per-method golden
captures (request, response body, HTTP status, content type) from the pinned
Core release as checked-in files, each recording the pinned Core version so a
version bump forces a visible refresh. Add a `#[ignore]`d live harness that
spawns a pinned `bitcoind` (model the lifecycle on
`scripts/run-g14-bitcoin-core-mainnet-ibd.sh`), loads the fixture chain through
`submitblock`, captures Core output, and regenerates the golden fixtures. Gate
this harness on an env-var evidence contract, matching the repo's existing gate
convention.

3. Add a comparison module in the test-support crate. Compare a bitcoin-rs
response against a Core fixture by JSON structure, key presence (required vs
optional), value types, and wire-visible ordering. Compare numeric values by
decoded value, never by text spelling: use `f64::to_bits()` for values computed
by Rust before serialization, and a documented tolerance or canonicalized
representation for values parsed from JSON text. Follow the guidance in
`docs/solutions/logic-errors/core-float-parity-is-value-parity-not-json-text-parity.md`.
Compare JSON-RPC error responses by `code`, error category, and stable message,
not merely by presence of an error.
===============================================================================

Task: 3

Goal: differentially verify every RPC method the manifest marks `supported` or
`supported_with_deviations`, using the Phase 2 harness and comparison engine,
and close the error-code mapping gap for the methods under test.

1. Add manifest-driven differential tests: for each `supported` method, compare
bitcoin-rs output against the golden fixture on JSON shape, types, optional
fields, wire-visible ordering, and numeric semantics through the Phase 2
comparison engine. Cover both positional and named parameters where the manifest
declares them. If a method cannot be compared, record a documented fixture-based
reason in the manifest instead of silently skipping it.

2. Add tests for error paths: invalid params, unknown objects, pruned/missing
data, disabled index, and method-not-found, comparing error `code` and stable
message against Core fixtures. Extend `crates/rpc/src/error.rs` to map only the
additional Core `RPC_*` codes that the covered methods actually require (for
example `RPC_INVALID_ADDRESS_OR_KEY`, `RPC_IN_WARMUP`,
`RPC_CLIENT_IN_INITIAL_DOWNLOAD`); do not add codes that no covered method uses.
Verify HTTP 401 on auth failure for JSON-RPC and unauthenticated access for REST
using the socket harness. Verify disabled wallet methods return the stable
`NO_PRIVATE_KEYS` policy error rather than a partial imitation.

3. Extend the socket-level tests (pattern from `crates/rpc/tests/auth.rs`) to
differentially check batch requests, v2 notifications, malformed JSON, HTTP
status codes, content type, and keep-alive against Core's framing behavior.
Verify JSON-RPC v1 vs v2 status-code and envelope differences match Core.

4. Add tests that drive the harness through initial sync, caught-up, reorg (via
`invalidateblock`), index building/stale, pruning, and shutdown states, and
assert state-dependent RPC output (for example `initialblockdownload`,
`verificationprogress`, `getindexinfo.synced`) matches Core semantics.
===============================================================================

Task: 4

Goal: lock the small, intentional REST surface (`chaininfo`, `headers`) and
verify it differentially against Core, without growing the surface.

1. Add differential tests over the socket harness for `/rest/chaininfo.json` and
`/rest/headers/*`, covering status codes, content types, `.json`/`.hex`/`.bin`
encodings, malformed hashes, out-of-range `count`, side branches, header-only
ranges, and the disabled-REST 404. Preserve the load-bearing 404-vs-400
distinction described in `docs/rest-interface.md`, because the enforcer depends
on it.

2. Update `docs/rest-interface.md` to state that REST is unauthenticated and to
define safe listener/exposure guidance. Add an enforcer-oriented fixture that
proves expected discovery and rescan behavior, reusing the wiring in
`tools/bip300301-enforcer/`. Keep the REST surface size unchanged: mark every
unimplemented Core REST route as `not_implemented` in the manifest, not as a new
route.
===============================================================================

Task: 5

Goal: make the ZMQ frame layout an enforced public contract against Core
fixtures, prove subscriber failures cannot touch consensus, and document the
projection posture honestly, keeping mempool A/R events explicitly out of scope.

1. Add fixture-based tests in `crates/node/tests/` (or extend the
`zmq_publisher.rs` test module) verifying, per supported topic, the multipart
frame layout, topic names, byte order, and the trailing little-endian `u32`
sequence counter against captured Core bytes. Make connect/disconnect (`C`/`D`)
ordering part of the documented `pubsequence` contract, matching the exact
`CONCEPTS.md` wording that disconnects go tip-first before connects. Verify
configured-topic discovery through `getzmqnotifications` matches the manifest
and the running endpoints.

2. Add tests for slow/disconnected subscribers, high-water-mark drop, socket
failure, reconnection, and sequence-counter wraparound. Assert that block apply
and disconnect in `crates/node/src/apply.rs` complete regardless of ZMQ delivery
outcome, so a notification failure never affects consensus.

3. Update `CONCEPTS.md` and `docs/rest-interface.md` to describe ZMQ as a
low-latency projection, not a durable replay mechanism, and to point event-loss
recovery at `#77`'s durable event/cursor model. State that mempool A/R
`pubsequence` parity remains unavailable until `#74`, and mark this in both the
manifest and `DEVIATIONS.md`.
===============================================================================

Task: 6

Goal: give clients a readiness vocabulary that distinguishes real
Core-equivalent data from partial node/index state, and expose it through
existing RPC surfaces consistent with the manifest.

1. Add a readiness state enum (`absent`, `building`, `caught_up`, `stale`,
`failed`) in the crate that owns index/node readiness. Do not put this in
`crates/index/src/status.rs`; that file handles Electrum status hashing and is
unrelated. Derive the state from existing signals: header tip versus applied
tip, index applied height, and failure/stall conditions.

2. Extend `getindexinfo` and `getblockchaininfo` in
`crates/rpc/src/handlers/chain.rs` to expose applied tip versus header tip,
per-index readiness state, pruning/body availability, configured ZMQ
topics/endpoints, and feature-disabled methods, staying within the manifest's
declared readiness behavior. Pin this new readiness output in the Phase 3.4
differential suite so it is verified, not merely present. Keep the readiness
reporting truthful: a partial index or partial node state must report its actual
state, never claim Core-equivalent completeness.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!