memq

High-performance Bitcoin mempool query engine with real-time fee estimation over gRPC and shared memory.

What it does

memq subscribes to a local bitcoind via ZMQ (rawtx and sequence topics), reconstructs the mempool in-process using a lock-free DashMap, and serves fee estimates through a tonic gRPC interface. Fee estimation is mempool-based — it simulates greedy miner block assembly over the current mempool state, rather than relying on historical confirmation data the way Bitcoin Core's estimatesmartfee does. This is the same conceptual approach used by mempool.space, though the implementation here is a standalone Rust process with ~58 ns in-process estimation latency (see Performance) and sub-millisecond end-to-end gRPC round-trip latency, instead of a full web stack. For targets beyond 6 blocks the estimator blends in Core's estimatesmartfee as a fallback, since the live mempool provides diminishing signal at longer horizons.

Key capabilities

• Cluster Mempool v1. Tracks transaction dependency graphs, groups connected transactions into clusters, linearizes them using a greedy ancestor-feerate heuristic, and decomposes the linearization into monotone-feerate chunks — the same conceptual model being adopted by Bitcoin Core (PR #33629).
• Feerate-diagram RBF validation. Replacement transactions are accepted only when their feerate diagram strictly dominates the diagram of the transactions they evict.
• CPFP-aware estimation. Calculates the exact child fee rate needed to bump a stuck parent (and its ancestor package) into a target block.
• Probabilistic confirmation prediction. Blends live mempool simulation with an online calibration model to produce per-fee-rate cumulative confirmation distributions.
• Fee pressure forecasting. Tracks the rate-of-change of the mempool fee distribution with dual EMA windows to detect surges and RBF storms.
• Shared memory interface. Publishes hot state (buckets, projections) to a POSIX SHM segment with a seqlock for sub-100 ns reads from co-located processes — no network, no serialization.

Performance

Measured with criterion on commodity hardware (profile.bench inherits opt-level = 3, lto = "fat", codegen-units = 1).

Operation	Latency	Notes
Fee estimation (in-process)	~58 ns	O(193) lock-free atomic bucket sweep
Fee estimation (gRPC e2e)	~100–150 µs	Includes protobuf serialization + TCP loopback
Incremental bucket update	~12 ns	O(1) single atomic bucket increment/decrement
Tx decode (consensus)	~326 ns	Raw bytes → TxEntry, zero recomputation
Mempool insert	~90 ns/tx	DashMap insert + conflict index + atomic stats
Decode throughput	~3.3M tx/sec	Sustained, single thread

Thread model

The process runs 6+ OS threads with separated responsibilities:

Thread(s)	Role
ZMQ rawtx	Subscribes to raw transaction broadcasts, decodes, forwards over crossbeam channel
ZMQ sequence	Subscribes to membership events (Add/Remove) and block notifications (Connect/Disconnect)
Processor	Single consumer of both ZMQ channels; applies mempool mutations to the DashMap mirror, manages cluster linearization
Hydration worker(s)	Fetches fee-relevant data (ancestor info) from bitcoind RPC for unhydrated transactions. Count configurable via [hydration] workers (default 1)
gRPC server (x2)	tonic on a 2-thread tokio runtime serving estimation, simulation, CPFP, prediction, and pressure RPCs

The sequence topic is the sole membership authority — a transaction is only inserted once both the rawtx payload and the sequence:A event arrive (correlated by txid in a pending matcher).

Module structure

src/
├── accuracy/      # Block template comparison (Jaccard similarity + fee error EMA, α=0.1)
├── config/        # TOML configuration (ZMQ endpoints, RPC credentials, gRPC bind, tuning)
├── cpfp/          # CPFP-aware child fee estimation for stuck parent transactions
├── estimator/     # Blended estimator: mempool for ≤6 blocks, Core estimatesmartfee for >6
├── fee_engine/    # 193 hybrid log-scale buckets, O(1) insert/remove, O(193) estimation
│   ├── mod.rs     # Bucketed fee-rate distribution and block simulation
│   └── package.rs # Cluster-aware block assembly using pre-linearized chunks
├── forecast/      # Fee pressure forecasting and RBF surge detection (dual EMA windows)
├── ingestion/     # ZMQ subscribers + consensus decoder
│   ├── decoder.rs # Raw bytes → TxEntry (zero recomputation)
│   └── zmq_sub.rs # Dedicated threads for rawtx/sequence
├── interface/     # gRPC server (tonic) + shared memory publisher
│   ├── mod.rs     # MemqService implementation (10 RPCs)
│   └── shm.rs     # POSIX SHM seqlock for ultra-low latency local reads
├── mempool/       # Concurrent mempool mirror with cluster linearization
│   ├── mod.rs     # DashMap (16 shards) facade, insert/remove/hydrate
│   ├── types.rs   # MempoolEntry, InsertOutcome, HydratedMeta, etc.
│   ├── rbf.rs     # RBF victim detection + feerate-diagram-based replacement validation
│   ├── dep_graph.rs # Parent/child DAG: ancestor/descendant closure, connected components
│   ├── cluster.rs # Cluster lifecycle management (create, merge, split, limits)
│   ├── linearize.rs # Greedy ancestor-feerate linearization heuristic
│   ├── chunking.rs  # Monotone-feerate chunk construction from linearizations
│   └── feefrac.rs   # Integer fee/size fractions for overflow-safe feerate comparison
├── metrics/       # Prometheus metric export (counters, gauges, histogram) on [::1]:9100
├── prediction/    # Probabilistic confirmation time prediction (logistic + calibration EMA)
├── sync/          # RPC-driven initial sync, block reconciliation, full resync
├── processor.rs   # Single-writer event loop: correlation, hydration dispatch, block handling
├── cli.rs         # clap CLI argument definitions
├── lib.rs         # Library root (public modules for benchmarks)
└── main.rs        # Entry point, thread wiring, shutdown coordination

Quick start

# 1. Configure bitcoind (add to bitcoin.conf)
zmqpubrawtx=tcp://127.0.0.1:28332
zmqpubsequence=tcp://127.0.0.1:28333

# 2. Build and run
cargo build --release
./target/release/memq

# 3. Query fee estimate
grpcurl -plaintext -d '{"target_blocks": 1}' '[::1]:50051' memq.MemqService/EstimateFee

# 4. Check Prometheus metrics
curl http://[::1]:9100/metrics

Configuration is via memq.toml. See docs/setup.md for full options.

gRPC API

Defined in proto/memq.proto. Service: memq.MemqService.

RPC	Request field(s)	Returns
EstimateFee	target_blocks: u32	Fee rate (sat/vB), confidence, source (MEMPOOL / BITCOIN_CORE / MEMPOOL_FALLBACK), both underlying rates
GetMempoolStats	(none)	tx_count, total_vsize_bytes, total_fees_sat
GetFeeHistogram	num_buckets: u32	Bucketed fee-rate distribution (min/max rate, tx count, total vsize per bucket)
SimulateBlocks	num_blocks: u32	Cluster-aware block assembly projection: tx count, vsize, fees, min fee rate per block
GetTransaction	txid: string (hex)	Fee rate, fee, vsize, weight, RBF signal, receive timestamp, depends, spentby, ancestor/descendant counts
StreamFeeEstimates	target_blocks, interval_ms	Server-streaming fee estimates at a configurable interval (100 ms – 60 s)
EstimateConfirmationDistribution	fee_rate_sat_per_vb, max_blocks	Probabilistic confirmation distribution: marginal and cumulative probabilities per block
EstimateCpfpFee	parent_txid, target_blocks, child_vsize	Required child fee rate/fee to bump a stuck parent, package stats, already-sufficient flag
GetFeePressure	(none)	Surge score, high-fee inflow rate, RBF rate, floor delta, mempool growth rate
StreamFeePressure	interval_ms	Server-streaming fee pressure updates (500 ms – 60 s)

Shared memory interface

For co-located processes that need absolute minimum latency, memq publishes its state to a POSIX shared memory segment (/dev/shm/memq) using a seqlock for tear-free reads without inter-process mutexes. The published state includes mempool summary statistics, the full 193-bucket vsize distribution, and projections for the next 12 blocks.

See examples/shm_reader.rs for a consumer example.

Key design decisions

• DashMap over RwLock<HashMap> — 16-shard concurrent hashmap gives lock-free reads from gRPC threads; writes lock only the affected shard.
• crossbeam-channel over tokio::mpsc — bounded channel (default capacity 65 536) between OS-threaded ZMQ subscribers and the processor; avoids pulling the async runtime into the ingestion path.
• Fee buckets over sorted tree — 193 fixed buckets give O(1) insert/remove and O(193) estimation. Sorted views are built on-demand only for explicit SimulateBlocks calls (O(n log n)).
• ZMQ over RPC polling — push-based delivery eliminates polling latency and reduces bitcoind load. Dual-subscriber architecture (rawtx for payload, sequence for membership) avoids the race conditions of relying on a single topic.
• Cluster linearization — connected transaction components are linearized using a greedy ancestor-feerate heuristic, then decomposed into monotone-feerate chunks. RBF replacements are validated by comparing feerate diagrams (CompareChunks), matching the direction of Bitcoin Core's cluster mempool proposal.
• Feerate-diagram RBF — replacements must produce a feerate diagram that strictly dominates the diagram of the evicted set, preventing relay free-riding without relying on absolute-fee rules alone.
• Two-phase RBF — conflicts are collected from a separate OutPoint → Txid index without mutating state, validated, then applied atomically.
• Accuracy tracking — Jaccard similarity (|projected ∩ actual| / |projected ∪ actual|) and fee-rate prediction error, smoothed with EMA (α = 0.1, ~10-block window).

Limitations

• Heuristic linearization. The cluster linearizer uses a greedy ancestor-feerate approach — not Bitcoin Core's optimal SFL algorithm. Multi-child competition (two children competing to pull the same parent) is resolved first-come-first-served. There is no chunk-based eviction ordering yet.
• No historical confirmation data. The mempool-based estimator has no memory of past fee markets. For targets > 6 blocks, it falls back to Core's estimatesmartfee.

Links

• Architecture documentation — thread model, data structures, performance analysis, design tradeoffs
• Setup guide — bitcoind configuration, memq.toml reference, deployment notes
• Testing guide — step-by-step manual testing on signet and regtest

License

MIT