feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544

yongkangc · 2025-12-21T00:11:34Z

Summary

Complete RocksDB integration for history tables by wiring everything together. Closes #20388

This is PR 3/3 of a stacked PR series that splits #20412 for easier review.

Depends on: #20543 → #20542

Review focus

Correctness: Do RocksDB history lookups match MDBX semantics
Batch handling: Is the shared batch + last-shard cache correct?
Deduplication: Unwind uses dedup for RocksDB's "last write wins" semantics

Stack

PR refactor(storage): extract shared find_changeset_block_from_index algorithm [1/3] #20542:
PR feat(storage): add RocksDB history lookup methods and owned batch type [2/3] #20543:
PR feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544:

yongkangc · 2025-12-22T08:38:58Z

crates/storage/provider/src/either_writer.rs

+    ///
+    /// Uses the rank/select logic to efficiently find the first block >= target
+    /// where the account was modified.
+    pub fn account_history_info(


this is the wiring up part, which is different from pr2 where its just rocks db

joshieDo · 2025-12-22T14:59:35Z

crates/storage/provider/src/providers/database/provider.rs

+    /// Single `RocksDB` batch for history index operations.
+    /// All history index writes accumulate here and commit together at provider commit time.
+    #[cfg(all(unix, feature = "rocksdb"))]
+    pending_rocks_batch: parking_lot::Mutex<crate::providers::rocksdb::RocksDBBatch>,
+    /// In-memory cache of history-index "last shards" (the `u64::MAX` shard that receives appends)
+    /// for the shared `RocksDB` batch.
+    ///
+    /// History index appends are read-modify-write: read last shard, merge new block numbers,
+    /// then write it back. With MDBX, the write transaction can read its own writes, so
+    /// subsequent appends see earlier updates. `RocksDBBatch` is write-only, so without this
+    /// cache each append would read only the persisted state and clobber earlier batch writes.
+    #[cfg(all(unix, feature = "rocksdb"))]
+    pending_history_index_last_shards: parking_lot::Mutex<PendingHistoryIndexLastShards>,
+    /// Pending `RocksDB` batches to be committed at provider commit time (from `EitherWriter`).
    #[cfg(all(unix, feature = "rocksdb"))]
    pending_rocksdb_batches: parking_lot::Mutex<Vec<rocksdb::WriteBatchWithTransaction<true>>>,


feels like rocksdb is leaking too much here

and it's not clear to me why we even need mutexes for this

joshieDo · 2025-12-22T15:00:54Z

crates/storage/provider/src/providers/database/provider.rs

+        // MDBX path
+        let mut cursor = self.tx.cursor_read::<tables::TransactionHashNumbers>()?;
+        Ok(cursor.seek_exact(tx_hash)?.map(|(_, v)| v))


Suggested change

// MDBX path

let mut cursor = self.tx.cursor_read::<tables::TransactionHashNumbers>()?;

Ok(cursor.seek_exact(tx_hash)?.map(|(_, v)| v))

Ok(self.tx.get::<tables::TransactionHashNumbers>(tx_hash)?)

joshieDo · 2025-12-22T15:01:28Z

crates/storage/provider/src/providers/database/provider.rs


            #[cfg(all(unix, feature = "rocksdb"))]
            {
+                // Commit the shared history index batch


Suggested change

// Commit the shared history index batch

joshieDo · 2025-12-22T15:02:03Z

crates/storage/provider/src/providers/database/provider.rs


            #[cfg(all(unix, feature = "rocksdb"))]
            {
+                // Commit the shared history index batch


Suggested change

// Commit the shared history index batch

joshieDo

i think overall this needs to be refactored so rocksdb logic is mostly contained in a rocksdb type, right now its leaking with the mutexes and/or methods ending with _rocksdb

This wires RocksDB into the history lookup paths: - Adds account_history_info and storage_history_info methods to EitherReader - Updates HistoricalStateProviderRef to use EitherReader for lookups - Adds RocksDBProviderFactory trait bounds to provider impls - Uses the pure find_changeset_block_from_index API with lazy cursor.prev()

github-project-automation bot added this to Reth Tracker Dec 21, 2025

github-project-automation bot moved this to Backlog in Reth Tracker Dec 21, 2025

yongkangc changed the title ~~feat(storage): wire RocksDB into history lookups and HistoryWriter~~ feat(storage): wire RocksDB into history lookups and HistoryWriter [3/3] Dec 21, 2025

yongkangc changed the title ~~feat(storage): wire RocksDB into history lookups and HistoryWriter [3/3]~~ feat(storage): wire RocksDB into DB Provider And Historical Provider Dec 21, 2025

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 3fe37d1 to 75a5a5d Compare December 21, 2025 00:20

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch from e45acf9 to 40f84f2 Compare December 21, 2025 02:12

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 75a5a5d to 28e5af4 Compare December 21, 2025 02:12

yongkangc mentioned this pull request Dec 21, 2025

feat(storage): Use EitherReader/EitherWriter in DatabaseProvider and HistoricalStateProvider #20412

Closed

yongkangc changed the title ~~feat(storage): wire RocksDB into DB Provider And Historical Provider~~ feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] Dec 21, 2025

This was referenced Dec 21, 2025

feat(storage): add RocksDB history lookup methods and owned batch type [2/3] #20543

Open

refactor(storage): extract shared find_changeset_block_from_index algorithm [1/3] #20542

Open

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from de43672 to 9e5131e Compare December 22, 2025 03:01

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch from 7b937ec to 9aefeba Compare December 22, 2025 03:05

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch 2 times, most recently from 27ea434 to 6f04c58 Compare December 22, 2025 04:51

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch from e885749 to 6b3de83 Compare December 22, 2025 04:55

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 6f04c58 to 48852bd Compare December 22, 2025 04:55

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch 2 times, most recently from 46f23c6 to 5a518d5 Compare December 22, 2025 06:06

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 48852bd to 1a553ac Compare December 22, 2025 07:47

yongkangc self-assigned this Dec 22, 2025

yongkangc marked this pull request as ready for review December 22, 2025 08:12

yongkangc requested review from joshieDo, mediocregopher, rakita and shekhirin as code owners December 22, 2025 08:12

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch 2 times, most recently from 9161ae0 to 7549845 Compare December 22, 2025 08:37

yongkangc commented Dec 22, 2025

View reviewed changes

joshieDo reviewed Dec 22, 2025

View reviewed changes

joshieDo requested changes Dec 22, 2025

View reviewed changes

github-project-automation bot moved this from Backlog to In Progress in Reth Tracker Dec 22, 2025

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch 2 times, most recently from 85460a0 to 9497ae7 Compare December 23, 2025 04:24

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch from fad81ba to 59af41c Compare December 23, 2025 04:45

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 9497ae7 to 878fc62 Compare December 23, 2025 04:45

yongkangc force-pushed the yk/pr2-rocksdb-infrastructure branch from 59af41c to f3021e2 Compare December 23, 2025 06:17

yongkangc force-pushed the yk/pr3-rocksdb-history-routing branch from 878fc62 to 0a6a08f Compare December 23, 2025 08:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544

feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544

yongkangc commented Dec 21, 2025 •

edited

Loading

Uh oh!

yongkangc Dec 22, 2025

Uh oh!

joshieDo Dec 22, 2025

Uh oh!

joshieDo Dec 22, 2025

Uh oh!

joshieDo Dec 22, 2025

Uh oh!

joshieDo Dec 22, 2025

Uh oh!

joshieDo Dec 22, 2025

Uh oh!

joshieDo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544

Are you sure you want to change the base?

feat(storage): wire RocksDB into DB Provider And Historical Provider [3/3] #20544

Conversation

yongkangc commented Dec 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yongkangc Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo Dec 22, 2025

Choose a reason for hiding this comment

Uh oh!

joshieDo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yongkangc commented Dec 21, 2025 •

edited

Loading