feat(storage): add RocksDB history lookup methods and owned batch type [2/3] #20543
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yongkangc
changed the title
yongkangc
force-pushed
the
yk/pr2-rocksdb-infrastructure
branch
from
e45acf9 to
40f84f2
Compare
yongkangc
force-pushed
the
yk/pr2-rocksdb-infrastructure
branch
from
7b937ec to
9aefeba
Compare
yongkangc
requested review from
joshieDo,
rakita and
shekhirin
as code owners
yongkangc
commented
Dec 22, 2025
| pub struct RocksDBBatch<'a> { | ||
| provider: &'a RocksDBProvider, | ||
| pub struct RocksDBBatch { | ||
| provider: Arc<RocksDBProviderInner>, |
Member
Author
There was a problem hiding this comment.
If RocksDBBatch<'a> borrows &'a RocksDBProvider, then DatabaseProvider would need to:
- Store both the
RocksDBProviderand theRocksDBBatch<'a>that borrows from it - This creates a self-referential struct
Collaborator
There was a problem hiding this comment.
isnt this just RocksDBProvider? RocksDBProvider is already an arc over an inner.
I think we might have to revisit this struct in another PR, because it feels odd
Above we specify
Uses
WriteBatchWithTransactionfor atomic writes without full transaction overhead.
but in fact RocksDbProviderInner holds a Transaction.
And if we're implementing read, maybe the best option would be to use the optimistic TX directly? unsure if we need to read through our writes here, but seems like a footgun if we do now or in the future
yongkangc
force-pushed
the
yk/pr2-rocksdb-infrastructure
branch
from
e885749 to
6b3de83
Compare
yongkangc
requested review from
gakonst,
klkvr,
mattsse and
mediocregopher
as code owners
yongkangc
force-pushed
the
yk/pr2-rocksdb-infrastructure
branch
4 times, most recently
from
fad81ba to
59af41c
Compare
yongkangc
force-pushed
the
yk/pr1-history-info-abstraction
branch
2 times, most recently
from
be6d534 to
bb4782b
Compare
Split the history lookup logic: - `needs_prev_shard_check`: pure helper to check if cursor.prev() is needed - `find_changeset_block_from_index`: pure const fn for final decision The cursor.prev() call uses && short-circuit at the call site, so it's only executed in the rare case where rank==0 and found_block != target. Also reorganized file to follow ordering convention: enums -> structs -> functions
…_index The new name better describes the function's purpose: given a history index shard, find which changeset block contains the historical value for a target block. Also moved the function below the impl block for better code organization.
Enhance RocksDB provider infrastructure for history lookups: - Change RocksDBBatch to own Arc<RocksDBProviderInner> instead of borrowing This allows storing RocksDBBatch in DatabaseProvider without lifetime issues. - Add RocksTx::account_history_info() and storage_history_info() methods These use the shared history_info_from_shard algorithm to lookup history in RocksDB with the same semantics as MDBX. - Update rocksdb_stub.rs with matching API (batch(), commit(), len(), is_empty()) This prepares the infrastructure for PR3 which will wire RocksDB into HistoricalStateProvider and DatabaseProvider HistoryWriter.
…_index in RocksDB
- Introduced `raw_iterator_cf` method for bidirectional traversal in RocksDB. - Updated `account_history_info` and `storage_history_info` methods to utilize raw iterators, enabling efficient previous shard detection. - Improved error handling and decoding for key/value retrieval in history lookups. - Added comprehensive tests for various scenarios, including single and multiple shard lookups, and edge cases for history queries.
- Enhanced `raw_iterator_cf` method to support bidirectional traversal and added `raw_iter_status_ok` for error checking. - Updated `account_history_info` and `storage_history_info` methods to utilize the new iterator features for efficient history lookups. - Improved documentation for clarity on how history lookups work with sharded data. - Added tests to validate the behavior of the new iterator methods and ensure correct handling of edge cases.
Change EitherWriter::RocksDB to borrow the batch instead of owning it. This allows callers to retain ownership and access the batch after the writer is dropped, which is needed for history lookups that must peek at pending writes. - RocksBatchArg changes from owned to `&'a mut RocksDBBatch` - Remove `into_raw_rocksdb_batch()` - no longer needed with borrowing - Update tests to use scoped writer pattern
Change EitherWriter::RocksDB to borrow the batch instead of owning it. This allows callers to retain ownership and access the batch after the writer is dropped, which is needed for history lookups that must peek at pending writes. - RocksBatchArg changes from owned to `&'a mut RocksDBBatch` - Remove `into_raw_rocksdb_batch()` - no longer needed with borrowing - Update tests to use scoped writer pattern
… RocksDBBatch Address review feedback: RocksDBProvider is already an Arc wrapper, so use it directly instead of duplicating the Arc access pattern.
Adapt RocksDB history functions to use the new closure-based has_previous_shard parameter from find_changeset_block_from_index. This aligns with the PR1 change to make cursor.prev() lazy.
yongkangc
force-pushed
the
yk/pr2-rocksdb-infrastructure
branch
from
59af41c to
f3021e2
Compare
yongkangc
force-pushed
the
yk/pr1-history-info-abstraction
branch
from
43961c9 to
03e7abb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add RocksDB infrastructure for history lookups without changing any behavior.
This is PR 2/3 of a stacked PR series that splits #20412 for easier review. Closes #20388
Depends on: #20542
Review focus:
RocksTxhistory lookup methodsRocksDBBatchownership change is safeChanges
rocksdb/provider.rsRocksDBBatchto ownArc<RocksDBProviderInner>instead of borrowingRocksDBBatchinDatabaseProviderwithout lifetime issuesRocksTx::account_history_info()methodStack
Did some testing and benchmark here: https://gist.github.com/yongkangc/6dfedca18b842cd41ea2864f248c1a07
MDBX wins by 2x but we are still using Rocksdb for the optmised writes.