Skip to content

feat: introduce rocksdb blobdb #2251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ninsmiracle
Copy link
Contributor

What problem does this PR solve?

#2201

What is changed and how does it work?

introduce rocksdb blobdb interface

Tests
  • Unit test
  • Manual test (add detailed scripts or steps below)
Code changes
  • Has interface methods change in src/server/pegasus_server_impl_init.cpp

@github-actions github-actions bot added the cpp label May 19, 2025
@@ -693,6 +745,15 @@ pegasus_server_impl::pegasus_server_impl(dsn::replication::replica *r)
_data_cf_opts.max_bytes_for_level_base = FLAGS_rocksdb_max_bytes_for_level_base;
_data_cf_opts.max_bytes_for_level_multiplier = FLAGS_rocksdb_max_bytes_for_level_multiplier;

// open db with key-value separation option (rocksdb blobdb)
_data_cf_opts.enable_blob_files = FLAGS_rocksdb_enable_blob_files;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happen if enable this switch on an already running rocksdb instance? Will the data to be lost?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

enable_blob_files will affects flush/compaction task after your change, and will not interrupting ongoing background tasks.
If your turn it from 'true' to 'false', and at the same time enable_blob_garbage_collection=true.:
The background Blob GC thread will continue to scan old files. When the garbage rate of the entire file exceeds the threshold (blob_garbage_collection_force_threshold), the surviving entries in the file will be moved to the new SST (not the new blob), and then the original file will be deleted.
On the other hand, if enable_blob_garbage_collection=false, or the keys of this blob file remain unchanged for a long time, the blob file will stay in rocksdb for a long time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ninsmiracle If enable_blob_files is set as true at the beginning, I wrote k1 -> v1 to a table. Then set it to false and reboot the replica servers, can I read the former k1 from the DB? How it act if the data in both SST and blob files?

DSN_DEFINE_bool(pegasus.server,
rocksdb_enable_blob_files,
false,
"switch of the key-value separation function");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the description in rocksdb, i.e. "When set, large values (blobs) are written to separate blob files, and only pointers to them are stored in SST files. This can reduce write amplification for large-value use cases at the cost of introducing a level of indirection for reads."

It could provide more information, avoid missunderstanding or ambiguity.

Other options are the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants