-
Notifications
You must be signed in to change notification settings - Fork 314
feat: introduce rocksdb blobdb #2251
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
@@ -693,6 +745,15 @@ pegasus_server_impl::pegasus_server_impl(dsn::replication::replica *r) | |||
_data_cf_opts.max_bytes_for_level_base = FLAGS_rocksdb_max_bytes_for_level_base; | |||
_data_cf_opts.max_bytes_for_level_multiplier = FLAGS_rocksdb_max_bytes_for_level_multiplier; | |||
|
|||
// open db with key-value separation option (rocksdb blobdb) | |||
_data_cf_opts.enable_blob_files = FLAGS_rocksdb_enable_blob_files; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happen if enable this switch on an already running rocksdb instance? Will the data to be lost?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
enable_blob_files
will affects flush/compaction task after your change, and will not interrupting ongoing background tasks.
If your turn it from 'true' to 'false', and at the same time enable_blob_garbage_collection=true
.:
The background Blob GC thread will continue to scan old files. When the garbage rate of the entire file exceeds the threshold (blob_garbage_collection_force_threshold
), the surviving entries in the file will be moved to the new SST (not the new blob), and then the original file will be deleted.
On the other hand, if enable_blob_garbage_collection=false
, or the keys of this blob file remain unchanged for a long time, the blob file will stay in rocksdb for a long time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ninsmiracle If enable_blob_files
is set as true at the beginning, I wrote k1
-> v1
to a table. Then set it to false and reboot the replica servers, can I read the former k1
from the DB? How it act if the data in both SST and blob files?
DSN_DEFINE_bool(pegasus.server, | ||
rocksdb_enable_blob_files, | ||
false, | ||
"switch of the key-value separation function"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use the description in rocksdb, i.e. "When set, large values (blobs) are written to separate blob files, and only pointers to them are stored in SST files. This can reduce write amplification for large-value use cases at the cost of introducing a level of indirection for reads."
It could provide more information, avoid missunderstanding or ambiguity.
Other options are the same.
What problem does this PR solve?
#2201
What is changed and how does it work?
introduce rocksdb blobdb interface
Tests
Code changes
src/server/pegasus_server_impl_init.cpp