feat: introduce rocksdb blobdb #2251
Conversation
| @@ -693,6 +745,15 @@ pegasus_server_impl::pegasus_server_impl(dsn::replication::replica *r) | |||
| _data_cf_opts.max_bytes_for_level_base = FLAGS_rocksdb_max_bytes_for_level_base; | |||
| _data_cf_opts.max_bytes_for_level_multiplier = FLAGS_rocksdb_max_bytes_for_level_multiplier; | |||
|
|
|||
| // open db with key-value separation option (rocksdb blobdb) | |||
| _data_cf_opts.enable_blob_files = FLAGS_rocksdb_enable_blob_files; | |||
There was a problem hiding this comment.
What happen if enable this switch on an already running rocksdb instance? Will the data to be lost?
There was a problem hiding this comment.
enable_blob_files will affects flush/compaction task after your change, and will not interrupting ongoing background tasks.
If your turn it from 'true' to 'false', and at the same time enable_blob_garbage_collection=true.:
The background Blob GC thread will continue to scan old files. When the garbage rate of the entire file exceeds the threshold (blob_garbage_collection_force_threshold), the surviving entries in the file will be moved to the new SST (not the new blob), and then the original file will be deleted.
On the other hand, if enable_blob_garbage_collection=false, or the keys of this blob file remain unchanged for a long time, the blob file will stay in rocksdb for a long time.
There was a problem hiding this comment.
@ninsmiracle If enable_blob_files is set as true at the beginning, I wrote k1 -> v1 to a table. Then set it to false and reboot the replica servers, can I read the former k1 from the DB? How it act if the data in both SST and blob files?
| DSN_DEFINE_bool(pegasus.server, | ||
| rocksdb_enable_blob_files, | ||
| false, | ||
| "switch of the key-value separation function"); |
There was a problem hiding this comment.
Use the description in rocksdb, i.e. "When set, large values (blobs) are written to separate blob files, and only pointers to them are stored in SST files. This can reduce write amplification for large-value use cases at the cost of introducing a level of indirection for reads."
It could provide more information, avoid missunderstanding or ambiguity.
Other options are the same.
What problem does this PR solve?
#2201
What is changed and how does it work?
introduce rocksdb blobdb interface
Tests
Code changes
src/server/pegasus_server_impl_init.cpp