Update for next release 10.0 (#13417)

Summary: Updated version, HISTORY, compatibility script and folly hash for 10.0 release. Included a HISTORY.md update backported from 10.0 branch: c1f63e1. Pull Request resolved: #13417 Test Plan: CI Reviewed By: hx235 Differential Revision: D70029393 Pulled By: cbi42 fbshipit-source-id: f8276bb31cc69648b47e0cbcd728d2a33fbf531f
facebook · Feb 24, 2025 · 3740fcc · 3740fcc
1 parent 4c975a7
commit 3740fcc
Show file tree

Hide file tree

Showing 16 changed files with 26 additions and 15 deletions.
diff --git a/HISTORY.md b/HISTORY.md
@@ -1,6 +1,29 @@
 # Rocksdb Change Log
 > NOTE: Entries for next release do not go here. Follow instructions in `unreleased_history/README.txt`
 
+## 10.0.0 (02/21/2025)
+### New Features
+* Introduced new `auto_refresh_iterator_with_snapshot` opt-in knob that (when enabled) will periodically release obsolete memory and storage resources for as long as the iterator is making progress and its supplied `read_options.snapshot` was initialized with non-nullptr value.
+* Added the ability to plug-in a custom table reader implementation. See include/rocksdb/external_table_reader.h for more details.
+* Experimental feature: RocksDB now supports FAISS inverted file based indices via the secondary indexing framework. Applications can use FAISS secondary indices to automatically quantize embeddings and perform K-nearest-neighbors similarity searches. See `FaissIVFIndex` and `SecondaryIndex` for more details. Note: the FAISS integration currently requires using the BUCK build.
+* Add new DB property `num_running_compaction_sorted_runs` that tracks the number of sorted runs being processed by currently running compactions
+* Experimental feature: added support for simple secondary indices that index the specified column as-is. See `SimpleSecondaryIndex` and `SecondaryIndex` for more details.
+* Added new `TransactionDBOptions::txn_commit_bypass_memtable_threshold`, which enables optimized transaction commit (see `TransactionOptions::commit_bypass_memtable`) when the transaction size exceeds a configured threshold.
+
+### Public API Changes
+* Updated the query API of the experimental secondary indexing feature by removing the earlier `SecondaryIndex::NewIterator` virtual and adding a `SecondaryIndexIterator` class that can be utilized by applications to find the primary keys for a given search target.
+* Added back the ability to leverage the primary key when building secondary index entries. This involved changes to the signatures of `SecondaryIndex::GetSecondary{KeyPrefix,Value}` as well as the addition of a new method `SecondaryIndex::FinalizeSecondaryKeyPrefix`. See the API comments for more details.
+* Minimum supported version of ZSTD is now 1.4.0, for code simplification. Obsolete `CompressionType` `kZSTDNotFinalCompression` is also removed.
+
+### Behavior Changes
+* `VerifyBackup` in `verify_with_checksum`=`true` mode will now evaluate checksums in parallel. As a result, unlike in case of original implementation, the API won't bail out on a very first corruption / mismatch and instead will iterate over all the backup files logging success / _degree_of_failure_ for each.
+* Reversed the order of updates to the same key in WriteBatchWithIndex. This means if there are multiple updates to the same key, the most recent update is ordered first. This affects the output of WBWIIterator. When WriteBatchWithIndex is created with `overwrite_key=true`, this affects the output only if Merge is used (#13387).
+* Added support for Merge operations in transactions using option `TransactionOptions::commit_bypass_memtable`.
+
+### Bug Fixes
+* Fixed GetMergeOperands() API in ReadOnlyDB and SecondaryDB
+* Fix a bug in `GetMergeOperands()` that can return incorrect status (MergeInProgress) and incorrect number of merge operands. This can happen when `GetMergeOperandsOptions::continue_cb` is set, both active and immutable memtables have merge operands and the callback stops the look up at the immutable memtable.
+
 ## 9.11.0 (01/17/2025)
 ### New Features
 * Introduce CancelAwaitingJobs() in CompactionService interface which will allow users to implement cancellation of running remote compactions from the primary instance

diff --git a/Makefile b/Makefile
@@ -2489,7 +2489,7 @@ checkout_folly:
 	fi
 	@# Pin to a particular version for public CI, so that PR authors don't
 	@# need to worry about folly breaking our integration. Update periodically
-	cd third-party/folly && git reset --hard f1c1542fb071e9d88181279ec24043400222aef1
+	cd third-party/folly && git reset --hard 78286282478e1ae05b2e8cbcf0e2139eab283bea
 	@# NOTE: this hack is required for clang in some cases
 	perl -pi -e 's/int rv = syscall/int rv = (int)syscall/' third-party/folly/folly/detail/Futex.cpp
 	@# NOTE: this hack is required for gcc in some cases

diff --git a/include/rocksdb/version.h b/include/rocksdb/version.h
@@ -12,7 +12,7 @@
 // NOTE: in 'main' development branch, this should be the *next*
 // minor or major version number planned for release.
 #define ROCKSDB_MAJOR 10
-#define ROCKSDB_MINOR 0
+#define ROCKSDB_MINOR 1
 #define ROCKSDB_PATCH 0
 
 // Do not use these. We made the mistake of declaring macros starting with

diff --git a/tools/check_format_compatible.sh b/tools/check_format_compatible.sh
@@ -125,7 +125,7 @@ EOF
 
 # To check for DB forward compatibility with loading options (old version
 # reading data from new), as well as backward compatibility
-declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb" "9.9.fb" "9.10.fb" "9.11.fb")
+declare -a db_forward_with_options_refs=("8.6.fb" "8.7.fb" "8.8.fb" "8.9.fb" "8.10.fb" "8.11.fb" "9.0.fb" "9.1.fb" "9.2.fb" "9.3.fb" "9.4.fb" "9.5.fb" "9.6.fb" "9.7.fb" "9.8.fb" "9.9.fb" "9.10.fb" "9.11.fb" "10.0.fb")
 # To check for DB forward compatibility without loading options (in addition
 # to the "with loading options" set), as well as backward compatibility
 declare -a db_forward_no_options_refs=() # N/A at the moment

diff --git a/unreleased_history/behavior_changes/parallelize_backup_verification.md b/unreleased_history/behavior_changes/parallelize_backup_verification.md
diff --git a/unreleased_history/behavior_changes/wbwi-ordering.md b/unreleased_history/behavior_changes/wbwi-ordering.md
diff --git a/unreleased_history/bug_fixes/get_merge_operand_fix.md b/unreleased_history/bug_fixes/get_merge_operand_fix.md
diff --git a/unreleased_history/bug_fixes/get_merge_operand_imm.md b/unreleased_history/bug_fixes/get_merge_operand_imm.md
diff --git a/unreleased_history/new_features/auto_refresh_iterator.md b/unreleased_history/new_features/auto_refresh_iterator.md
diff --git a/unreleased_history/new_features/external_table_reader.md b/unreleased_history/new_features/external_table_reader.md
diff --git a/unreleased_history/new_features/faiss_ivf_index.md b/unreleased_history/new_features/faiss_ivf_index.md
diff --git a/unreleased_history/new_features/num_running_compaction_sorted_runs.md b/unreleased_history/new_features/num_running_compaction_sorted_runs.md
diff --git a/unreleased_history/new_features/simple_secondary_index.md b/unreleased_history/new_features/simple_secondary_index.md
diff --git a/unreleased_history/public_api_changes/secondary_index_iterator.md b/unreleased_history/public_api_changes/secondary_index_iterator.md
diff --git a/unreleased_history/public_api_changes/secondary_index_primary_key.md b/unreleased_history/public_api_changes/secondary_index_primary_key.md
diff --git a/unreleased_history/public_api_changes/zstd.md b/unreleased_history/public_api_changes/zstd.md