Add multi-database support to cluster mode #1671

xbasel · 2025-02-05T15:28:52Z

cluster: add multi-database support in cluster mode

Add multi-database support in cluster mode to align with standalone mode
and facilitate migration. Previously, cluster mode was restricted to a
single database (DB0). This change allows multiple databases while
preserving the existing slot-based key distribution.

Key Features:

Database-Agnostic Hashing. The hashing algorithm is unchanged.
Identical keys always map to the same slot across all databases,
ensuring consistent key distribution and compatibility with
existing single-database setups.
Multi-DB commands support. SELECT, MOVE, and COPY are now supported in
cluster mode.
Fully backward compatible with no API changes.
SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks.

Command-Level Changes:

SELECT / MOVE / COPY are now supported in cluster mode.
MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies.
SWAPDB will return an error if used when cluster mode is enabled.
GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database.
This means, for example, that migrating keys in a slot will require iterating and repeating across all databases.

Slot Migration Process:

Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases.

Transaction Handling (MULTI/EXEC):

getNodeByQuery key lookup behavior changed:
- No key lookups when queuing commands in MULTI, only cross-slot
  validation.
- Key lookups happen at EXEC time.
- SELECT inside MULTI/EXEC is now checked, ensuring key validation
  uses the selected DB at lookup.

Valkey-cli:

valkey-cli has been updated to support resharding across all databases.

Configuration:

Introduce new configuration cluster-databases.
The new configuration controls the maximal number of databases in cluster mode.

Implements #1319

codecov · 2025-02-05T15:45:24Z

Codecov Report

Attention: Patch coverage is 91.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 70.84%. Comparing base (2d200df) to head (f2ed97c).
Report is 7 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/valkey-cli.c	80.00%	8 Missing ⚠️
src/cluster.c	96.87%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1671      +/-   ##
============================================
- Coverage     71.02%   70.84%   -0.18%     
============================================
  Files           123      123              
  Lines         66116    66173      +57     
============================================
- Hits          46956    46879      -77     
- Misses        19160    19294     +134

Files with missing lines	Coverage Δ
src/cluster_legacy.c	`86.78% <100.00%> (+0.37%)`	⬆️
src/config.c	`78.39% <ø> (-0.05%)`	⬇️
src/db.c	`89.99% <100.00%> (+0.42%)`	⬆️
src/server.c	`87.94% <100.00%> (+0.03%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/valkey-benchmark.c	`62.42% <100.00%> (+0.24%)`	⬆️
src/cluster.c	`90.24% <96.87%> (+0.21%)`	⬆️
src/valkey-cli.c	`54.60% <80.00%> (-1.32%)`	⬇️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

src/db.c

src/cluster.c

soloestoy · 2025-02-13T09:07:19Z

I'm happy that we did "Unified db rehash method for both standalone and cluster #12848" when developing kvstore , which made the implementation of multi-database simpler.

hpatro

We need to add history to SWAPDB, SELECT, MOVE json files to indicate it's supported since 9.0.

src/cluster_legacy.c

tests/support/cluster.tcl

tests/cluster/tests/05-cluster-multidatabases.tcl

tests/unit/cluster/cli.tcl

src/cluster.c

src/db.c

tests/unit/lazyfree.tcl

src/db.c

src/cluster.c

xbasel · 2025-03-03T09:57:25Z

documentation: valkey-io/valkey-doc#242

hwware · 2025-03-05T14:59:08Z

It looks like there are still some test cases failed related to multiply db feature. Please fix them first, Thanks

Co-authored-by: Ran Shidlansik <[email protected]> Signed-off-by: xbasel <[email protected]>

Signed-off-by: xbasel <[email protected]>

src/cluster.c

Signed-off-by: xbasel <[email protected]>

madolson · 2025-05-04T04:12:47Z

I created issues for everything I noticed was a followup, wanted to merge to allow other PRs like atomic slot migration to rebase.

zuiderkwast · 2025-05-05T06:41:23Z

Almost all tests in Daily failed today. What happened here?

SELECT 1 failed: ERR DB index is out of range

https://github.com/valkey-io/valkey/actions/runs/14826557992

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR #1671. Fixes #2049 Signed-off-by: xbasel <[email protected]>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of #1671. --------- Signed-off-by: Madelyn Olson <[email protected]>

## cluster: add multi-database support in cluster mode Add multi-database support in cluster mode to align with standalone mode and facilitate migration. Previously, cluster mode was restricted to a single database (DB0). This change allows multiple databases while preserving the existing slot-based key distribution. ### Key Features: - Database-Agnostic Hashing. The hashing algorithm is unchanged. Identical keys always map to the same slot across all databases, ensuring consistent key distribution and compatibility with existing single-database setups. - Multi-DB commands support. SELECT, MOVE, and COPY are now supported in cluster mode. - Fully backward compatible with no API changes. - SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks. ### Command-Level Changes: - SELECT / MOVE / COPY are now supported in cluster mode. - MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies. - SWAPDB will return an error if used when cluster mode is enabled. - GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database. This means, for example, that migrating keys in a slot will require iterating and repeating across all databases. ### Slot Migration Process: - Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases. ### Transaction Handling (MULTI/EXEC): - getNodeByQuery key lookup behavior changed: - No key lookups when queuing commands in MULTI, only cross-slot validation. - Key lookups happen at EXEC time. - SELECT inside MULTI/EXEC is now checked, ensuring key validation uses the selected DB at lookup. ### Valkey-cli: - valkey-cli has been updated to support resharding across all databases. ### Configuration: - Introduce new configuration `cluster-databases`. The new configuration controls the maximal number of databases in cluster mode. Implements valkey-io#1319 --------- Signed-off-by: xbasel <[email protected]> Signed-off-by: zhaozhao.zz <[email protected]> Co-authored-by: zhaozhao.zz <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]> Co-authored-by: Ran Shidlansik <[email protected]>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Fixes valkey-io#2049 Signed-off-by: xbasel <[email protected]>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of valkey-io#1671. --------- Signed-off-by: Madelyn Olson <[email protected]>

To support multiple databases in cluster mode (see valkey-io#1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. Signed-off-by: Simon Baatz <[email protected]>

To support multiple databases in cluster mode (see #1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. --------- Signed-off-by: Simon Baatz <[email protected]> Signed-off-by: Simon Baatz <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]>

…#2206) To support multiple databases in cluster mode (see valkey-io#1671), `getNodeByQuery` temporarily switches databases when tracking `SELECT` statements during slot migration/import. The intended logic is to revert any database change after the operation. However, this approach is flawed: in some transactions the database change is not properly reverted, causing the client to remain on the wrong database. For example, if a transaction includes `SELECT` statements, the current database may be changed even if the transaction is never executed (see added test). Fix the issue by saving the original database once and restoring to it after a switch. --------- Signed-off-by: Simon Baatz <[email protected]> Signed-off-by: Simon Baatz <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]>

xbasel mentioned this pull request Feb 5, 2025

[NEW] Multiple DB supports in cluster mode #1319

Closed

xbasel marked this pull request as draft February 6, 2025 10:01

xbasel mentioned this pull request Feb 6, 2025

[NEW] Multi-database support in cluster mode - Implementation Plan #1681

Closed

xbasel marked this pull request as ready for review February 10, 2025 21:37

xbasel requested a review from zuiderkwast February 10, 2025 22:13

JoBeR007 reviewed Feb 11, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

soloestoy requested review from soloestoy and removed request for zuiderkwast February 12, 2025 06:28

soloestoy reviewed Feb 13, 2025

View reviewed changes

src/cluster.c Outdated Show resolved Hide resolved

ranshid added the release-notes This issue should get a line item in the release notes label Feb 17, 2025

hpatro reviewed Feb 17, 2025

View reviewed changes

src/cluster_legacy.c Outdated Show resolved Hide resolved

tests/support/cluster.tcl Outdated Show resolved Hide resolved

madolson reviewed Feb 17, 2025

View reviewed changes

ranshid added the client-changes-needed Client changes may be required for this feature label Feb 24, 2025

hwware reviewed Feb 26, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

hwware reviewed Feb 26, 2025

View reviewed changes

src/cluster.c Show resolved Hide resolved

xbasel added the documentation label Feb 26, 2025

xbasel force-pushed the multidb branch 2 times, most recently from 3dc03d6 to af2d46a Compare February 27, 2025 23:03

xbasel mentioned this pull request Mar 3, 2025

Multi-database support in cluster mode valkey-io/valkey-doc#242

Merged

xbasel requested a review from a team March 5, 2025 11:38

xbasel force-pushed the multidb branch from 0e9bb7a to f627245 Compare March 5, 2025 13:41

xbasel marked this pull request as draft March 5, 2025 18:36

xbasel force-pushed the multidb branch 4 times, most recently from 538e23e to 63151ae Compare March 6, 2025 12:14

xbasel and others added 3 commits April 29, 2025 14:10

Update src/cluster.c

01e39f8

Co-authored-by: Ran Shidlansik <[email protected]> Signed-off-by: xbasel <[email protected]>

optimize if statement in getNodeByQuery

d68702b

Signed-off-by: xbasel <[email protected]>

Merge remote-tracking branch 'origin/unstable' into multidb

34e6292

xbasel requested a review from ranshid April 29, 2025 12:01

ranshid reviewed Apr 29, 2025

View reviewed changes

src/cluster.c Outdated Show resolved Hide resolved

modify comment

f2ed97c

Signed-off-by: xbasel <[email protected]>

xbasel requested a review from ranshid April 29, 2025 18:38

zuiderkwast added the to-be-merged Almost ready to merge label May 3, 2025

madolson merged commit 2fe08f8 into valkey-io:unstable May 4, 2025
51 checks passed

github-project-automation bot moved this from In Progress to Done in Valkey 9.0 May 4, 2025

madolson removed the to-be-merged Almost ready to merge label May 4, 2025

This was referenced May 5, 2025

Add CLUSTER FLUSHSLOT command #1384

Merged

Test failure: Exception LOADING in valkey_deferring_client #2049

Closed

xbasel added a commit to xbasel/valkey that referenced this pull request May 6, 2025

Restore omitted singledb config in tests

ca6f0cd

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

xbasel mentioned this pull request May 6, 2025

Restore omitted singledb config in tests #2052

Merged

zuiderkwast mentioned this pull request May 6, 2025

Use the correct port for migration #2050

Merged

xbasel added a commit to xbasel/valkey that referenced this pull request May 6, 2025

Restore omitted singledb config in tests

4334d90

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

zuiderkwast pushed a commit that referenced this pull request May 6, 2025

Restore omitted singledb config in tests (#2052)

614980e

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR #1671. Fixes #2049 Signed-off-by: xbasel <[email protected]>

gmbnomis mentioned this pull request Jun 9, 2025

Multi-database support in cluster mode weakens script key declaration guarantees #2190

Open

gmbnomis mentioned this pull request Jun 11, 2025

Prevent getNodeByQuery from leaking DB changes into client #2206

Merged

zvi-code mentioned this pull request Jul 3, 2025

Support multi-db in CME valkey-io/valkey-search#204

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add multi-database support to cluster mode #1671

Add multi-database support to cluster mode #1671

Uh oh!

xbasel commented Feb 5, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

soloestoy commented Feb 13, 2025

Uh oh!

hpatro left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xbasel commented Mar 3, 2025

Uh oh!

hwware commented Mar 5, 2025

Uh oh!

Uh oh!

Uh oh!

madolson commented May 4, 2025

Uh oh!

zuiderkwast commented May 5, 2025

Uh oh!

Uh oh!

Add multi-database support to cluster mode #1671

Add multi-database support to cluster mode #1671

Uh oh!

Conversation

xbasel commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cluster: add multi-database support in cluster mode

Key Features:

Command-Level Changes:

Slot Migration Process:

Transaction Handling (MULTI/EXEC):

Valkey-cli:

Configuration:

Uh oh!

codecov bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

soloestoy commented Feb 13, 2025

Uh oh!

hpatro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xbasel commented Mar 3, 2025

Uh oh!

hwware commented Mar 5, 2025

Uh oh!

Uh oh!

Uh oh!

madolson commented May 4, 2025

Uh oh!

zuiderkwast commented May 5, 2025

Uh oh!

Uh oh!

xbasel commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading