Add multi-database support to cluster mode #1671

xbasel · 2025-02-05T15:28:52Z

cluster: add multi-database support in cluster mode

Add multi-database support in cluster mode to align with standalone mode
and facilitate migration. Previously, cluster mode was restricted to a
single database (DB0). This change allows multiple databases while
preserving the existing slot-based key distribution.

Key Features:

Database-Agnostic Hashing. The hashing algorithm is unchanged.
Identical keys always map to the same slot across all databases,
ensuring consistent key distribution and compatibility with
existing single-database setups.
Multi-DB commands support. SELECT, MOVE, and COPY are now supported in
cluster mode.
Fully backward compatible with no API changes.
SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks.

Command-Level Changes:

SELECT / MOVE / COPY are now supported in cluster mode.
MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies.
SWAPDB will return an error if used when cluster mode is enabled.
GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database.
This means, for example, that migrating keys in a slot will require iterating and repeating across all databases.

Slot Migration Process:

Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases.

Transaction Handling (MULTI/EXEC):

getNodeByQuery key lookup behavior changed:
- No key lookups when queuing commands in MULTI, only cross-slot
  validation.
- Key lookups happen at EXEC time.
- SELECT inside MULTI/EXEC is now checked, ensuring key validation
  uses the selected DB at lookup.

Valkey-cli:

valkey-cli has been updated to support resharding across all databases.

Configuration:

Introduce new configuration cluster-databases.
The new configuration controls the maximal number of databases in cluster mode.

Implements #1319

codecov · 2025-02-05T15:45:24Z

Codecov Report

Attention: Patch coverage is 91.00000% with 9 lines in your changes missing coverage. Please review.

Project coverage is 70.84%. Comparing base (2d200df) to head (f2ed97c).
Report is 7 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/valkey-cli.c	80.00%	8 Missing ⚠️
src/cluster.c	96.87%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1671      +/-   ##
============================================
- Coverage     71.02%   70.84%   -0.18%     
============================================
  Files           123      123              
  Lines         66116    66173      +57     
============================================
- Hits          46956    46879      -77     
- Misses        19160    19294     +134

Files with missing lines	Coverage Δ
src/cluster_legacy.c	`86.78% <100.00%> (+0.37%)`	⬆️
src/config.c	`78.39% <ø> (-0.05%)`	⬇️
src/db.c	`89.99% <100.00%> (+0.42%)`	⬆️
src/server.c	`87.94% <100.00%> (+0.03%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/valkey-benchmark.c	`62.42% <100.00%> (+0.24%)`	⬆️
src/cluster.c	`90.24% <96.87%> (+0.21%)`	⬆️
src/valkey-cli.c	`54.60% <80.00%> (-1.32%)`	⬇️

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JoBeR007 · 2025-02-11T10:09:32Z

src/db.c

@@ -1728,12 +1714,6 @@ void swapMainDbWithTempDb(serverDb *tempDb) {
 void swapdbCommand(client *c) {
    int id1, id2;

-    /* Not allowed in cluster mode: we have just DB 0 there. */


Would that be enough for swapdb to work in cluster mode? What will happen in setup with 2 shards, each responsible for half of slots in db's?

With this implementation SWAPDB must be executed in all primary nodes. There are three options:

Allow SWAPDB and shift responsibility to the user – Risky, non-atomic, can cause temporary inconsistency and data corruption. Needs strong warnings.

Keep SWAPDB disabled in cluster mode – Safest, avoids inconsistency.

Make SWAPDB cluster-wide and atomic or – Complex, unclear feasibility.

I think option 2 is the safest bet. @JoBeR007 wdyt?

Is SWAPDB replicated as a single command? Then it's atomic.

If it's risky, it's risky in standslone mode with replicas too, right?

I think we can allow it. Swapping the data can only be done in some non-realtime workloads anyway I think.

I think risky because of replication and risky because of the need to execute SWAPDB on all primary nodes are unrelated just because as a user you can't control first, but user is the main risk in the second case.
I would keep SWAPDB disabled in cluster mode, if we decide to continue with this implementation

In cluster mode, consistency is per slot.

Yes, FLUSHDB is very similar in this regard. If a failover happens just before this command has been propagated to replicas, it's a big thing, but it's no surprise I think. The client can use WAIT or check replication offset to make sure the FLUSHDB or SWAPDB was successful on the replicas.

Regarding this, I think it is not just an issue of Multi-database but is more related to atomic slot migration. If a shard is in a stable state (not undergoing slot migration), then flushdb/flushall/swapdb are safe. However, if slot migration is in progress, it might lead to data inconsistency.

I think this needs to be considered alongside atomic-slot-migration:

During the ATM process, for slots being migrated, if we encounter flushall/flushdb, we can send a command like flushslot or flushslotall to the target shard

As for swapdb, I recommend temporarily prohibiting execution during the ATM process

@PingXie @enjoy-binbin , please also take note of this.

make sense. @murphyjacob4 FYI

I made a comment on the issue about this, but also worth mentioning it's hard to orchestrate SWAPDB. Even in steady state, flushdb and flushall are idempotent (you can send them multiple times) but swapdb isn't. If a command times out on one node, it's hard to reason about if it was successful and how to retry it. I think we should continue to disable SWAPDB in cluster mode for now, unless we introduce an idempotent way to do the swap.

Maybe introduce UUID tracking for SWAPDB requests works.
disabling SWAPDB for now.

src/cluster.c

soloestoy · 2025-02-13T09:07:19Z

I'm happy that we did "Unified db rehash method for both standalone and cluster #12848" when developing kvstore , which made the implementation of multi-database simpler.

hpatro

We need to add history to SWAPDB, SELECT, MOVE json files to indicate it's supported since 9.0.

src/cluster_legacy.c

tests/support/cluster.tcl

tests/cluster/tests/05-cluster-multidatabases.tcl

tests/unit/cluster/cli.tcl

src/cluster.c

madolson · 2025-02-17T20:07:53Z

src/db.c

@@ -1728,12 +1714,6 @@ void swapMainDbWithTempDb(serverDb *tempDb) {
 void swapdbCommand(client *c) {
    int id1, id2;

-    /* Not allowed in cluster mode: we have just DB 0 there. */


I made a comment on the issue about this, but also worth mentioning it's hard to orchestrate SWAPDB. Even in steady state, flushdb and flushall are idempotent (you can send them multiple times) but swapdb isn't. If a command times out on one node, it's hard to reason about if it was successful and how to retry it. I think we should continue to disable SWAPDB in cluster mode for now, unless we introduce an idempotent way to do the swap.

tests/unit/lazyfree.tcl

src/db.c

src/cluster.c

xbasel · 2025-03-03T09:57:25Z

documentation: valkey-io/valkey-doc#242

hwware · 2025-03-05T14:59:08Z

It looks like there are still some test cases failed related to multiply db feature. Please fix them first, Thanks

Signed-off-by: xbasel <[email protected]>

madolson

I'm good with the current iteration.

src/cluster.c

Co-authored-by: Ran Shidlansik <[email protected]> Signed-off-by: xbasel <[email protected]>

Signed-off-by: xbasel <[email protected]>

src/cluster.c

Signed-off-by: xbasel <[email protected]>

madolson · 2025-05-04T04:12:47Z

I created issues for everything I noticed was a followup, wanted to merge to allow other PRs like atomic slot migration to rebase.

zuiderkwast · 2025-05-05T06:41:23Z

Almost all tests in Daily failed today. What happened here?

SELECT 1 failed: ERR DB index is out of range

https://github.com/valkey-io/valkey/actions/runs/14826557992

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR #1671. Fixes #2049 Signed-off-by: xbasel <[email protected]>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of #1671. --------- Signed-off-by: Madelyn Olson <[email protected]>

## cluster: add multi-database support in cluster mode Add multi-database support in cluster mode to align with standalone mode and facilitate migration. Previously, cluster mode was restricted to a single database (DB0). This change allows multiple databases while preserving the existing slot-based key distribution. ### Key Features: - Database-Agnostic Hashing. The hashing algorithm is unchanged. Identical keys always map to the same slot across all databases, ensuring consistent key distribution and compatibility with existing single-database setups. - Multi-DB commands support. SELECT, MOVE, and COPY are now supported in cluster mode. - Fully backward compatible with no API changes. - SWAPDB is not supported in cluster mode. It is unsafe due to inconsistency risks. ### Command-Level Changes: - SELECT / MOVE / COPY are now supported in cluster mode. - MOVE / COPY (with db) are rejected (TRYAGAIN error) during slot migration to prevent multi-DB inconsistencies. - SWAPDB will return an error if used when cluster mode is enabled. - GETKEYSINSLOT, COUNTKEYSINSLOT and MIGRATE will operate in the context of the selected database. This means, for example, that migrating keys in a slot will require iterating and repeating across all databases. ### Slot Migration Process: - Multi-DB support in cluster mode affects slot migration. Operators should now iterate over all the configured databases. ### Transaction Handling (MULTI/EXEC): - getNodeByQuery key lookup behavior changed: - No key lookups when queuing commands in MULTI, only cross-slot validation. - Key lookups happen at EXEC time. - SELECT inside MULTI/EXEC is now checked, ensuring key validation uses the selected DB at lookup. ### Valkey-cli: - valkey-cli has been updated to support resharding across all databases. ### Configuration: - Introduce new configuration `cluster-databases`. The new configuration controls the maximal number of databases in cluster mode. Implements valkey-io#1319 --------- Signed-off-by: xbasel <[email protected]> Signed-off-by: zhaozhao.zz <[email protected]> Co-authored-by: zhaozhao.zz <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]> Co-authored-by: Madelyn Olson <[email protected]> Co-authored-by: Ran Shidlansik <[email protected]>

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Fixes valkey-io#2049 Signed-off-by: xbasel <[email protected]>

One of the new tests that was added uses `CONFIG GET PORT`, which isn't right one for TLS. Also removed some other use of the helper which aren't actually used. Introduced as part of valkey-io#1671. --------- Signed-off-by: Madelyn Olson <[email protected]>

xbasel mentioned this pull request Feb 5, 2025

[NEW] Multiple DB supports in cluster mode #1319

Open

xbasel marked this pull request as draft February 6, 2025 10:01

xbasel mentioned this pull request Feb 6, 2025

[NEW] Multi-database support in cluster mode - Implementation Plan #1681

Closed

xbasel marked this pull request as ready for review February 10, 2025 21:37

xbasel requested a review from zuiderkwast February 10, 2025 22:13

JoBeR007 reviewed Feb 11, 2025

View reviewed changes

soloestoy requested review from soloestoy and removed request for zuiderkwast February 12, 2025 06:28

soloestoy reviewed Feb 13, 2025

View reviewed changes

src/cluster.c Outdated Show resolved Hide resolved

ranshid added the release-notes This issue should get a line item in the release notes label Feb 17, 2025

hpatro reviewed Feb 17, 2025

View reviewed changes

src/cluster_legacy.c Outdated Show resolved Hide resolved

tests/support/cluster.tcl Outdated Show resolved Hide resolved

madolson reviewed Feb 17, 2025

View reviewed changes

ranshid added the client-changes-needed Client changes may be required for this feature label Feb 24, 2025

hwware reviewed Feb 26, 2025

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

hwware reviewed Feb 26, 2025

View reviewed changes

src/cluster.c Show resolved Hide resolved

xbasel added the documentation label Feb 26, 2025

xbasel force-pushed the multidb branch 2 times, most recently from 3dc03d6 to af2d46a Compare February 27, 2025 23:03

xbasel mentioned this pull request Mar 3, 2025

Multi-database support in cluster mode valkey-io/valkey-doc#242

Open

xbasel requested a review from a team March 5, 2025 11:38

xbasel force-pushed the multidb branch from 0e9bb7a to f627245 Compare March 5, 2025 13:41

xbasel marked this pull request as draft March 5, 2025 18:36

xbasel force-pushed the multidb branch 4 times, most recently from 538e23e to 63151ae Compare March 6, 2025 12:14

Reformat.

89494cb

Signed-off-by: xbasel <[email protected]>

xbasel force-pushed the multidb branch from 7bf915b to 89494cb Compare April 28, 2025 10:53

madolson mentioned this pull request Apr 28, 2025

Automatically test a random DB for clustering tests #2021

Open

madolson approved these changes Apr 28, 2025

View reviewed changes

ranshid reviewed Apr 29, 2025

View reviewed changes

src/cluster.c Outdated Show resolved Hide resolved

src/cluster.c Outdated Show resolved Hide resolved

xbasel and others added 4 commits April 29, 2025 14:10

Update src/cluster.c

7bc3de7

Co-authored-by: Ran Shidlansik <[email protected]> Signed-off-by: xbasel <[email protected]>

Update src/cluster.c

01e39f8

Co-authored-by: Ran Shidlansik <[email protected]> Signed-off-by: xbasel <[email protected]>

optimize if statement in getNodeByQuery

d68702b

Signed-off-by: xbasel <[email protected]>

Merge remote-tracking branch 'origin/unstable' into multidb

34e6292

xbasel requested a review from ranshid April 29, 2025 12:01

ranshid reviewed Apr 29, 2025

View reviewed changes

src/cluster.c Outdated Show resolved Hide resolved

modify comment

f2ed97c

Signed-off-by: xbasel <[email protected]>

xbasel requested a review from ranshid April 29, 2025 18:38

zuiderkwast added the to-be-merged Almost ready to merge label May 3, 2025

madolson merged commit 2fe08f8 into valkey-io:unstable May 4, 2025
51 checks passed

github-project-automation bot moved this from In Progress to Done in Valkey 9.0 May 4, 2025

madolson removed the to-be-merged Almost ready to merge label May 4, 2025

This was referenced May 5, 2025

add cluster flushslot command. #1384

Open

Test failure: Exception LOADING in valkey_deferring_client #2049

Closed

xbasel added a commit to xbasel/valkey that referenced this pull request May 6, 2025

Restore omitted singledb config in tests

ca6f0cd

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

xbasel mentioned this pull request May 6, 2025

Restore omitted singledb config in tests #2052

Merged

zuiderkwast mentioned this pull request May 6, 2025

Use the correct port for migration #2050

Merged

xbasel added a commit to xbasel/valkey that referenced this pull request May 6, 2025

Restore omitted singledb config in tests

4334d90

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR valkey-io#1671. Signed-off-by: xbasel <[email protected]>

zuiderkwast pushed a commit that referenced this pull request May 6, 2025

Restore omitted singledb config in tests (#2052)

614980e

Re-adds a statement to restore the `singledb` config that was accidentally removed in PR #1671. Fixes #2049 Signed-off-by: xbasel <[email protected]>

Add multi-database support to cluster mode #1671

Add multi-database support to cluster mode #1671

Uh oh!

Conversation

xbasel commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

cluster: add multi-database support in cluster mode

Key Features:

Command-Level Changes:

Slot Migration Process:

Transaction Handling (MULTI/EXEC):

Valkey-cli:

Configuration:

Uh oh!

codecov bot commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xbasel Feb 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

soloestoy commented Feb 13, 2025

Uh oh!

hpatro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xbasel commented Mar 3, 2025

Uh oh!

hwware commented Mar 5, 2025

Uh oh!

madolson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

madolson commented May 4, 2025

Uh oh!

zuiderkwast commented May 5, 2025

Uh oh!

Uh oh!

xbasel commented Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 5, 2025 •

edited

Loading

xbasel Feb 11, 2025 •

edited

Loading