Chapter 18 — Cluster & Sharding

Part IV — Operations & Administration

← Chapter 17 — Sentinel & High Availability | Table of Contents | Chapter 19 — Security, ACL & TLS →

When a single Redis instance exceeds available RAM or CPU, Redis Cluster shards data across multiple masters using 16,384 hash slots. Each master owns a subset of slots; replicas provide HA per shard. This chapter covers cluster topology, slot assignment, cross-slot pitfalls, hash tags, resharding with redis-cli --cluster, and client cluster mode.

Cluster trades operational complexity for horizontal scale. Use it when one node cannot hold your working set or handle write throughput.

Recipe 18.1 — Cluster Topology and Minimum Viable Deployment

Problem

You need a production Redis Cluster but are unsure how many nodes, how slots are distributed, and how clients route commands.

Solution

Minimum production topology: 6 nodes (3 masters + 3 replicas):

┌─────────────────────────────────────────────────────────────┐
│                     Redis Cluster                            │
├──────────────┬──────────────┬──────────────┬──────────────┤
│ Master A     │ Master B     │ Master C     │              │
│ slots 0-5460 │ slots 5461-  │ slots 10923- │              │
│              │ 10922        │ 16383        │              │
├──────────────┼──────────────┼──────────────┼──────────────┤
│ Replica A'   │ Replica B'   │ Replica C'   │              │
│ (of Master A)│ (of Master B)│ (of Master C)│              │
└──────────────┴──────────────┴──────────────┴──────────────┘

Per-node config snippet (redis.conf):

port 6379
cluster-enabled yes
cluster-config-file nodes-6379.conf
cluster-node-timeout 5000
appendonly yes
dir /var/lib/redis

# Production hardening
cluster-require-full-coverage yes
cluster-allow-replica-migration yes

Create cluster (6 fresh instances on ports 6379–6384):

# Start 6 redis-server instances (different ports/hosts in production)
for port in 6379 6380 6381 6382 6383 6384; do
  redis-server /etc/redis/redis-$port.conf
done

# Create cluster — 3 masters, 1 replica each
redis-cli --cluster create \
  127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 \
  127.0.0.1:6382 127.0.0.1:6383 127.0.0.1:6384 \
  --cluster-replicas 1
# Type 'yes' to accept slot assignment

Inspect cluster state:

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_known_nodes:6
cluster_size:3

127.0.0.1:6379> CLUSTER NODES
# Node ID, flags (master/slave/myself), master-ref, ping, pong, config-epoch, link, slot ranges

Command routing:

Clients hash the key → slot → master node. Wrong node returns MOVED or ASK redirect:

127.0.0.1:6379> SET user:9999:name "Bob"
(error) MOVED 13779 127.0.0.1:6381

Cluster-aware clients follow redirects automatically.

Discussion

3 masters minimum for meaningful sharding and fault tolerance. Running 3 masters with 0 replicas is possible for dev but loses HA—one master failure loses its slot range until manual recovery.

cluster-require-full-coverage yes (default): if any slot is uncovered, the cluster rejects writes. Set no only for maintenance windows with operational understanding of partial outage.

Cross-AZ latency affects cluster-node-timeout. Increase to 15000–20000 ms for multi-AZ deployments to avoid false failovers.

Recipe 18.2 — Hash Slots and Key-to-Slot Mapping

Problem

You need to predict which node owns a key, debug routing issues, and design keys for even distribution.

Solution

Slot calculation:

Redis Cluster maps keys to slots using CRC16 mod 16384:

slot = CRC16(key) mod 16384

Only the portion between { and } (hash tag) is hashed if present (Recipe 18.4).

Compute slot in redis-cli:

127.0.0.1:6379> CLUSTER KEYSLOT user:1001
(integer) 11884
127.0.0.1:6379> CLUSTER KEYSLOT user:1002
(integer) 9842

Find which node owns a slot:

127.0.0.1:6379> CLUSTER NODES | grep master
# Look for slot ranges: 0-5460, 5461-10922, 10923-16383
127.0.0.1:6379> CLUSTER GETKEYSINSLOT 11884 10
# Lists up to 10 keys in slot 11884 (use cautiously in production)

Check slot distribution:

redis-cli --cluster check 127.0.0.1:6379
# Reports keys per node, slot coverage, replicas

Even distribution guidance:

# Good — high cardinality prefix, varying suffix
session:{uuid-v4}
cache:product:{sku}
user:{user_id}:profile

# Bad — all keys in one slot
config:global          # single key, single slot (OK if one key)
user:fixed-bucket      # if millions of users share one key name pattern without variation

Slot migration states during resharding:

State	Meaning
`MIGRATING`	Source node exporting slot
`IMPORTING`	Destination node receiving slot
`STABLE`	Normal operation

Discussion

16384 slots is a balance between metadata overhead and migration granularity. With 3 masters, each owns ~5461 slots regardless of key count—empty slots still belong to a master.

Hot keys on a single slot cannot be sharded further without splitting the value (e.g., local cache) or redesigning the key. Monitor per-node INFO commandstats and key frequency.

Recipe 18.3 — MGET, Multi-Key, and Cross-Slot Operations

Problem

Your application uses MGET, MSET, transactions, or Lua scripts spanning multiple keys. In Cluster mode, commands fail with CROSSSLOT errors.

Solution

The rule: Multi-key commands require all keys to hash to the same slot.

127.0.0.1:6379> MGET user:1:name user:2:name
(error) CROSSSLOT Keys in request don't hash to the same slot

Diagnose:

127.0.0.1:6379> CLUSTER KEYSLOT user:1:name
(integer) 9842
127.0.0.1:6379> CLUSTER KEYSLOT user:2:name
(integer) 11884
# Different slots → MGET fails

Fix with hash tags (Recipe 18.4):

127.0.0.1:6379> MGET user:{1001}:name user:{1001}:email user:{1001}:age
# All hash on "1001" → same slot

Application-level fan-out (when hash tags are inappropriate):

import redis.cluster

rc = redis.cluster.RedisCluster(host="127.0.0.1", port=6379)

def mget_cluster(keys: list[str]) -> dict:
    """Fetch keys individually — cluster client routes each."""
    pipe = rc.pipeline()
    for k in keys:
        pipe.get(k)
    values = pipe.execute()
    return dict(zip(keys, values))

Operations that require same slot:

Command / Feature	Same-slot required
`MGET`, `MSET`, `DEL` (multi-key)	Yes
`RENAME`, `RENAMENX`	Yes
`SUNION`, `SINTER`, `SDIFF`	Yes
`ZUNIONSTORE`, `ZINTERSTORE`	Yes
`MULTI`/`EXEC` transactions	Yes
Lua scripts touching multiple keys	Yes
`RPOPLPUSH` source and dest	Yes

Cluster-compatible alternatives:

Use hashes to colocate fields: HGETALL user:1001 instead of MGET user:1001:name user:1001:email
Use RedisJSON (module) for document fields on one key
Use Streams on one key for related events

Discussion

Pipelining unrelated keys still works—the client sends commands to appropriate nodes in parallel. The restriction is atomic multi-key operations, not throughput.

SCAN is per-node in Cluster. Use SCAN on each master or redis-cli --cluster call host:port KEYS pattern (avoid KEYS in production—use SCAN).

For UNLINK/DEL on many keys, group by slot in application code or accept per-key round trips.

Recipe 18.4 — Hash Tags for Colocation

Problem

You need related keys (or a Redis Set and its members' metadata) on the same slot for atomic operations or efficient multi-key access.

Solution

Syntax: Only the substring inside {...} is hashed. If no braces, the entire key is hashed.

127.0.0.1:6379> CLUSTER KEYSLOT user:{1001}:profile
(integer) 5526
127.0.0.1:6379> CLUSTER KEYSLOT user:{1001}:sessions
(integer) 5526
127.0.0.1:6379> CLUSTER KEYSLOT user:{1001}:settings
(integer) 5526
# All share tag "1001" → same slot

Atomic transaction example:

127.0.0.1:6379> MULTI
127.0.0.1:6379> HINCRBY user:{1001}:wallet balance -50
127.0.0.1:6379> HINCRBY user:{1001}:wallet pending 50
127.0.0.1:6379> EXEC

Leaderboard with related keys:

leaderboard:{2026-Q1}:scores    → ZADD
leaderboard:{2026-Q1}:metadata  → HSET
leaderboard:{2026-Q1}:participants → SADD
# MULTI/EXEC across all three works

Design guidelines:

Pattern	Tag choice	Risk
Per-user data	`{user_id}`	Hot user → hot slot
Per-tenant	`{tenant_id}`	Large tenant → imbalance
Global leaderboard	`{fixed}`	Entire leaderboard on one slot
Sharded leaderboard	`{shard_N}` where N = hash(user)%32	Better spread

Avoid tag explosion with hot spots:

# Bad — 10M users, celebrity user creates hot slot
key = f"feed:{{user:{celebrity_id}}}:posts"

# Better — partition celebrity feed across sub-tags
shard = hash(post_id) % 16
key = f"feed:{{celeb:{celebrity_id}:s{shard}}}:post:{post_id}"

Discussion

Hash tags are the primary tool for correctness (transactions) in Cluster. Overuse on high-cardinality hot entities creates hot slots—one master overloaded while others idle.

Monitor per-node ops/sec and latency. If one master is hot, revisit tag granularity.

Empty hash tag {} hashes empty string—all such keys land on same slot. Never use {} as a "default."

Recipe 18.5 — Resharding and Rebalancing

Problem

One master is at 90% memory while others are at 40%. You need to move hash slots without downtime.

Solution

Check current balance:

redis-cli --cluster check 127.0.0.1:6379
redis-cli --cluster info 127.0.0.1:6379
# Shows keys per master, slot counts

Add a new master (scale out):

# 1. Start new node with cluster-enabled yes
redis-server /etc/redis/redis-6385.conf

# 2. Add to cluster
redis-cli --cluster add-node 127.0.0.1:6385 127.0.0.1:6379

# 3. Rebalance slots to new master
redis-cli --cluster rebalance 127.0.0.1:6379 \
  --cluster-weight 127.0.0.1:6379=1 127.0.0.1:6380=1 \
  127.0.0.1:6381=1 127.0.0.1:6385=1 \
  --cluster-threshold 1

Move specific slot range manually:

# Move slots 0-1000 from 6379 to 6385
redis-cli --cluster reshard 127.0.0.1:6379
# Interactive prompts: source, dest, slot range, confirm

# Non-interactive
redis-cli --cluster reshard 127.0.0.1:6379 \
  --cluster-from <source-node-id> \
  --cluster-to <dest-node-id> \
  --cluster-slots 500 \
  --cluster-yes

Monitor migration:

127.0.0.1:6379> CLUSTER NODES
# migrating slot markers on source, importing on dest
127.0.0.1:6379> CLUSTER SETSLOT 500 MIGRATING <dest-node-id>  # advanced/manual

Resharding runbook:

PRE-FLIGHT
  [ ] cluster_state:ok
  [ ] All masters have replicas
  [ ] Backup recent (Chapter 15)
  [ ] Alert team: latency may spike during migration

EXECUTE
  [ ] rebalance or reshard with --cluster-use-empty-masters if needed
  [ ] Monitor application MOVED/ASK rate
  [ ] Verify: redis-cli --cluster check

POST
  [ ] Keys evenly distributed (±10% target)
  [ ] No slots in migrating/importing state
  [ ] Update capacity plans

Remove a node:

# Drain slots first
redis-cli --cluster reshard 127.0.0.1:6379 --cluster-from <node-id> --cluster-to <other-id> --cluster-slots all --cluster-yes
redis-cli --cluster del-node 127.0.0.1:6379 <node-id>

Discussion

Resharding moves keys online using MIGRATE. Large keys slow migration—consider MEMORY USAGE key before moving hot slots.

redis-cli --cluster fix repairs broken states (use with care, understand prompts). Always run --cluster check first.

Adding replicas before resharding protects against master failure during long migrations.

Recipe 18.6 — redis-cli --cluster Operations Reference

Problem

You need a practical cheat sheet for day-two cluster operations.

Solution

Essential commands:

# Health check
redis-cli --cluster check HOST:PORT

# Cluster summary
redis-cli --cluster info HOST:PORT

# Create cluster
redis-cli --cluster create NODE1 NODE2 NODE3 [NODE4 ...] --cluster-replicas 1

# Add/remove nodes
redis-cli --cluster add-node NEW_HOST:PORT EXISTING_HOST:PORT
redis-cli --cluster add-node NEW_HOST:PORT EXISTING_HOST:PORT --cluster-slave --cluster-master-id <id>
redis-cli --cluster del-node HOST:PORT NODE_ID

# Reshard / rebalance
redis-cli --cluster reshard HOST:PORT
redis-cli --cluster rebalance HOST:PORT --cluster-threshold 5

# Run command on all masters
redis-cli --cluster call HOST:PORT INFO memory

# Import external RDB (specialized — lab use)
redis-cli --cluster import HOST:PORT --cluster-from HOST:PORT --cluster-copy

Failover simulation:

# Manual failover (replica promotes, old master becomes replica)
redis-cli -h replica-host -p 6382 CLUSTER FAILOVER

# Force failover (primary partition scenario — use carefully)
redis-cli -h replica-host -p 6382 CLUSTER FAILOVER FORCE

Reset cluster (destructive — dev only):

redis-cli -h 127.0.0.1 -p 6379 CLUSTER RESET HARD
# Repeat on all nodes; then recreate cluster

Verify client sees correct topology:

127.0.0.1:6379> CLUSTER SLOTS
# Returns slot ranges with master/replica endpoints

Discussion

Always pass at least one known-good cluster node to --cluster commands; the CLI learns the full topology via CLUSTER SLOTS.

For TLS clusters (Chapter 19):

redis-cli --cluster check host:6379 \
  --tls --cert /path/client.crt --key /path/client.key --cacert /path/ca.crt

Recipe 18.7 — Client Cluster Mode

Problem

Your application connects to a single Redis URL but needs to work with a 6-node Cluster behind it.

Solution

Python (redis-py 4.x+):

from redis.cluster import RedisCluster

rc = RedisCluster(
    host="127.0.0.1",
    port=6379,
    decode_responses=True,
    # Or explicit startup nodes:
    # startup_nodes=[ClusterNode("10.0.1.10", 6379), ...]
)

rc.set("user:1001:name", "Alice")
rc.get("user:1001:name")

# Pipeline — batches per node
with rc.pipeline() as pipe:
    pipe.set("a", 1)
    pipe.set("b", 2)
    pipe.execute()

Node.js (ioredis Cluster):

const Redis = require('ioredis');

const cluster = new Redis.Cluster([
  { host: '127.0.0.1', port: 6379 },
  { host: '127.0.0.1', port: 6380 },
  { host: '127.0.0.1', port: 6381 },
], {
  redisOptions: { password: 'secret' },
  scaleReads: 'slave',  // read from replicas
});

await cluster.set('key', 'value');

Go (go-redis):

import "github.com/redis/go-redis/v9"

rdb := redis.NewClusterClient(&redis.ClusterOptions{
    Addrs: []string{
        "127.0.0.1:6379",
        "127.0.0.1:6380",
        "127.0.0.1:6381",
    },
    Password: "secret",
    RouteByLatency: true,
    RouteRandomly:  false,
})

Configuration checklist:

[ ] Use cluster-aware client (not single-node client + MOVED handling)
[ ] Provide multiple startup nodes (not just one)
[ ] Handle ClusterDownError during failover (retry with backoff)
[ ] Set maxRedirects (default 16 usually sufficient)
[ ] For read scaling: scaleReads / ReadOnly on replicas (accept stale reads)
[ ] Connection pool per node — monitor total connections × app instances
[ ] Avoid KEYS, FLUSHALL — use per-node admin tools

Connection math:

6 nodes × 50 app instances × 10 connections = 3000 connections cluster-wide
# Set maxclients accordingly on each node

Discussion

Non-cluster clients connecting to a random node will receive MOVED errors for most keys. Some proxies (Envoy, custom) hide Cluster from clients—prefer native cluster clients when possible for correctness.

During failover, slots are briefly unavailable. Clients should retry on CLUSTERDOWN and TRYAGAIN errors with exponential backoff.

Managed Redis (ElastiCache, Memorystore, Redis Cloud) often exposes a configuration endpoint—still requires cluster-aware client.

Recipe 18.8 — Cluster Failure Recovery and Slot Migration Pitfalls

Problem

After a node crash or operator error, the cluster shows cluster_state:fail, uncovered slots, or keys trapped in migrating state.

Solution

Diagnose:

redis-cli --cluster check 127.0.0.1:6379
# [ERR] Not all 16384 slots are covered by nodes.
# [WARNING] Node ... has slots in migrating state

Uncovered slots — replace failed master:

# If master A died and replica A' exists:
redis-cli -h replica-a CLUSTER FAILOVER
# Promotes replica; slots restored

# If no replica — add new node and rebalance from survivors (data loss on dead master's slots)
redis-cli --cluster add-node NEW:6379 EXISTING:6379
redis-cli --cluster fix EXISTING:6379  # follow prompts carefully

Stuck MIGRATING/IMPORTING state:

# On source node
127.0.0.1:6379> CLUSTER SETSLOT 500 STABLE

# On destination node
127.0.0.1:6385> CLUSTER SETSLOT 500 STABLE

Only run SETSLOT STABLE when no MIGRATE operations are in flight—verify with CLUSTER NODES.

Forgotten node in gossip:

redis-cli --cluster del-node LIVE:6379 DEAD_NODE_ID

Full cluster rebuild (last resort — data loss):

# On EACH node
redis-cli CLUSTER RESET HARD
# Recreate cluster from backups (Chapter 15) or empty cluster

Prevention checklist:

[ ] Every master has ≥1 replica
[ ] cluster-node-timeout appropriate for network
[ ] cluster-require-full-coverage yes in production
[ ] Automated alerts on cluster_state != ok
[ ] Regular redis-cli --cluster check in cron

Discussion

CLUSTER FAILOVER with FORCE promotes a replica even if the primary is reachable—use only when the primary is known bad and split-brain risk is accepted.

During resharding, clients handle ASK redirects for importing slots. If migration aborts mid-flight, keys may exist on both nodes temporarily—CLUSTER SETSLOT STABLE after verifying key counts.

Keep nodes.conf (cluster-config-file) backed up—it stores the cluster epoch and slot map. Corruption requires --cluster fix.

FilesExpand file tree

18-cluster-and-sharding.md

Latest commit

History

18-cluster-and-sharding.md

File metadata and controls

Chapter 18 — Cluster & Sharding

Recipe 18.1 — Cluster Topology and Minimum Viable Deployment

Problem

Solution

Discussion

See Also

Recipe 18.2 — Hash Slots and Key-to-Slot Mapping

Problem

Solution

Discussion

See Also

Recipe 18.3 — MGET, Multi-Key, and Cross-Slot Operations

Problem

Solution

Discussion

See Also

Recipe 18.4 — Hash Tags for Colocation

Problem

Solution

Discussion

See Also

Recipe 18.5 — Resharding and Rebalancing

Problem

Solution

Discussion

See Also

Recipe 18.6 — redis-cli --cluster Operations Reference

Problem

Solution

Discussion

See Also

Recipe 18.7 — Client Cluster Mode

Problem

Solution

Discussion

See Also

Recipe 18.8 — Cluster Failure Recovery and Slot Migration Pitfalls

Problem

Solution

Discussion

See Also