Chapter 17 — Sentinel & High Availability

Part IV — Operations & Administration

← Chapter 16 — Replication | Table of Contents | Chapter 18 — Cluster & Sharding →

Running a primary and replica pair protects against data loss on one node, but someone must detect primary failure and promote a replica. Redis Sentinel automates monitoring, notification, and failover for non-Cluster deployments. This chapter covers Sentinel architecture, sentinel.conf, the failover state machine, client discovery, split-brain mitigation, and production monitoring.

Sentinel is the right HA layer when your dataset fits on one primary and you do not need horizontal sharding (Chapter 18).

Recipe 17.1 — Understand Sentinel Architecture

Problem

You need automatic failover but are unsure how Sentinel differs from Cluster, how many Sentinels to deploy, or what "quorum" means.

Solution

Components:

                    ┌─────────────┐
                    │  Sentinel 1 │
                    └──────┬──────┘
                           │ gossip + monitoring
┌─────────────┐    ┌───────┴───────┐    ┌─────────────┐
│  Sentinel 2 │────│   Primary     │────│  Replica 1  │
└─────────────┘    │  (master)     │    └─────────────┘
       │           └───────┬───────┘           │
       │                   │                   │
┌──────┴──────┐            │            ┌──────┴──────┐
│  Sentinel 3 │────────────┴────────────│  Replica 2  │
└─────────────┘                         └─────────────┘

Sentinels monitor primaries/replicas via Redis protocol (PING, INFO).
They exchange state on a Sentinel bus (pub/sub on port 26379 by default).
When enough Sentinels agree the primary is subjectively down (SDOWN), one initiates failover.
After objective down (ODOWN), a replica is promoted; other replicas reconfigured; clients notified.

Minimum production deployment:

Component	Count	Notes
Sentinels	3 (odd)	Tolerates 1 Sentinel failure; quorum typically 2
Primary	1	Serves writes
Replicas	≥2	Failover target + read scaling

Sentinel vs Cluster:

Feature	Sentinel + Replication	Redis Cluster
Sharding	No — single dataset	Yes — 16,384 hash slots
Failover	Automatic via Sentinel	Automatic per shard
Client complexity	Moderate (Sentinel discovery)	Higher (slot routing)
Max dataset	Single node RAM	Sum of all master RAM

Check Sentinel view:

redis-cli -p 26379 SENTINEL master mymaster
redis-cli -p 26379 SENTINEL replicas mymaster
redis-cli -p 26379 SENTINEL sentinels mymaster

Discussion

Sentinels are lightweight but must run on independent failure domains—not all three on the same VM as the primary. Spread across availability zones when possible.

Sentinel does not store data; it stores configuration state in memory and persists to sentinel.conf on failover (auto-rewrite).

Recipe 17.2 — Configure sentinel.conf

Problem

You need a working Sentinel deployment for a master named mymaster with three Sentinel instances.

Solution

Primary (redis.conf):

bind 0.0.0.0
port 6379
requirepass redis-secret
masterauth redis-secret
dir /var/lib/redis
appendonly yes

Replica (redis.conf):

port 6379
requirepass redis-secret
masterauth redis-secret
replicaof 10.0.1.10 6379
replica-priority 100

Sentinel (sentinel.conf) — deploy identical on 3 hosts, change bind/IP:

port 26379
bind 0.0.0.0
sentinel monitor mymaster 10.0.1.10 6379 2
sentinel auth-pass mymaster redis-secret
sentinel down-after-milliseconds mymaster 5000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 60000
sentinel deny-scripts-reconfig yes

# Optional notification
# sentinel notification-script mymaster /var/redis/notify.sh
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh

dir /var/lib/redis-sentinel
logfile /var/log/redis/sentinel.log

Key directives explained:

Directive	Meaning
`monitor mymaster 10.0.1.10 6379 2`	Watch primary; quorum = 2 Sentinels must agree for ODOWN
`down-after-milliseconds 5000`	Mark SDOWN if no response for 5s
`parallel-syncs 1`	During failover, resync 1 replica at a time to primary
`failover-timeout 60000`	Failover state machine timeout (ms)
`auth-pass`	Password for primary/replicas

Start Sentinel:

redis-sentinel /etc/redis/sentinel.conf
# or
redis-server /etc/redis/sentinel.conf --sentinel

Verify quorum and master info:

127.0.0.1:26379> SENTINEL master mymaster
 1) "name"
 2) "mymaster"
 3) "ip"
 4) "10.0.1.10"
 5) "flags"
 6) "master"
 7) "num-slaves"
 8) "2"
 9) "num-other-sentinels"
10) "2"
11) "quorum"
12) "2"
13) "failover-state"
14) "no-failover"

Runtime reconfiguration:

127.0.0.1:26379> SENTINEL SET mymaster down-after-milliseconds 10000
127.0.0.1:26379> SENTINEL SET mymaster parallel-syncs 2

Discussion

The quorum in sentinel monitor is not the number of Sentinels required to run—it's how many must agree the master is unreachable to enter ODOWN. With 3 Sentinels and quorum 2, one misbehaving Sentinel cannot trigger failover alone.

sentinel deny-scripts-reconfig yes (default in modern Redis) prevents dangerous script injection via SENTINEL SET.

After failover, Sentinels rewrite sentinel.conf with the new primary address. Ensure the file is writable by the redis user.

Recipe 17.3 — Failover Process and Runbook

Problem

The primary crashed during peak traffic. You need to understand what Sentinel does automatically and what operators should (and should not) do.

Solution

Automatic failover state machine (simplified):

1. SDOWN  — one Sentinel thinks master is down
2. ODOWN  — quorum agrees master is down
3. Failover started — Sentinel elected as leader runs failover
4. Select replica — filter by priority, offset, runid
5. REPLICAOF NO ONE on chosen replica → new primary
6. Reconfigure other replicas → REPLICAOF new-primary
7. Update Sentinel config; publish +switch-master
8. Failover complete

Observe live failover:

# Watch Sentinel logs
tail -f /var/log/redis/sentinel.log

# Monitor flags
watch -n1 'redis-cli -p 26379 SENTINEL master mymaster | grep -E "flags|ip|failover"'

Simulate primary failure (lab only):

# Kill primary process
redis-cli -h 10.0.1.10 -a redis-secret DEBUG sleep 0 &
kill -9 $(pgrep -f "redis-server.*6379")

# Sentinels detect within down-after-milliseconds
# Failover completes within failover-timeout (typically seconds)
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
# Returns new primary IP:port

Operator runbook — primary failure:

DETECT
  [ ] Alert: master_link down / Sentinel ODOWN
  [ ] Confirm: redis-cli -h old-primary PING → timeout
  [ ] Check: SENTINEL master mymaster → flags: o_down,action_failover

DURING FAILOVER (do not promote manually)
  [ ] Wait for Sentinel to complete (usually < 30s)
  [ ] Verify: get-master-addr-by-name returns new IP
  [ ] Verify: new primary accepts writes SET __failover_test 1

POST-FAILOVER
  [ ] Repair old primary as replica (see below)
  [ ] Confirm all replicas: master_link_status:up
  [ ] Verify application reconnected (client logs)
  [ ] Root-cause: OOM, disk, network, human error

REJOIN OLD PRIMARY (was dead, now back):
  [ ] Old primary may still think it is master — Sentinel fixes via REPLICAOF
  [ ] Or manually: redis-cli REPLICAOF <new-primary-ip> 6379
  [ ] Never start old primary as master without coordination

Manual failover (planned maintenance):

127.0.0.1:26379> SENTINEL failover mymaster

Forces failover even if current primary is healthy. Use before patching the primary host.

Check replica selection criteria:

127.0.0.1:26379> SENTINEL replicas mymaster
# Look at: slave-priority, slave-repl-offset, flags

Sentinel picks: lowest replica-priority (excluding 0), highest replication offset, then lexicographic runid.

Discussion

Split-brain window: If the old primary is alive but network-partitioned, it may continue accepting writes briefly. When it rejoins, Sentinels reconfigure it as a replica; conflicting writes are lost. Minimize with min-replicas-to-write on the primary:

min-replicas-to-write 1
min-replicas-max-lag 10

Primary rejects writes if fewer than 1 replica is connected and lag ≤ 10 seconds.

Recipe 17.4 — Client Discovery and Reconnection

Problem

Hardcoding the primary IP breaks after failover. Clients must discover the current master via Sentinel.

Solution

Discover master manually:

redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster
1) "10.0.1.20"
2) "6379"

Python (redis-py) with Sentinel:

from redis.sentinel import Sentinel

sentinel = Sentinel(
    [("10.0.1.30", 26379), ("10.0.1.31", 26379), ("10.0.1.32", 26379)],
    socket_timeout=0.5,
    password="redis-secret",
)

# Returns connection to current master
master = sentinel.master_for("mymaster", socket_timeout=0.5, password="redis-secret")
master.set("key", "value")

# Returns connection to replica(s)
replica = sentinel.slave_for("mymaster", socket_timeout=0.5, password="redis-secret")
replica.get("key")

Node.js (ioredis):

const Redis = require('ioredis');

const sentinel = new Redis({
  sentinels: [
    { host: '10.0.1.30', port: 26379 },
    { host: '10.0.1.31', port: 26379 },
    { host: '10.0.1.32', port: 26379 },
  ],
  name: 'mymaster',
  password: 'redis-secret',
  sentinelPassword: 'redis-secret',  // if Sentinels require auth
  role: 'master',
});

await sentinel.set('key', 'value');

Go (go-redis):

import "github.com/redis/go-redis/v9"

rdb := redis.NewFailoverClient(&redis.FailoverOptions{
    MasterName:    "mymaster",
    SentinelAddrs: []string{"10.0.1.30:26379", "10.0.1.31:26379", "10.0.1.32:26379"},
    Password:      "redis-secret",
})

Subscribe to failover events (debugging):

redis-cli -p 26379 SUBSCRIBE +switch-master
# Message when new master elected:
# "mymaster" "10.0.1.10" "6379" "10.0.1.20" "6379"

Application checklist after failover:

[ ] Client library supports Sentinel (not just single-host URL)
[ ] Connection pool drains stale connections to old master
[ ] Timeouts configured (failover causes brief unavailability)
[ ] Retry logic for READONLY errors during transition
[ ] Health checks use Sentinel, not cached master IP

Discussion

Typical failover downtime: 10–30 seconds (detection + promotion + client reconnect). Tune down-after-milliseconds vs false positives—5s is common; 1s is aggressive on noisy networks.

Some clients cache the master address. Ensure your library listens for +switch-master or re-queries Sentinels on connection errors.

For TLS environments (Chapter 19), Sentinel connections may also require TLS depending on deployment—check client docs.

Recipe 17.5 — Split-Brain Mitigation

Problem

A network partition could leave two nodes believing they are primary, accepting conflicting writes.

Solution

Layer 1 — Sentinel quorum (odd count):

# 3 Sentinels, quorum 2 — partition must include 2 Sentinels to fail over
sentinel monitor mymaster 10.0.1.10 6379 2

Layer 2 — min-replicas on primary:

# Primary rejects writes if insufficient healthy replicas
min-replicas-to-write 1
min-replicas-max-lag 10

127.0.0.1:6379> SET key value
(error) NOREPLICAS Not enough good replicas to write.

During partition, isolated primary loses replica connections → writes blocked → reduces split-brain write volume.

Layer 3 — replica-priority 0 for non-candidates:

# DR site replica — never auto-promoted
replica-priority 0

Layer 4 — fencing (infrastructure):

For critical systems, integrate STONITH: if Sentinel promotes a new primary, fence the old primary at the hypervisor/load-balancer layer (pull traffic, isolate network) before it can accept writes.

Layer 5 — client-side validation:

def safe_write(master, key, value, sentinel, master_name="mymaster"):
    addr = sentinel.discover_master(master_name)
    if master.connection_pool.connection_kwargs["host"] != addr[0]:
        raise RuntimeError("Connected master does not match Sentinel view")
    master.set(key, value)

Detection — two primaries symptom:

# On suspected split-brain
for host in 10.0.1.10 10.0.1.20; do
  echo -n "$host: "
  redis-cli -h $host -a redis-secret INFO replication | grep role
done
# If both show role:master — split brain active; escalate

Recovery runbook:

1. Identify canonical master via Sentinel majority: get-master-addr-by-name
2. Fence old master (stop app traffic, iptables, or shutdown redis)
3. Reconfigure old master: REPLICAOF <canonical> 6379
4. Verify replication offsets converge
5. Re-enable traffic

Discussion

Redis Sentinel prioritizes availability within a single site. min-replicas-to-write trades availability for consistency during partitions—writes may fail while the cluster is degraded.

True strong consistency across sites requires application design (conflict-free data structures, CRDTs) or accepting that Redis is not the system of record for multi-master writes.

Recipe 17.6 — Sentinel Monitoring and Alerting

Problem

You need dashboards and alerts for Sentinel health, not just Redis nodes.

Solution

Metrics to collect:

Source	Metric	Alert condition
Sentinel	`num-other-sentinels`	< 2 (lost Sentinel peers)
Sentinel	`flags` on master	`o_down`, `failover_in_progress`
Primary	`role`	Unexpected `master` on backup host
Replica	`master_link_status`	`down` > 60s
All	`sentinel_masters` count	Config drift

Health check script:

#!/bin/bash
# sentinel-health.sh
SENTINELS=("10.0.1.30:26379" "10.0.1.31:26379" "10.0.1.32:26379")
MASTER_NAME="mymaster"
FAIL=0

for s in "${SENTINELS[@]}"; do
  host="${s%%:*}"; port="${s##*:}"
  if ! redis-cli -h "$host" -p "$port" PING 2>/dev/null | grep -q PONG; then
    echo "CRITICAL: Sentinel $s unreachable"
    FAIL=1
  fi
done

MASTER=$(redis-cli -h "${SENTINELS[0]%%:*}" -p 26379 SENTINEL get-master-addr-by-name "$MASTER_NAME")
IP=$(echo "$MASTER" | head -1)
if [ -z "$IP" ]; then
  echo "CRITICAL: No master returned for $MASTER_NAME"
  exit 2
fi

if ! redis-cli -h "$IP" -p 6379 -a "$REDIS_PASSWORD" --no-auth-warning PING 2>/dev/null | grep -q PONG; then
  echo "CRITICAL: Master $IP not responding"
  exit 2
fi

echo "OK: master=$IP"
exit $FAIL

Log patterns to alert:

+odown master mymaster ...
+failover-start master mymaster ...
+switch-master mymaster ...
+tilt # Sentinel entered TILT mode — investigate config/network

TILT mode: Sentinel stops acting when it detects something inconsistent (clock skew, repeated errors). Check logs; restart Sentinel after fixing root cause.

Prometheus redis_exporter: Scrape Redis nodes; use separate Sentinel exporter or custom script for Sentinel-specific metrics.

Discussion

Monitor all three Sentinels, not just one. A single Sentinel returning stale master info indicates partition or misconfiguration.

Test failover quarterly in staging (SENTINEL failover mymaster) and measure client recovery time. Document results.

Recipe 17.7 — Sentinel Authentication and Multi-Environment Setup

Problem

You run staging and production on overlapping networks. Sentinels must authenticate to Redis and optionally require client auth to prevent unauthorized failover observation.

Solution

Redis ACL for Sentinel (Redis 6+):

127.0.0.1:6379> ACL SETUSER sentinel-user on >sentinel-secret ~* &* \
  +ping +info +role +client +subscribe +psync +replconf +slaveof +config|rewrite
127.0.0.1:6379> ACL SAVE

sentinel.conf with auth:

sentinel monitor mymaster 10.0.1.10 6379 2
sentinel auth-pass mymaster redis-secret
sentinel auth-user mymaster sentinel-user

# Sentinel-to-Sentinel and client auth (Redis 6.2+)
requirepass sentinel-admin-secret
sentinel sentinel-user sentinel-user
sentinel sentinel-pass sentinel-secret

Separate master names per environment:

# Production
sentinel monitor prod-redis 10.0.1.10 6379 2

# Staging — different port, different quorum
sentinel monitor staging-redis 10.0.2.10 6379 2

Clients connect to the correct master name:

prod = sentinel.master_for("prod-redis", ...)
staging = sentinel.master_for("staging-redis", ...)

Firewall rules:

Allow: App → Redis 6379
Allow: Sentinel → Redis 6379
Allow: Sentinel ↔ Sentinel 26379
Allow: App → Sentinel 26379 (for discovery only)
Deny:  Internet → Redis, Sentinel

Validate Sentinel auth:

redis-cli -p 26379 -a sentinel-admin-secret SENTINEL master prod-redis
# Without password → NOAUTH

Discussion

Never share auth-pass between environments. A staging misconfiguration pointing at production Sentinels has caused real outages—use distinct master names and network segmentation.

When rotating Redis passwords, update sentinel auth-pass on all Sentinels before changing the primary password, or use SENTINEL SET mymaster auth-pass newpass for rolling updates.

Notification scripts for integration:

sentinel notification-script mymaster /etc/redis/sentinel-notify.sh
sentinel client-reconfig-script mymaster /etc/redis/sentinel-reconfig.sh

sentinel-notify.sh receives arguments: <event-type> <event-description>. Use it to page PagerDuty or post to Slack on +odown, +switch-master. The client-reconfig script runs when clients should update—useful for updating HAProxy or Consul registrations during failover.

Schedule quarterly game days: kill the primary in staging, measure detection-to-recovery time, and verify notification scripts fire. Document results and tune down-after-milliseconds and client retry policies based on findings. Target sub-30-second failover in the same availability zone under normal conditions.

Navigation Menu

Chapter 17 — Sentinel & High Availability

Recipe 17.1 — Understand Sentinel Architecture

Problem

Solution

Discussion

See Also

Recipe 17.2 — Configure sentinel.conf

Problem

Solution

Discussion

See Also

Recipe 17.3 — Failover Process and Runbook

Problem

Solution

Discussion

See Also

Recipe 17.4 — Client Discovery and Reconnection

Problem

Solution

Discussion

See Also

Recipe 17.5 — Split-Brain Mitigation

Problem

Solution

Discussion

See Also

Recipe 17.6 — Sentinel Monitoring and Alerting

Problem

Solution

Discussion

See Also

Recipe 17.7 — Sentinel Authentication and Multi-Environment Setup

Problem

Solution

Discussion

See Also

FilesExpand file tree

17-sentinel-and-ha.md

Latest commit

History

17-sentinel-and-ha.md

File metadata and controls

Chapter 17 — Sentinel & High Availability

Recipe 17.1 — Understand Sentinel Architecture

Problem

Solution

Discussion

See Also

Recipe 17.2 — Configure sentinel.conf

Problem

Solution

Discussion

See Also

Recipe 17.3 — Failover Process and Runbook

Problem

Solution

Discussion

See Also

Recipe 17.4 — Client Discovery and Reconnection

Problem

Solution

Discussion

See Also

Recipe 17.5 — Split-Brain Mitigation

Problem

Solution

Discussion

See Also

Recipe 17.6 — Sentinel Monitoring and Alerting

Problem

Solution

Discussion

See Also

Recipe 17.7 — Sentinel Authentication and Multi-Environment Setup

Problem

Solution

Discussion

See Also