RedisCluster: Transient node failure causes thundering herd on initialize(), leading to pool exhaustion under high concurrency

[ngabhanenetskope]

Environment

• redis-py version: 6.4.0 (also examined 7.4.0 — partially improved but not fully fixed)
• Python: 3.10
• Concurrency: 200+ threads sharing one RedisCluster client
• Redis: 3-node cluster with replicas

Problem

When a single Redis cluster node becomes temporarily unreachable, the RedisCluster client's error recovery mechanism itself causes a cascading failure that makes the entire application unresponsive — even though the cluster majority is still healthy.

This is distinct from #3929 (permanent unrecoverable state after topology change). Here, the client's recovery behavior crashes the application within seconds of a transient failure, before the cluster has a chance to self-heal.

Failure Sequence (from code analysis)

Given: 200+ concurrent threads sharing one RedisCluster client, max_connections_per_node=50, socket_timeout=5

1. A single node becomes unreachable

2. Threads with active connections to that node block for socket_timeout seconds (5s)

3. Each thread that gets TimeoutError or ConnectionError hits this code path:

   # cluster.py:1245-1258
   except (ConnectionError, TimeoutError) as e:
       if connection is not None:
           connection.disconnect()
       self.nodes_manager.startup_nodes.pop(target_node.name, None)
       target_node.redis_connection = None
       self.nodes_manager.initialize()      # <-- EVERY thread calls this
       raise e

4. Thundering herd on initialize(): Every thread that fails independently calls initialize(). In 6.4.0, there's no deduplication — each call acquires self._lock (threading.RLock) sequentially. In 7.4.0, epoch-based dedup reduces but doesn't eliminate this.

5. initialize() tries dead nodes: Inside initialize(), it iterates startup_nodes and calls CLUSTER SLOTS — potentially trying the dead node again (another socket_timeout wait while holding the lock):

   # cluster.py:1714-1726
   for startup_node in tuple(self.startup_nodes.values()):
       try:
           ...
           cluster_slots = str_if_bytes(r.execute_command("CLUSTER SLOTS"))

6. Pool exhaustion: While threads are queued on the lock or waiting on socket_timeout, they hold connections from the pool. With 200+ threads and only 50 connections per node, the pool saturates. New requests get MaxConnectionsError immediately (thanks to Fix ConnectionPool to raise MaxConnectionsError instead of Connection… #3698).

7. Application starvation: All thread pool slots are consumed by threads blocked on Redis. The application cannot serve any requests — including health checks — leading to container restarts.

Timeline: Single node failure → total application outage in ~5-10 seconds.

Root Causes

The core issue is that RedisCluster treats every failed request as a signal to do a full cluster topology refresh, with no:

1. Node health awareness — the client doesn't track which nodes are failing. It retries the dead node in every initialize() call, wasting socket_timeout seconds each time while holding the lock.
2. Deduplication — in 6.4.0, all threads call initialize() independently. 7.4.0 adds epoch-based dedup which helps but threads still queue on the lock.
3. Non-blocking refresh — initialize() is synchronous and holds a lock. All other threads queue behind it.
4. Fast-fail for known-dead nodes — no mechanism to skip nodes that have failed recently. Every attempt pays the full socket_timeout cost.

Impact

In production, this causes full application outages every time a single Redis node has even a brief connectivity issue. All pods in the deployment fail simultaneously (since they share the same Redis cluster), causing a complete service disruption.

Relation to existing issues/PRs

#3698 — I authored this fix to distinguish MaxConnectionsError from ConnectionError, preventing reinitialization loops when the pool is full. However, it doesn't prevent the pool from reaching saturation in the first place due to the thundering herd on initialize().
#3929 — Different trigger (permanent IP changes vs transient failure) but related: both expose the lack of per-node health tracking in the cluster client.

[petyaslavova]

Hi @ngabhanenetskope, I think one of the PRs that is going to be merged will improve the described situation - PR #4060 - it does randomize the nodes that will receive the CLUSTER SLOTS command AND the failed node will be the last one that will be tried.

[ngabhanenetskope]

Hi @petyaslavova, thanks for the quick response!

I looked at PR #4060 — randomizing nodes and trying the failed node last is a good improvement. It addresses the scenario where initialize() wastes socket_timeout seconds trying the dead node while holding the lock.

However, I believe the core problem still remains even with #4060 applied. Here's why:

The pool exhaustion happens before initialize() is ever called. The sequence is:

1. Node becomes unreachable
2. Threads currently executing commands against that node block at send_command() / parse_response() (cluster.py:1226-1227) for the full socket_timeout duration
3. With 200 threads and max_connections_per_node=50, all 50 pool slots for that node are consumed by blocked threads within milliseconds
4. The 51st thread gets MaxConnectionsError immediately (thanks to Fix ConnectionPool to raise MaxConnectionsError instead of Connection… #3698)
5. Only after a thread's socket_timeout expires does it reach the except (ConnectionError, TimeoutError) block and call initialize()

So by the time initialize() runs (and benefits from #4060's optimization), the damage is already done — the thread pool is saturated from step 2-3, and the application can't serve requests (including health checks, causing container restarts).

The pieces I think are still needed:

1. Per-node health tracking — after N consecutive failures on a node, mark it unavailable and fail fast on subsequent requests to that node's slots (without waiting for socket_timeout). This prevents the initial pool saturation.
2. Non-blocking initialize() — if another thread is already refreshing topology, return immediately instead of queueing on the lock (use stale topology until refresh completes).

Both of these are thread-model agnostic and would benefit all concurrency models. Happy to contribute a PR if you think this direction makes sense.

[ngabhanenetskope]

Hi @petyaslavova — following up on the discussion above. I've put together a PR (#4083) that implements the two fixes I mentioned:

Per-node health tracking — fail fast on known-dead nodes instead of blocking for socket_timeout
Non-blocking initialize() — try-lock pattern so only one thread refreshes topology
These are complementary to #4060 and address the pool saturation that happens before initialize() is ever called. Happy to iterate on the approach if needed.