Errors and Panic in Rueidis during Redis Cluster Shard Failover

[ashmeet-zepto]

Description

We observed a set of issues when using rueidis v1.0.69 with a Redis Cluster (v7.4).

During a failover in one shard of the cluster, the application experienced an increase in Redis request latency, followed by multiple client-side errors. No application code changes were deployed during this time.

The following issues were observed:

• Intermittent transaction errors:

EXEC was aborted by redis or connection closed

• Redis protocol parse errors:

rueidis: parse error: redis message type simple string is not a array
rueidis: parse error: redis message type array is not a string

• Connection-level errors:

EOF
dial tcp <ip>:<port>: operation was canceled

Once the parse errors started, the client did not recover and required restarting application pods.

Panic observed during cluster refresh

In the same time window, we observed a panic originating from the cluster topology refresh path.

panic: runtime error: index out of range [2] with length 0

goroutine 258842186 [running]:
github.com/redis/rueidis.parseSlots(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/cluster.go:378
github.com/redis/rueidis.clusterslots.parse(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/cluster.go:378
github.com/redis/rueidis.(*clusterClient)._refresh(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/cluster.go:167
github.com/redis/rueidis.(*call).do(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/cluster.go:217
github.com/redis/rueidis.(*call).LazyDo.func1(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/singleflight.go:58
github.com/redis/rueidis.(*call).LazyDo(...)
    /go/pkg/mod/github.com/redis/rueidis@v1.0.69/singleflight.go:53

Client Code

• Connection dial timeout: default
• Connection write timeout: default

Client Initialization

client, err := rueidisotel.NewClient(
    rueidis.ClientOption{
        InitAddress:       conf.NodeAddresses,
        Username:          conf.UserName,
        Password:          conf.Password,
        DisableCache:      conf.DisableCache,
        CacheSizeEachConn: cacheSizeEachConnection,
    },
)
if err != nil {
    return err
}

if err := client.Do(
    context.Background(),
    client.B().Ping().Build(),
).Error(); err != nil {
    return err
}

Client Usage

func (c *Cache) MGetWithClientSideCache(
    ctx context.Context,
    keys []string,
) (map[string]string, error) {

    results, err := rueidis.MGetCache(
        c.client,
        ctx,
        30*time.Second,
        keys,
    )
    if err != nil {
        return nil, err
    }

    finalResult := make(map[string]string, len(results))
    for key, result := range results {
        value, err := result.ToString()
        if err != nil && !rueidis.IsRedisNil(err) {
            return nil, err
        }
        finalResult[key] = value
    }

    return finalResult, nil
}

Environment

• Rueidis v1.0.69
• Go 1.23.0
• Redis Cluster v7.4
• Auto-Pipelining enabled

Questions

1. When does rueidis refresh the cluster topology?
   If no refresh interval is configured, what situations cause a topology refresh to happen automatically?

2. Is it normal to see EXEC abort errors during a shard failover?
   Should applications expect these errors during cluster changes?

3. What situations can lead to Redis protocol parse errors in rueidis?
   For example, can they happen due to connection interruptions or cluster changes?

4. After a protocol parse error, should the client recover on its own or be restarted?

[rueian]

Hi @ashmeet-zepto,

To reproduce the issue, I think we need more details:

1. The minor version of your Redis instances.
2. Your cluster setup, such as how many primaries and replicas.
3. How do you perform failover precisely, in steps?

Could you help provide the above information?

To your questions:

When does rueidis refresh the cluster topology?
If no refresh interval is configured, what situations cause a topology refresh to happen automatically?

It refreshes the topology whenever it encounters network errors and redirections, but at most once per second.

Is it normal to see EXEC abort errors during a shard failover?
Should applications expect these errors during cluster changes?

Yes, I think it is normal, but we may consider handling the retries of EXEC for client-side caching transparently.

What situations can lead to Redis protocol parse errors in rueidis?
For example, can they happen due to connection interruptions or cluster changes?

Usually, those are client-side bugs that use the wrong parsing functions, like ToArray() and ToString(), and should not happen due to connection interruptions or cluster changes. Otherwise, there might be server bugs that broke the RESP protocol. There have been similar server bugs before.

After a protocol parse error, should the client recover on its own or be restarted?

If it is simply using the wrong parsing function for a specific message, then the client should recover on its own. To verify whether it is the case, we may need to try to reproduce your situation and log out all TCP bytes.

[ashmeet-zepto]

Hi @rueian

The minor version of your Redis instances.

We are using Redis 7.4 for our Redis Cluster

Your cluster setup, such as how many primaries and replicas.

The cluster has 20 primary shards, each with 1 replica.
The replicas do not serve traffic and are used only for failover.

How do you perform failover precisely, in steps?

Failover is not initiated or controlled by us directly.
The cluster is deployed using a managed Redis service [by RedisLabs], where failover is handled automatically by the platform.

During a maintenance activity to reduce provisioned infrastructure, failover occurred on a subset of shards and replicas were promoted to primaries. Immediately following this activity, we observed increased latency, client-side errors, and the panic described in the issue.

[rueian]

We are using Redis 7.4 for our Redis Cluster

Sorry, I was actually asking about the patch version.

During a maintenance activity to reduce provisioned infrastructure

Did you initiate the activity manually through their UI/API, or was it an scheduled event?

failover occurred on a subset of shards and replicas were promoted to primaries.

What happened to their old primaries?

[ashmeet-zepto]

@rueian sharing a few more details below.

Did you initiate the activity manually through their UI/API, or was it a scheduled event?

The activity was manually initiated by us as part of a planned change, applied via Terraform. It was not an automatic or scheduled event.

What happened to the old primaries?

The previous primaries were demoted to replicas, and the replicas were promoted to primaries as part of the failover process.

Redis INFO output

Below is the output from INFO server

# Server
redis_version:7.4.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:0000000000000000000000000000000000000000
redis_mode:standalone
os:Linux 6.8.0-1044-aws x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:11.4.0
process_id:13811402
run_id:521dec1d0ba36249ee627ac9457cf33465241322
tcp_port:11281
server_time_usec:1766056649000000
uptime_in_seconds:752750
uptime_in_days:8
hz:10
lru_clock:0

[ashmeet-zepto]

Hi @rueian, were you able to reproduce this?

[rueian]

Hi @ashmeet-zepto,

I have tried failover, executing CLUSTER FAILOVER randomly, 3500 times on an OSS Redis Cluster 7.4.3 (4 shards, each has 1 replica), but I still don't see the issue while doing MGetCache concurrently on the cluster.

I wonder if the issue only exists on RedisLabs. But since I don't have a RedisLabs account, I am not able to try RedisLabs.

Below is the output from INFO server

redis_mode:standalone

Does RedisLabs return redis_mode:standalone instead of redis_mode:cluster? It seems like they do cluster differently than the OSS.

[ashmeet-zepto]

Hi @rueian — thanks for the clarification.

Yes, RedisLabs does return redis_mode:standalone instead of redis_mode:cluster.

Could you help explain the impact of this on the client behavior, and what exactly changes in Rueidis when it sees standalone instead of cluster? In particular, I’m trying to understand how this affects topology refresh, routing, and failover handling.

I also spent some time going through the internal flow of how DoMultiCache is executed. One thing I’m confused about is how I’m seeing errors like:

rueidis: parse error: redis message type simple string is not an array

even though I don’t call .ToArray() anywhere in my service code.

From what I can tell, this error seems to originate from the internal handling and would have been propagated via askingMultiCache (see permalink).

Could you help explain under what conditions this can happen?
For example, is this expected during failover, connection reuse, or when the server response type doesn’t match what the client expects due to cluster/standalone mode mismatches?

[rueian]

Hi @ashmeet-zepto,

Could you help explain the impact of this on the client behavior, and what exactly changes in Rueidis when it sees standalone instead of cluster? In particular, I’m trying to understand how this affects topology refresh, routing, and failover handling.

That doesn't affect the client's behavior. Rueidis always probes the server with CLUSTER SHARDS or CLUSTER SLOTS command to see whether the server is a cluster or not. However, redis_mode:standalone is a sign that the cluster is different from the open source version, and this could be the reason why I couldn't reproduce the issue.

From what I can tell, this error seems to originate from the internal handling and would have been propagated via askingMultiCache (see permalink).

As the askingMultiCache name suggests, it is triggered after receiving an -ASK redirection from the server, and yes, it could be where that returns the parse error. But returning the parse error from there is unexpected. I think it could only happen when the server messages were delivered out of order.