Elevated MOVED errors while doing SETEX operations

[sriramtallapragada]

Context:
We observed elevated Redis GET/SET failures that correlated strongly with increased DNS latency. Scaling up the CoreDNS pods reduced DNS latency. Subsequently, restarting the application service pods mitigated the Redis errors. While the temporal correlation is clear, the exact mechanism linking DNS latency to client side MOVED errors is not yet fully understood.

One plausible explanation is that increased DNS latency caused Redis redirect handling to fail at the client. When a MOVED response was returned, DNS resolution or connection attempts to the redirected node may have exceeded client timeouts, causing the redirect to fail and the original MOVED error to surface. Repeated failures may have left clients with stale topology until service pods were restarted. Is the understanding here accurate? As a follow-up, whether any client-side settings such as retries on MOVED, redirect-specific timeouts, and topology refresh on failures can be tuned to enable self-healing from transient DNS or topology issues?

Error message:
failed to execute SETEX: MOVED 7555 cache-usw2-001-cell-003-0019-001.cache-usw2-001-cell-003.f6zuq8.usw2.cache.amazonaws.com:6379

We were seeing this MOVED errors across multiple slots and valkey nodes, above is one such example.

Environment
Rueidis v1.0.66
Go 1.25.3
AWS Elastic cache Valkey Cluster v8.1
Auto-Pipelining enabled
Client Side Tracking enabled.

Cluster Configuration
Node instance type: cache.r7g.4xlarge
Total nodes per cluster: 40 primary + 40 replicas

[rueian]

Hi @sriramtallapragada,

Which client method are you using? client.Do()? Unless a MaxMovedRedirections is configured, I think client.Do() should keep retrying until it gets other responses or times out, and should not raise a MOVED. This is indeed weird.

[sriramtallapragada]

Yes, this is the code, and yeah my expectation was similar that MOVED is transparently handled by the client.

	region := rtrace.StartRegion(ctx, "cache.client.Do.SETEX")
		result := v.client.Do(ctx, v.client.B().Setex().Key(key).
			Seconds(int64(finalTTL.Seconds())).
			Value(string(marshaledPayload)).
			Build())
		region.End()
       
		if err := result.Error(); err != nil {
			return fmt.Errorf("failed to execute SETEX: %w", err)
		}

This is how the client is configured:

clientOptions := rueidis.ClientOption{
		InitAddress:       []string{parsedURI.Host},
		Password:          resolvedAuthToken,
		CacheSizeEachConn: config.CacheSizeEachConn,
		// Connection pool
		PipelineMultiplex:   config.PipelineMultiplex,   // 2^N TCP connections
		ConnWriteTimeout:    config.ConnWriteTimeout,    // Connection write timeout
		ReadBufferEachConn:  config.ReadBufferEachConn,  // Read buffer per connection
		WriteBufferEachConn: config.WriteBufferEachConn, // Write buffer per connection
		RingScaleEachConn:   config.RingScaleEachConn,   // 2^N ring buffer size
		MaxFlushDelay:       config.MaxFlushDelay,       // Request batching delay
		// Cluster topology refresh
		ClusterOption: rueidis.ClusterOption{
			ShardsRefreshInterval: shardsRefreshInterval,
		},
	}

Here ShardsRefreshInterval was set to "0s", to disables periodic refresh

One more callout is that we do this set operation in async way and is wrapped around timeout, so could that have caused the issue here, when MOVED redirection was happening?

ctx, cancel := context.WithTimeoutCause(ctx, 1*time.Second, ErrAsyncSetTimeout)
defer cancel()

[rueian]

I don’t think the context timeout could be the cause otherwise you should see a context error instead of the MOVED.

Could you help check if your v.client is actually a *clusterClient not a *singleClient or a *dedicatedClusterClient.

[sriramtallapragada]

I am using this function to create the client: func NewClient(option ClientOption) (client Client, err error) { and InitAddress we are using the AWS ElasticCache cluster URL. Is there a other way to determine the type, from my understanding it should be clusterClient

[rueian]

It should be a clusterClient. One way to verify that is by checking if the output of v.client.Mode() is cluster.

The only scenario I found that MOVED can pop up is when the client is closing or closed:

rueidis/cluster.go

Line 1411 in fa8f4fc

if err != nil && err != Nil && err != ErrDoCacheAborted && atomic.LoadUint32(&c.stop) == 0 {

See the atomic.LoadUint32(&c.stop) == 0 check. When the stop flag is set by calling the Close() method, it will stop retrying and throw the MOVED error.

[sriramtallapragada]

I see but based on the documentation of close() in the rueidis, it says: Close will make further calls to the client be rejected with ErrClosing, and Close will wait until all pending calls finished. So based on my understanding it should wait for inflight requests even when the c.stop != 0 right?

[rueian]

Yes, the docs said that, but the actual behavior seems drifted and can be considered a bug. We may fix it by removing the check or returning ErrClosing directly.

So, did you observe the MOVED while calling Close()?

[sriramtallapragada]

It doesn't seem to be the case for all errors, to give some context we have a 1 global client per K8's pod and when I was checking the pod lifetime it was alive for almost 4 hours, and we close the client only when the pod shutdown's. Based on the logs we were seeing moved errors in between those 4 hours not just when the pod was shutting down.

[rueian]

Ok, thanks for the confirmation. But right now, I have no more clue how this can happen during normal operation. If you could somehow capture the stack trace, that would be super helpful.

[sriramtallapragada]

Unfortunately don't have stack trace handy, and not able reproduce it locally so not sure on how aid you on further investigating this but we had lot of these error and had to restart the pods to mitigate this once DNS latencies were stabilized.

[sriramtallapragada]

@rueian on looking more into the logs timelines we have observed this pattern for leading to moved errors. Don't have stack trace yet, but sharing to see if this helps further into the looking into this.

1. First error that we see is i/o timeout for the lookup: failed to execute SETEX: dial tcp: lookup cache-0031-001.cache.f6zuq8.usw2.cache.amazonaws.com: i/o timeout  (lookup timeout for this shard 0031-001
2. Followed by moved error for that same shard: failed to execute SETEX: MOVED 14995 cache-0031-001.cache.f6zuq8.usw2.cache.amazonaws.com:6379

[billshen99]

@rueian is this a validate path to trigger this MOVE error constantly showed up. and i have PR fix to address this connection reuse issue. #944

Network Issue (DNS, TCP timeout, node restart)
    ↓
Connection Enters Error State
    ↓
Topology Refresh Runs (periodic or error-triggered)
    ↓
Blindly reuses broken connection (no Error() check)
    ↓
Broken connection assigned to slots
    ↓
Commands route to broken connection → "wrong shard/slot" errors
    ↓
Subsequent refreshes blindly reuse same broken connection
    ↓
ERRORS PERSIST FOR HOURS
    ↓
Manual Intervention: Kill pod → new process → fresh connections → fixed

[rueian]

Hi @billshen99, I don't think that is a valid path, and your fix doesn't look right either.

1. No Error() check at the cluster client is okay because the conn (mux) handles reconnections internally.
2. Commands route to broken connection → "wrong shard/slot" errors If the connection is broken, the error won't be "wrong shard/slot".
3. We can't do things like "Preserve old slot mappings for any slots not covered by the new topology".