How to debug redis-ha replication failures on argocd upgrade?

[johnmreynolds]

I'm currently trying to upgrade (using the helm chart) our current (old) ArgoCD install v2.9.4, and running in to issues with redis-ha.

Upgrading to v2.10.9 I think is ok, but around v2.12.6 the new redis-ha pods fail with:

1:S 07 Apr 2025 14:33:27.138 * Connecting to MASTER 172.20.209.119:6379
1:S 07 Apr 2025 14:33:27.138 * MASTER <-> REPLICA sync started
1:S 07 Apr 2025 14:33:27.139 * Non blocking connect for SYNC fired the event.
1:S 07 Apr 2025 14:33:27.139 * Master replied to PING, replication can continue...
1:S 07 Apr 2025 14:33:27.140 # Unable to AUTH to MASTER: -ERR AUTH called without any password configured for the default user. Are you sure your configuration is correct?

Reverting the upgrade seems to fix this. It doesn't seem to directly be the redis version, as it looks like instances have both worked and failed with redis 7.2.4.

I can't see anything about this in the ArgoCD upgrade notes.

Does anyone have any idea what is going wrong here or how to debug this?

Or alternatively, am I right in thinking that ArgoCD doesn't actually need the contents of the redis cache? If so, how do I do a clean redeployment of the redis-ha pods, as currently a rolling upgrade will again fail the replication.

[rumstead]

Does the argocd-redis secret exist and have a secret for redis to use? It seems like your pods cannot auth.

https://argo-cd.readthedocs.io/en/stable/faq/#how-to-rotate-redis-secret

[johnmreynolds]

Yes, when running on v2.9.4 I have an argocd-redis secret, with a key "auth" within that.

When upgrading to v2.12.6 I get the above error.

[rumstead]

Do you see the env variable REDIS_PASSWORD AUTH set inside the container?

[johnmreynolds]

Do you see the env variable REDIS_PASSWORD AUTH set inside the container?

So, I'm currently running on v2.9.4 and I think the answer is no to both of those. At least they don't appear in the list from printenv.

[rumstead]

it sounds like you arent using the correct manifests with the image version then. The auth env variable is set when using the HA deployment in 2.12.6.

[johnmreynolds]

Yeah, so that's what I think is the problem, it looks like the auth isn't needed in 2.9 but in some later version (2.12?) it becomes needed. Unfortunately this looks like it breaks the upgrade process when using HA as the new redis pods are trying to authenticate to the old master pod (which doesn't have auth configured) and treating this as a hard failure.

It's not clear if this is an ArgoCD issue or an issue with the charts for it (I'm using the charts from https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd ). Probably not an especially critical one given it's an issue with an edge case in a rather old update. The next time I try this update I guess I'll try nuking the redis pods assuming I can work out how to do that cleanly.

[rumstead]

There are a few changes when adding auth, like an init container. Are you using resource quotas, and maybe the init container cannot start?