Tracking: Valkey multi-instance

[MoralCode]

Opening this issue to track valkey support covering both regular usage and in production/cloud deployments.

Single instance

From what ive heard, valkey should just be a drop-in replacement in the compose stack

Production/multi-instance

Seems like valkey can be run in a cluster mode through its Sentinel feature (https://valkey.io/topics/sentinel/).

To do this, There may need to be special behavior/connection parameters augur has to support (per @JohnStrunk in JohnStrunk/augur-on-ocp#22)

Maybe this is the correct document?
https://valkey.io/topics/sentinel-clients/

[sgoggins]

I am using Valkey on my very large bare metal test instance, FWIW. This required no changes to Augur code. I think a more formal evaluation is likely best, but I am not sure what that looks like given that Valkey is apparently working as advertised: Drop and run.

[MoralCode]

ah ok, yeah in that case we can have this issue scope be for using valkey in sentinel (i.e. a cluster) mode insofar as its needed to support the openshift/kubernetes high availability/fault tolerance case.

[JohnStrunk]

A bit more context:

This issue is about being able to run a valkey (or redis) cluster as opposed to a single instance. The benefit is that the KV store becomes fault tolerant (important for kubernetes/openshift environments).
For valkey/redis clusters, they are managed by a 2nd service, Sentinel, that handles quorum issues and redirects client requests properly.

In order to support clusters, Augur will need to connect to Sentinel instead of a single valkey instance. I briefly started looking at the changes required, and IIRC, the library used by Augur does support Sentinel, but it's not a simple drop-in replacement of the connection string. I believe it ends up being a list instead of a singular value.
So, the main work here is going to be plumbing a list through the code (and config db???) instead of the current single connection string. --- Take this w/ a grain of salt as I didn't get very far on this.

[MoralCode]

I think a more formal evaluation is likely best

Including this from slack for more context as well:

What is needed is a little bit of due diligence about the "design by contract" commitments Valkey is making. Like OpenSearch, I expect that contract to be "drop in", but I also expect a few of the idiosyncracies related to startup of the service, as in most contexts it will need to be started and stopped with its own namespace ...

Im not sure if im familiar enough with redis/valkey to evaluate this, but maybe someone else is