Support for multi-replica AuthN deployments

[eric-higgins-ai]

Describe the feature

Currently (I think) the AuthN server only supports single-replica deployments, which means any updates incur downtime. For example, if we cycle the node that the AuthN server is running on, or if we make an update to the AuthN server itself.

Use Case

We want to be able to deploy changes with no user-facing downtime.

Proposed Solution

No response

Other Information

I'm checking "I may be able to implement this feature request," but I'd need some guidance on the approach since I think it would be a fairly big change.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AIStore build (latest, v3.22, ...)

v4.2

Environment details (OS name and version, etc.)

Oracle Kubernetes Engine 1.34.1, OS: Oracle Linux

[aaronnw]

Yes, this is a major limitation we're aware of and I'm currently working towards supporting it. There are a few core issues that make it more difficult than just scaling up the deployment:

• authN currently only supports BuntDB persisted to disk. That means for multi-replica we need a way for other replicas to share this, which requires a distributed storage class and brings in other challenges with locking etc. The other option is adding support for other DBs. I think this makes more sense long term.
• Signing is done with an RSA key pair. If we support multiple replicas, either the leader does signing or we must distribute this secret. Either way, if a replica goes down a new one must be able to continue signing. Question is if we should use the same key pair to cut down on JWKS size or maybe generate a new one.
• Validation can be done either by the RSA public key OR by OIDC issuer lookup. We would need to decide if supporting a static public key can be allowed with multi-replica.

I updated the docs here but there are a lot of design decisions still pending.

[shreyaskommuri]

I've put together an implementation in #290. Quick summary of the approach:

• Storage: Added a Redis backend (auth.db.type = "Redis"). All replicas share users, roles, clusters, and the revoked-token list in Redis. BuntDB is unchanged and remains the default.
• RSA keys: All replicas mount the same private key from a Kubernetes Secret. The server now fails at startup with a clear error if the key file is missing in multi-replica mode, rather than silently generating a different key per replica.
• Token revocation: Already push-based to AIS clusters, so with shared Redis it works correctly across replicas with no logic changes.
• Key rotation: Returns a clear error when using a shared/external key — update the Secret and redeploy instead.

Not included in this PR: tests for the Redis driver and Helm chart updates — happy to add those if the approach looks good.