Spike: investigate what it would take to support multi-tenant

[shannonwells]

Request

Look into how to allow multiple providers to be supported for a single gateway app.

Context

For the MVI, we will be completely centralized and running one instance (each) of some gateway services, and will need to support more than one provider per instance.

Outline or additional Details

Preferences:

1. Provider keys survive restart (must have)
2. Keys can't be exfiltrated (must have)
3. Easier maintenance is better
4. Easier implementation is better
5. Adding new keys without restarting would be nice
6. Nice to have: Provider can self-provision - with limitations that should be outlined, e.g. nobody should be able to add signing keys if they aren't already a Provider, and we should always verify that they are, using the submitted public key.

Some off-the-cuff possibilities, which may or may not be reasonable:

• Maybe there is already a library for this, a plugin for NestJS and we don't have to write one

• Provider registers webhooks for signing payloads or transactions

  • Advantages: maybe a little more security since Gateway doesn't get access to keys, registered webhooks can be safely stored and reloaded after restarts, Provider can self-provision
  • Disadvantages: anything that needs signing takes longer, MITM attacks, increased traffic, Provider has to keep endpoint live all the time

• Support multiple Providers in environment file

  • Advantages: maybe the easiest solution in that it's already in use,
  • Disadvantages: every time a new Provider is added to environment file, you have to restart the service, Provider cannot self-provision

• Provider submits their secret keys through an API call, and they are somehow stored securely somewhere in the instance. Maybe stored encrypted in Redis using unique server encryption key that lives in the environment file.

  • Advantages: survives restarts, Provider can self-provision, can add new keys without restarting
  • Disadvantages: ....add yours

• Maybe you thought of something even better

[PM/Eng]: Acceptance criteria

• Come up with at least 3 reasonable possibilities
• For each, summarize what it would take to implement
• Write up advantages / disadvantages of each
• Present with recommendation and reasoning

[JoeCap08055]

Some thoughts:

• Solution should still allow for a single-Provider deployment via env configuration, so as not to break existing deployments
• Not sure if Redis is a good persistent storage mechanism... we use it in "persistent" mode to survive app restarts so we don't lose in-flight tasks in the queue, but maybe it's not good to rely on for persistent secret storage?
• We could provision the app with a token to access AWS secrets directly, and have the app fetch a keypair on-demand from AWS (could be plugin-based to support different cloud/secret providers). We did this in Community Rewards, check it out. (Might be more difficult to self-provision, though)

I will say, we have long needed a better/more secure secret-provisioning solution than environment injection, so I'm leaning toward full integration with a secrets engine.

[sai-so]

We have the option to use AWS Secrets Manager (which we are already integrated with) --> External Secrets Operator --> Eks-secret --> Mounted as files in pod --> App reads from files. This workflow enables us to create and update secrets in the aws secrets manager without using env variables or deployment updates to reflect in cluster. Looking into this and will be POCing this in staging.

[JoeCap08055]

One idea for self-provisioning:

Add an endpoint to Gateway to provision a new public key for the Provider:

• Provider requests a new keypair from Gateway via an endpoint
• Gateway creates a keypair and stores the secret key in its secrets manager (AWS, etc), keyed/named using the Provider ID + public key. Return the public key as the reponse.
• Provider signs an AddKey payload to add the public key as a control key
• Provider submits the payload to Gateway to add the public key

Advantages:

• Provider can self-provision
• Private key is never exposed externally
• Provider can revoke the key using normal means (free extrinsic, Provider Dashboard)

Disadvantages:

• Complex flow? Requires Provider to sign payloads, etc. Perhaps add to Provider Dashboard to streamline?
• ?

Session Management

Any multi-tenant approach for Gateway will likely involve the need for Gateway to implement some kind of access control. This may involve one or more of:

• Authentication + session token/session management
• Sessionless authentication header on every request
• ?

[shannonwells]

Meeting notes:

• not critical path for MVI, if we need it we can stand up >1 instance of Gateway
• perhaps update Gateway release artifacts to include a monolithic deployment (but not for MVI)