Multi-tenancy with Argo Workflows & Argo Events

[smashingraph]

Hello!

I'm seeking advice on designing a multi-tenant architecture using Argo Workflows and Argo Events with a JetStream-based EventBus. My primary goals are strong tenant isolation (preventing access to unintended messages) and minimizing resource consumption.

I can think of several scenarios, but none seems perfect.

Based on following architecture assumption:

• Namespace: argo-workflows to run argo-workflows controller, non-namespaced, watches workflows in any namespace
• Namspaces:
  • tenant-1-workflow-namespace, running workflows for tenant 1
  • tenant-2-workflow-namespace, running workflows for tenant 2
  • tenant-3-workflow-namespace, running workflows for tenant 3
  • ...
• Namespace: argo-events to run argo-events controller
  • non-namespaced, watches Argo Events resources in any namespace (Argo EventSources and Argo Events Sensors)
• EventBus implementation based on Jetstream (don't have an existing kafka instance, and seems more complex than Jetstream)

1st Scenario
Having a single argo-events EventBus running in argo-events namespace
Every tenant create EventSources and Sensors in argo-events namespace

• Pros
  • Simple
  • EventBus resources optimization
• Cons
  • Every messages are sharing same event bus scope
  • Only one account in the bus
  • Single point of failure (SPOF), if EventBus fails, every tenants suffer
  • EventSources and Sensors are visible to any tenant
  • Every tenant need to have access to argo-events namespace

2nd Scenario
Each tenant is running and managing its own EventBus
And they can create EventSources and Sensors in their dedicated tenant namespace.

• Pros
  • Events are visible only within respective tenant namespace, and not seen by any other tenant
  • No multi tenant SPOF, if one eventbus fails for a tenant, other tenants are not impacted
  • No need to give access to tenant on argo-events namespace
• Cons
  • Requires for each tenant an EventBus, therefore more resources (usually at least 3 pods).
  • Argo Events upgrade complexities ?
    • requires every tenant namespaces to upgrade their EventBus at the same time

3rd Scenario
Deploy a shared jetstream bus with helm chart from https://github.com/nats-io/k8s/tree/main/helm/charts (not using EventBus CRD), in a dedicated namespace, and configure accounts per tenants (based on https://docs.nats.io/running-a-nats-service/configuration#authentication-and-authorization)
Configure in each tenant namespace an EventBus with spec.jetstreamExotic configuration and proper credentials for an account.

• Pros

  • Events are isolated and not mixed with other tenants, as each will have a dedicated account on the bus
  • EventBus resources optimization
  • More control on the Bus configuration ?
  • Argo Events upgrade less complex (event bus centralized)

• Cons

  • Single point of failure (SPOF), if EventBus fails, every tenants suffer
  • Managing the EventBus more complex when not using EventBus CRD ?

3rd Scenario Bis
Instead of deploying Jetstream with helm chart from https://github.com/nats-io/k8s/tree/main/helm/charts, configure EventBus with a specific configuration for multi-accounts, would this work ?

Thanks in advance if you have some input to share on how you may have implemented this, and I think It would be incredibly valuable to have some recommendations in the ArgoEvents documentation about this subject.

Best Regards,