Multi Region Nodes when using clusterAPI

[thameezb]

Is it possible to achieve multi-region/multi-cloud node support when using clusterAPi?

Example cluster would be:

• create a k8s cluster using Kamaji (within the management cluster)
• deploy an AWS machinepool in region A for above cluster
• deploy an AWS machinepool in region B for above cluster
  -...

The outcome would be a functioning CP running as pods within the management cluster. And sets of worker nodes running in different AWS regions.
I know this is possible if I deploy the infra myself and use kubeadm to join the nodes to the CP (therefore technically it should be possible using clusterAPI)

[prometherion]

Migrated to discussion, Issues are used only to track known bugs.

[prometherion]

I think this is possible given the nature of the CAPA implementation with Kamaji, and this can be done in two separate steps

1. Create the Tenant Control Plane and extract its API Server endpoint, such as the IP and the Port: the resource must be created outside of the Cluster API ecosystem because, otherwise, it wouldn't work
2. Create all the required CAPA resources, except for the Control Plane ones: despite being a bit odd, Cluster API allows you to avoid defining the Control Plane. Populate the CAPA Cluster resource's control plane endpoint with the Tenant Control Plane address and port, and let CAPA create all the required resources for you.

Step no.2 can be repeated for the second region, I didn't test this personally but it could work (famous last words): the downside of this approach is on Cluster API which doesn't allow having multiple infrastructure providers for the same cluster.

I'm not arguing if this is a good or a bad idea, it's up to your considerations, especially about networking, egress costs, etc.: happy to receive more info about the use case and promote more about it, tho.

[thameezb]

Thank you for the response! I will attempt to POC this later this week and report back.

The aim is to have a multi-tenanted cluster where each tenant is a single deployment application which can be started in a number of regions amongst a number of clouds (although this cloud requirement is still being discussed).

These tenanted services do not interact with each other so latency between nodes amongst different regions and clouds will be very low. Traffic will have to traverse between CP pods and worker nodes for scheduling etc (acceptable cost as these loads do not change that frequently). As long as the latency is not too high where the master components can handle it, it should be okay.

Infra components is probably where the largest amount of cluster level traffic would occur (for example prometheus scraping metrics). And where the question of public/private net access still needs to be answered.

Step one would be getting this pattern to work in the primary cloud in multiple regions using clusterAPI

Secondary clouds I guess could be bootstrapped using kubeadm or cloudinit scripts if we must (although being able to achieve this using Kestra and clusterAPI would be a great win)