Connecting US Cloud services to EU cluster and load balancer

[maggie44]

Hi all,

I am having a think about potential means of connecting US Cloud servers (ASH etc.) in to a cluster setup in the EU. The obvious blocker is Hetzner's virtual networks do not allow for cross region communication. So instead I am looking to use Nebula (https://github.com/slackhq/nebula) to create an overlay network that will link my EU and US clusters. It is simple enough to Dockerise and allow the services to communicate via exposed IPs.

The question then becomes how best to manage the two together? Ideally a US server could just be added as an additional node to the existing EU cluster, but that is perhaps more of a significant change to the terraform code. Alternatively, two parallel instances could be deployed, one to the EU and one to the US and then the API of each becomes accessible although it requires a lot more servers (min 3 for HA for example) simply to add a US instance, and not sure if that provides as many advantages (other than security of the overlay network over exposing the IP).

Any thoughts welcome. If there seems to be a solid approach I will look at putting it together.

[mysticaltech]

@maggie44 Have a look at https://github.com/submariner-io/submariner, it's by the Rancher team, the same folks behind k3s which we use, so it will be compatible. And if you can make it work and it requires additions to our codebase, please PRs welcome!

[mysticaltech]

But you technically you will probably need two clusters instead of one, but if I understood correctly, it should behave as one in the end (never tried it).

[mysticaltech]

@maggie44 FYI k3s now supports tailscale, this could help you https://docs.k3s.io/installation/network-options#embedded-k3s-multicloud-solution

[maggie44]

That looks ideal! It seems it would allow a much better integration where US servers could run nodes and appear in the cluster like any other.

https://github.com/k3s-io/k3s/blob/master/docs/adrs/integrate-vpns.md

It also looks like it needs a much deeper integration in to the project than I can pick up and do easily, sadly it's probably beyond my limits right now.

[mysticaltech]

There are other users interested that might be interested, see #894. So who knows! But if you want to be able to run such a setup, I suggest to try implementing it, it's probably not that hard.

Anything you do not understand, just ask ChatGPT-4, it's an expert terraform, kubenetes teacher, really 😂

[thecaffeinatedengineer]

@maggie44 There's also Kubevela multi cluster delivery, I am currently looking into it. You can use either oam's cluster gateway or open cluster management.

I am looking into achieving the same thing (using both eu and us clusters to serve the same application), will feedback here if it works.

[caniko]

Why would you do this, and have another layer of abstraction when you can just connect machines across DCs, ultimately having one cluster (#1086)? I this something that isn't supported by Hetzner?

[mysticaltech]

By default not supported from what I know, but you can use the tools shared above!

[caniko]

I am just curious why we would be creating multiple clusters across DCs when we could unify the nodes under one cluster, would you say the added networking complexity outweighs the complexity of multi-cluster? Just looking for reasons for and against either method.

[codeagencybe]

@caniko
The first logic would also occur to just pool the nodes from all zones in a single cluster, but the problem is Hetzner does not support this from their own private networking level.

A second problem is latency issues due to the physical distance. If you would run applications and eg your database lands on a node in EU and your app in US, your visitors will not have a great experience on your website or application.
Also, from what I read in the past, there were many people who experience all kinds of weird problems with scheduling etc... because of latency issues. You can imagine if probes and healthchecks from your controlplane run into latency or worse timeout errors because of latency, this can get very rough.

For this reason, the most used approach is running a cluster in each zone. There are products which are already mentioned above that abstract away the limitations from Hetzner and simulate a multi-cluster as a single cluster but they are still standalone clusters in the end.

[caniko]

This is a great answer, thanks @codeagencybe

[mysticaltech]

Can, with tunnels it's possible. However the geo locations are so far away that it may not be ideal. However PRs welcome to support that if that's something that you can implement. We would have to deal with multiple Hetzner private networks and distribute them per geo location, and run the tunnel on top and handle the routing between the two ourselves.

…

On Tue, Nov 28, 2023 at 9:15 AM Can H. Tartanoglu ***@***.***> wrote: I am just curious why we would be creating multiple clusters across DCs when we could unify the nodes under one cluster, would you say the added networking complexity outweighs the complexity of multi-cluster? Just looking for reasons for and against either method. — Reply to this email directly, view it on GitHub <#883 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD6TG64SRL6LT67GPOJVZLYGWMSZAVCNFSM6AAAAAA2ERM3U6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMOBZGQZDG> . You are receiving this because you commented.Message ID: <kube-hetzner/terraform-hcloud-kube-hetzner/repo-discussions/883/comments/7689423 @github.com>

[caniko]

I would use pod topology to make this viable