Submariner Security Architecture Issue: Alternative to Exposing K8S API in Broker Cluster

[artem-zinnatullin]

Hi Submariner team, we're actively figuring out how to roll out Submariner for secure cross-cluster VPNs in our many-clusters setup across the world.

After days of research it appears that Submariner architecture requires exposing entire K8S API in Broker cluster for Agent clusters to connect to it directly which is to say it lightly far from ideal from real security and formal compliance/audit standpoint.

My understanding is, and please correct me where I'm wrong, that Submariner uses Broker Cluster K8S CRDs essentially as a database which is fine, but it appears that to simplify the work with Submariner K8S CRDs the code expects direct access to K8S API in Broker Cluster and this is not good.

NSA/CISA Kubernetes Hardening Guidance (version 1.2, August 2022) Control Plane Hardening section page 18:

The Kubernetes API server runs on port 6443, which should be protected by a firewall to accept only expected traffic. The Kubernetes API server should not be exposed to the Internet or an untrusted network.

What would you like to be added:

Has it been considered to expose Submariner Broker-specific HTTP/GRPC/etc server in Broker cluster so that:

1. K8S API can stay off the public network
2. Submariner agents would connect to Submariner Broker-specific HTTP/GRPC/etc server instead of K8S API with appropriate tokens
3. Submariner Broker-specific HTTP/GRPC/etc server will still store its state in K8S CRDs

Why is this needed:

1. Exposing K8S API to public internet is dangerous as you risk exposing it to known CVEs, 0 day vulnerabilities and DDoS attacks, etc.
2. Hardening security of K8S API in Broker cluster while still making it accessible for Submariner agent clusters over public internet is very hard and error-prone. It essentially requires maintaining a static/dynamic list of Gateway node IPs from all other clusters that can connect to Submariner Broker which is chicken-and-egg problem if you want to use Submariner as secure communication layer between clusters in the first place. Not to mention that it requires quite complicated error-prone custom sync-jobs that look up public gateway node IPs and somehow securely deliver it to broker cluster firewall rules…
3. Many production K8S setups restrict access to K8S API via separate VPN, but then such VPN is needed in each agent cluster which adds so much complexity when Submariner was supposed to be the VPN solution for multi-cluster setups in the first place.

Perhaps we're missing something about Submariner Architecture?

---

Related issues and links:

• Submariner Architecture https://submariner.io/getting-started/architecture/
• Support client cert auth for agent/broker communication Support client cert auth for agent/broker communication #1115
• NSA/CISA Kubernetes Hardening Guidance (version 1.2, August 2022) Control Plane Hardening: https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF
• Kubernetes API Access Security Hardening https://goteleport.com/blog/kubernetes-api-access-security/
• Best Practices for Kubernetes Security https://www.styra.com/blog/best-practices-for-kubernetes-security/

Thank you for this project and apologies if we misunderstood something!

[tpantelis]

Thank you reporting the issue and for the excellent write up.

Submariner agents would connect to Submariner Broker-specific HTTP/GRPC/etc server instead of K8S API with appropriate tokens

Something like this has not yet been considered but it's an interesting idea. The current developers working on the project wouldn't have the bandwidth for such a significant feature in the near future but we welcome contributions.

Would having a small, dedicated cluster/K8S API just for brokering CRDs between the data clusters alleviate your concerns?