The project involves creating a Kubernetes cluster stretched across multiple sites. It will consist of 3 nodes:
- 2 nodes on a private network
- 1 node on a publicly accessible network
A VPN server will be hosted on the publicly accessible node to create a common private network for all 3 nodes on which the cluster will be based.
The topology is as follows:
eagle: the VPS. Hosts the VPN, publicly accessible. Must accessk8s-node1andk8s-node2to expose services hosted on these nodes.k8s-node1: one of the machines on the private network. Must access the VPS and services hosted on it via the VPN rather than via the public IP.k8s-node2: same as above
The machinesk8s-node1andk8s-node2must remain accessible via the private network on which they are hosted but must also be able to communicate with each other via the VPN.
We would theoretically have 3 networks:
- Public (Eagle only)
- VPN (Eagle, node1 and node2)
- Private (node1 and node2)
The VPN chosen for this project is Wireguard for its simplicity of configuration and installation. Ideal for adding other nodes to the cluster later.
apt update
apt install wireguard
cd /etc/wireguard
wg genkey > private && cat private | wg pubkey > pubkey[Interface]
Address = 10.10.0.1/24
ListenPort = 51820
PrivateKey = <eagle's private key>
PostUp = sysctl -w net.ipv4.ip_forward=1
[Peer]
PublicKey = <k8s-node1's public key>
AllowedIPs = 10.10.0.2/32
[Peer]
PublicKey = <k8s-node2's public key>
AllowedIPs = 10.10.0.3/32 [Interface]
Address = 10.10.0.2/24
PrivateKey = <k8s-nodeX's private key>
[Peer]
PublicKey = <eagle's public key>
Endpoint = <eagle's public ip>:51820
AllowedIPs = 10.10.0.0/24
PersistentKeepalive = 25 Since k8s-node1 and k8s-node2 need to be mutually accessible via the VPN, IP routing must be enabled:
echo 'net.ipv4.ip_forward = 1' >> /etc/sysctl.confThe 3 nodes should now be able to communicate with each other via the 10.10.0.0 network
The creation of the k8s cluster will be done via kubeadm. The plugins used for networking and ingress control will be flannel and ingress-nginx.
Kubeadm uses a kubeadm-config.yaml configuration file to set up the cluster. The configuration is initiated from eagle:
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 10.10.0.1 # We specify to use the interface on the VPN
nodeRegistration:
criSocket: unix:///var/run/cri-dockerd.sock # We'll use docker's CRI
taints: null
---
apiVersion: kubeadm.k8s.io/v1beta4
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.33.1
networking:
podSubnet: "10.244.0.0/16" # Required for flannel
proxy: { }
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemdOnce the configuration is done, we initialize the cluster:
kubeadm init --config ./kubeadm-config.yamlOnce the configuration is complete, a command allowing other nodes to join the cluster is displayed. In our case, we also specify the CRI to use:
kubeadm join 10.10.0.1:6443 --token <token> --discovery-token-ca-cert-hash <hash> --cri-socket unix:///var/run/cri-dockerd.sockThe nodes may not have the correct IP address:
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
eagle NotReady control-plane 5m25s v1.33.1 <eagle's public ip>
k8s-node1 NotReady <none> 4m36s v1.33.1 172.16.0.163
k8s-node2 NotReady <none> 4m27s v1.33.1 172.16.0.164
Each node needs to be configured to use the correct address:
echo "KUBELET_EXTRA_ARGS=--node-ip=10.10.0.X" > /etc/default/kubelet
systemctl daemon-reexec
systemctl restart kubelet
# once configuration is done on all nodes
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP
eagle NotReady control-plane 5m45s v1.33.1 10.10.0.1
k8s-node1 NotReady <none> 4m56s v1.33.1 10.10.0.2
k8s-node2 NotReady <none> 4m47s v1.33.1 10.10.0.3Networking will be done via Flannel. The plugin is added with the following command:
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
Once all pods are running, we can configure the ingress controller. We'll use ingress-nginx for this. We download the manifest before applying it because we're going to modify it:
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.2/deploy/static/provider/baremetal/deploy.yamlIn the Deployment section, we'll add a rule in nodeSelector to deploy the ingress controller on the publicly exposed node. Since the node is also part of the control-plane, a tolerations rule will also be added to allow deployment on the control-plane in this case:
...
---
apiVersion: apps/v1
kind: Deployment
...
spec:
...
template:
...
spec:
...
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
nodeSelector:
kubernetes.io/hostname: eagle
kubernetes.io/os: linux
... We now apply the ingress-nginx manifest:
kubectl -f ./deploy.yamlNow we need to modify the VPN configuration to allow routing IPs from the networks created by Flannel through the VPN.
On Eagle:
[Peer]
PublicKey = <k8s-node1's public key>
AllowedIPs = 10.10.0.2/32, 10.244.1.0/24
[Peer]
PublicKey = <k8s-node2's public key>
AllowedIPs = 10.10.0.3/32, 10.244.2.0/24 On the private nodes:
[Peer]
PublicKey = <eagle's public key>
Endpoint = <eagle's public ip>:51820
# We allow the VPN IPs and all other Flannel IPs
AllowedIPs = 10.10.0.0/24, 10.244.0.0/24, 10.244.2.0/24
PersistentKeepalive = 25 After restarting the VPN, it should now be possible to reach each node, either via the VPN IP or via the IP created by Flannel. There may already be routing rules for these IPs as shown here:
ip route
# returns
default via XXX.XXX.XXX.XXX dev eth0 onlink
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.1.0/24 via 10.244.1.0 dev flannel.1 onlink
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
...
# we remove them and restart the VPN
ip route del 10.244.1.0/24
ip route del 10.244.2.0/24
systemctl restart wg-quick@wg0The cluster is now fully configured and ready to host services!