-
|
Hi, I'm currently taking a closer look at hetzner-k3s and have already created and used a few test clusters. For productive use, however, I would like to add a bare-metal worker node with more resources than the cloud instances offer to the cluster. |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 23 replies
-
|
Hey, currently, this feature isn’t supported out of the box, but I’m thinking about adding it in the future since it could be really useful. For now, you can try to manually connect your dedicated server to the cluster. To do this, you need to have updated to Here’s how you can do it:
crystal run src/hetzner-k3s.cr -- create --config <path to your cluster's yaml file>The Lastly, run the join command on your dedicated server. If the dedicated server is in the same network as the cluster, it should work. Keep in mind that I haven’t tried this myself yet. Good luck, and let me know how it goes! |
Beta Was this translation helpful? Give feedback.
-
|
I think I see the problem. The Hetzner Cloud Controller Manager is designed for Hetzner Cloud, so when a new node joins the cluster it needs to create the necessary routes for the new node, but that won't work of course with dedicated servers. I think the same problem might happen with the CSI driver. The only workaround I can think of is to avoid the installation of the CCM and CSI drivers. I haven't tried it because I'd have to purchase dedicated servers just to test this, but perhaps you could try it? Check out the 2.1.1 branch and comment out these lines: Kubernetes::Software::Hetzner::Secret.new(configuration, settings).create
Kubernetes::Software::Hetzner::CloudControllerManager.new(configuration, settings).install
Kubernetes::Software::Hetzner::CSIDriver.new(configuration, settings).installI suspect that with this change it should be possible to add dedis to the cluster but it needs to be tested. If you can test this and it works, I can make the installation of those components optional (enabled by default). Keep in mind that without CCM and CSI though, you cannot create services of type |
Beta Was this translation helpful? Give feedback.
-
|
I am confused though because I think in the past some users have gotten the combo working. |
Beta Was this translation helpful? Give feedback.
-
|
I have now had some time to try out the setup. Dedicated ServerI bought a dedicated server and followed https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch to connect it to a cloud network. The following lines where added to This allows me to ping the cloud server from the bare-metal server and vice versa. hetzner-k3sTo allow communication with the dedicated server, I manually created a cloud-network ("test-network") using 10.0.0.0/16 and two subnets:
This is my cluster-config.yaml: ---
hetzner_token: <HCLOUD_TOKEN>
cluster_name: bm-test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.30.3+k3s1
networking:
ssh:
port: 22
use_agent: false # set to true if your key has a passphrase
public_key_path: "~/.ssh/id_rsa.pub"
private_key_path: "~/.ssh/id_rsa"
allowed_networks:
ssh:
- 0.0.0.0/0
api: # this will firewall port 6443 on the nodes
- 0.0.0.0/0
public_network:
ipv4: true
ipv6: true
private_network:
enabled: true
# subnet: 10.0.0.0/16
existing_network_name: "test-network"
cni:
enabled: true
encryption: false
mode: flannel
# cluster_cidr: 10.244.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for pod IPs
# service_cidr: 10.43.0.0/16 # optional: a custom IPv4/IPv6 network CIDR to use for service IPs. Warning, if you change this, you should also change cluster_dns!
# cluster_dns: 10.43.0.10 # optional: IPv4 Cluster IP for coredns service. Needs to be an address from the service_cidr range
datastore:
mode: etcd # etcd (default) or external
# external_datastore_endpoint: postgres://....
schedule_workloads_on_masters: false
# image: rocky-9 # optional: default is ubuntu-24.04
# autoscaling_image: 103908130 # optional, defaults to the `image` setting
# snapshot_os: microos # optional: specified the os type when using a custom snapshot
masters_pool:
instance_type: cx22
instance_count: 1
location: fsn1
worker_node_pools:
- name: small-static
instance_type: cx32
instance_count: 1
location: fsn1
- name: medium-autoscaled
instance_type: cpx31
instance_count: 2
location: fsn1
autoscaling:
enabled: true
min_instances: 0
max_instances: 3
embedded_registry_mirror:
enabled: false # Check if your k3s version is compatible before enabling this option. You can find more information at https://docs.k3s.io/installation/registry-mirror
api_server_hostname: cluster.mydomain.com # optional: DNS for the k8s API LoadBalancer. After the script has run, create a DNS record with the address of the API LoadBalancer.I'm using the current version (v2.2.7) of hetzner-k3s and added the line I had to slightly adjust the install script a bit so that it would run, but in the end k3s is installed. However, I get the following log messages (journalctl -xeu k3s-agent.service): When I run The Hetzner Cloud Controller Manager supports dedicated servers since v1.19.0, so theoretically it should be possible as far I understand? Maybe someone could guide me in the right direction. I would really need a node with more ressources in my cluster (which cloud servers not provide) to run my tasks. |
Beta Was this translation helpful? Give feedback.
-
|
I haven't had the chance to try a cluster with dedicated servers myself yet. If you're up for it, you could reset the OS on your dedicated server and let me have temporary access. That way, I can check things out without having to buy a server. Once I'm done, you could just reset it again. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @vitobotta, I would really appreciate this - thank you! I will send you the server details by email. |
Beta Was this translation helpful? Give feedback.
-
|
@vitobotta Did you think about adding support for such nodes? It would be nice even if it will be half-manual, so we can have something like this: So, to make it work we'll need to manually order dedicated server and do manual setup (install correct image, add vSwitch and correct config for it), after that I believe the tool can setup everything else? |
Beta Was this translation helpful? Give feedback.
-
|
Like I mentioned in my previous post in this thread, yes, adding support for dedicated servers is on my radar, but I don't have an ETA for now. But it's on my list |
Beta Was this translation helpful? Give feedback.
-
|
I was reading this https://community.hetzner.com/tutorials/kubernetes-on-hetzner-using-bare-metal-servers |
Beta Was this translation helpful? Give feedback.
-
|
The ClusteAPI etc approach is a lot more complicated compared to what hetzner-k3s offers. Plus, the article seems to be talking about a cluster made of baremetal servers only, not one that mixes them with cloud instances like we're dealing with here. I'm planning to add support for "external nodes" in hetzner-k3s. This would allow adding dedicated servers or even nodes from another provider if needed for any reason. The only catch is that CCM and CSI can't be installed since they're designed to work with cloud instances only. |
Beta Was this translation helpful? Give feedback.
-
|
@vitobotta Would dedicated server only cluster be out of scope for the project? |
Beta Was this translation helpful? Give feedback.
-
|
@neoromantique For now, yes. Running dedicated servers has quite a few limits when compared to cloud instances-things like no API control, slow setup times, and so on. So even though I plan to support dedicated servers as nodes, the control plane will probably stay on the cloud for now. Using only dedicated servers would defeat most of the features and automation that hetzner-k3s provides. If you want to create a cluster with dedicated servers only, check out this project https://github.com/alexellis/k3sup |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot to @if-mar for providing me access to a dedicated server as well as a Hetzner project with the necessary vSwitch setup, so that I could conduct some investigation. I got a cluster with the dedicated server working nicely, for the most part. Unfortunately it seems that even the latest version of the HCCM doesn't handle dedicated servers properly as it's supposed to. These are the steps I took to get the cluster working more or less correctly: 1 - Network setup with vSwitch link - done by @if-mar , who could add some instructions here as well - name: HCLOUD_NETWORK_ROUTES_ENABLED
value: “false”This disables the creation of routes by the HCCM but that should be fine because it's taken care by the CNI, unless you have some weird setup. Caveat:
@if-mar Please try this and let me know how it goes. I will automate this stuff in hetzner-k3s when I have a little time. I am thinking of adding a new |
Beta Was this translation helpful? Give feedback.
-
|
@vitobotta I can confirm that the node is added to the cluster. Sadly the taint $ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
debian-bookworm-latest-amd64-base Ready <none> 170m v1.30.8+k3s1 10.0.1.2 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-28-amd64 containerd://1.7.23-k3s2
ifos-cluster-master1 Ready control-plane,etcd,master 3h v1.30.3+k3s1 10.0.0.2 168.119.xxx.xxx Ubuntu 24.04.2 LTS 6.8.0-54-generic containerd://1.7.17-k3s1
ifos-cluster-master2 Ready control-plane,etcd,master 3h v1.30.3+k3s1 10.0.0.3 138.199.xxx.xxx Ubuntu 24.04.2 LTS 6.8.0-54-generic containerd://1.7.17-k3s1
ifos-cluster-master3 Ready control-plane,etcd,master 179m v1.30.3+k3s1 10.0.0.4 116.202.xxx.xxx Ubuntu 24.04.2 LTS 6.8.0-54-generic containerd://1.7.17-k3s1The internal IP seems to be detected correctly, but external ip is |
Beta Was this translation helpful? Give feedback.
-
|
Hi @if-mar , from what I've seen the taint is added again when restarting the HCCM. Does it happen otherwise for you? Re: external IP I am not sure. This is mostly a workaround to get some workloads running on the dedis, but without proper support from the HCCM it's hard to get it working perfectly. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @vitobotta, after serveral tries and some investigating, I managed to get it working. Here is, what I did:
Detailed instructions can be found in Hetzner docs (https://docs.hetzner.com/cloud/networks/connect-dedi-vswitch/)
private_network:
enabled: true
existing_network_name: "cluster-network"Create the cluster running - name: HCLOUD_NETWORK_ROUTES_ENABLED
value: "false"
- name: ROBOT_ENABLED
value: "true"
kubectl taint nodes <node> node.cloudprovider.kubernetes.io/uninitialized-Some additional notes:
|
Beta Was this translation helpful? Give feedback.
-
|
@if-mar Would you mind opening a PR with any additions etc to the guide I wrote that might be useful? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Starting with HCCM >= 1.24.0, it is sufficient to set HCLOUD_NETWORK_ROUTES_ENABLED=false, ROBOT_ENABLED=true and include the hcloud-user and password in the hcloud secret. No issues with the uninitialized taint. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @vitobotta, @if-mar Are there any updates on adding support for dedicated servers in hetzner-k3s? We really enjoy using hetzner-k3s, but we’ve run into a limitation: the cloud servers no longer provide enough storage for our needs, and network volumes are too slow for our database workloads when hosted on these instances. Because of this, we’re considering moving to dedicated servers. Ideally, we’d like to include these dedicated machines as part of our existing Kubernetes cluster, but we’re not entirely sure about the best way to approach this. I came across this thread and plan to give it a try, but I wanted to check if this is still the recommended (or only) solution. Thanks in advance for your help—and apologies if this isn’t the right place to ask! |
Beta Was this translation helpful? Give feedback.
-
|
Hi @data-trm - not yet unfortunately. I need to do this as soon as I can because we also need it at work for the same reason (storage). I have a lot on my plate at work plus other things and this task requires proper investigation so it needs some time. |
Beta Was this translation helpful? Give feedback.
Thanks a lot to @if-mar for providing me access to a dedicated server as well as a Hetzner project with the necessary vSwitch setup, so that I could conduct some investigation.
I got a cluster with the dedicated server working nicely, for the most part. Unfortunately it seems that even the latest version of the HCCM doesn't handle dedicated servers properly as it's supposed to.
These are the steps I took to get the cluster working more or less correctly:
1 - Network setup with vSwitch link - done by @if-mar , who could add some instructions here as well
2 - Create the cluster with the cloud nodes as usual
3 - Create the file
/etc/k8s-resolv.confon the dedicated server with the single line…