what's the best way to create a kubernetes cluster on proxmox (using pulumi, of course)

[ToddG]

create kubernetes clusters

How do you (best) create kubernetes clusters on proxmox?

I'm using microk8s, but I don't really know what's a good solution. For me, I want something that enables me to build an app suite composed of a website, postgres db, possibly a redis cache. I'll need to put it in a private network, allow some ingress from the outside world, and distribute traffic via a load balancer. Very similar to what this guy does with terraform on digital ocean:

https://www.digitalocean.com/community/tech-talks/building-a-minimal-production-ready-infrastructure-on-digitalocean

Of course I want to use pulumi to define and deploy my infrastructure and my application.

While working through a workaround to the cdrom drive issue, I mentioned that I'm cloning from cloud-init derived images and @lflorea-tc mentioned that they create their vms using direct full VM declarations, not template cloning. So I thought I'd create a thread here to flesh out the various pros and cons of how to create infra:

I'll start.

My use case: kubernetes cluster

1. download ISO from: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
2. manually convert ISO to templates, one for master, and one for workers... see attached convert-... script using cloud-config vendor files (attached)
3. use pulumi to create the vms by cloning the vms in the previous step
4. manually install missing microk8s add-ons that are, for whatever reason, not installing from the cloud-init file
5. manually find the join command on the k8s master
6. manually join the k8s workers to the master

This is a crap solution b/c

• manual steps don't scale
• looking at the proxmox GUI to find the ip address of a node to ssh in and do all this manual work is also a pita and doesn't scale

My questions:

• do you use k8s or something else?
• do you use cloud-configs or iso images
• how do you install packages and make other changes to the base image automatically? For example, (on ubuntu) installing stuff via apt or snap, updating config files, sysctemctl stuff, etc.

Attached are the files that I'm currently using for my manual steps... I'll abandon this stuff as I migrate to using pulumi for all this now that I'm unblocked.

cloud-init-ubuntu-base.yaml
prep.sh
cloud-init-ubuntu-mk8s-master.yaml
convert-ubuntu-iso-to-template.sh
mk8svms.py
cloud-init-ubuntu-mk8s-worker.yaml

[muhlba91]

hi,

i have converted your issue into a discussion as i believe this is a better place for it, especially because it's quite an opinionated topic. :)

you can find an example on how i manage a 3-node cluster using k0s on proxmox here: https://github.com/muhlba91/muehlbachler-public-services-infrastructure.

the gist is that i heavily use the Command provider to handle everything to be installed on the machine.

i also use a vendor config & cloud init to provide IP addresses, login user and SSH key and some basic packages and configuration: https://github.com/muhlba91/muehlbachler-public-services-infrastructure/blob/main/assets/vendor-config/cluster.yml.
however, i have seen in the past that with my usage of clusters, it's easier to only supply the basic cloud-init details and reduce the set of vendor config options to a bare minimum to just get the VM up and running. everything else i started handling with the Command provider. one of the reasons is that changing anything in there causes the VM to recreate (or reboot?) - either way, i was never able to safely manage such a change without issues in my cluster...

not using kubernetes though, but a very different example using plain docker compose is here: https://github.com/muhlba91/muehlbachler-mail-services-infrastructure. this is a Hetzner VM but shows my usage of the Command provider to configure everything accordingly.

[ToddG]

@muhlba91 Thank you for these examples! I think the Command provider is exactly what I am looking for. More soon.

[bartei]

I create my clusters in Proxmox with Nixos and Nix, I created a module for applying the nix configuration that runs from terraform currently and i think it can be easily ported and improved for pulumi, I have plans to experiment with pulumi as terraform+terragrunt which is my current setup is not very efficient.

With nixos+proxmox you can fully declaratively create kubernetes clusters and I've been very pleased with the results. I've been using it in production for over 2 years now, really like being able to easily update the clusters when needed and consistently configure the operating system without surprises.

Here is a link to my module if you're curious: https://gitlab.provvedo.com/provvedo/terraform-modules/nix-configuration

I can share here or somewhere you guys like my terraform configuration. I don't have a pulumi approach yet but i am planning to test it out soon by converting my module to a pulumi provider/module.

To me the power of Nix + IaC is unbeatable

[ToddG]

@bartei Woah. I finally get why nixos is so cool. I thought mise was a giant step forward, but now I see the magic of the entire os being declarative. Definitely share your terraform config here, too. I'll install nixos an get familiar with it and then circle back to using it in my pulumi IAC to create my k8s clusters.