Use in storage-constrained environments?

[jacobdanovitch]

Hi there,

I was hoping to use Kairos in an Openstack environment; however, the root disk of each server is capped at 20GB, with 30+ GB of ephemeral storage attached at /mnt. I have tried to change the layout and install with --no-format but it doesn't seem to change the storage requirement. Is it possible to use Kairos in this scenario?

Thanks in advance.

[mudler]

Hi!

It's not throughly documented yet, but it's possible to specify a different partitioning layout. I'll try to get back at you as soon as possible with a full sample to add im the docs.

The ARM images for example does fit 16G. Are you trying with a custom raw image? Can you provide more logs?

[jacobdanovitch]

Thanks for the quick reply! I was previously getting a brief error from Elemental saying that there weren't enough sectors available. I do remember seeing the RPI tutorial that specified a disk size so I went to check the docs and noticed the /advanced/build guide on building cloud provider images, and I found a way to make it work with Openstack. I used the following OSArtifact to build a raw disk image:

osartifact.yaml

apiVersion: build.kairos.io/v1alpha1
kind: OSArtifact
metadata:
  name: kairos-opensuse
spec:
  imageName: "quay.io/kairos/kairos-opensuse:latest"
  cloudImage: true
  cloudConfig: |
    #cloud-config
    users:
    - name: "kairos"
      passwd: "kairos"
    
    name: "Default deployment"

    k3s:
      enabled: true
      args:
        - --write-kubeconfig-mode="644"
    
    p2p:
      network_token: ...

    stages:
      boot:
      - name: "Fetch cloud provider's user data"
        datasource:
          providers:
          - "openstack"
          path: "/etc/cloud-data"
      - name: "Repart image"
        layout:
          device:
            label: COS_RECOVERY
          add_partitions:
            - fsLabel: COS_STATE
              size: 16240 # At least 16gb
              pLabel: state
      - name: "Repart image"
        layout:
          device:
            label: COS_RECOVERY
          add_partitions:
            - fsLabel: COS_PERSISTENT
              pLabel: persistent
              size: 0 # all space
              filesystem: "ext4"
      - if: '[ -f "/run/cos/recovery_mode" ] && [ ! -e /usr/local/.deployed ]'
        name: "Deploy cos-system"
        commands:
          - |
              # Use `elemental reset --system.uri docker:<img-ref>` to deploy a custom image
              elemental reset && \
              touch /usr/local/.deployed && \
              reboot

However, when I tried to upload the image to Openstack and boot a VM from it, it would hang at "Booting from hard disk ...", as it would when I ran it locally with QEMU. The trick is that, like with the other cloud providers, you have to explicitly tell Openstack to boot the image using UEFI. The process isn't well documented by Openstack as far as I could find, but this did the trick:

openstack image create --progress --disk-format raw  --property hw_firmware_type='uefi' --property hw_machine_type='q35' --property architecture='x86_64' --file $DISK_PATH kairos-opensuse

I booted two nodes from the image and after the reboot, they connected over p2p and formed a running k3s cluster! This is awesome, amazing work on this project. I just have a couple of follow up questions:

1. The additional disk of ephemeral storage attached at /dev/vdb doesn't get mounted with this setup, which I assume is due to the layout config I've specified. Will the upcoming custom partitioning docs cover that? And will it be possible to provide optional mounts, in the case that a node doesn't have any additional disks?
2. Is the reset and reboot stage necessary? It obviously takes a while to complete and it seemed to cause problems with the ubuntu-20-lts or alpine-ubuntu variants, as the nodes just perpetually booted from recovery. The opensuse variant had no issue with this, though.

Thanks again!