cluster support

Author: terion-labs

README.md

@@ -52,7 +52,8 @@ Terrarium provisions the host with:
 ## Supported Host
 
 - Ubuntu Server 24.04 LTS
-- Single-host install only
+- Single-host install by default
+- Experimental multi-node LXD clustering through `terrariumctl cluster ...`
 - LXC containers only
 
 ## Install Modes
@@ -185,6 +186,39 @@ What gets updated on change:
 - Self-hosted ZITADEL is enabled, disabled, restarted, and reconciled when its rendered config changes.
 - Terrarium then re-runs `terrariumctl proxy sync`, and when IDP mode is `local`, also re-runs `terrariumctl idp sync`.
 
+## Clustering
+
+Terrarium clustering uses LXD's native cluster membership and dqlite state. The
+Terrarium config is stored in LXD project metadata, so joined nodes can export
+the same config and reconfigure locally.
+
+Typical first member:
+
+```bash
+terrariumctl cluster init \
+  --member node1 \
+  --address 10.0.0.11:8443 \
+  --central-addresses 10.0.0.11,10.0.0.12,10.0.0.13 \
+  --peer-cidr 10.0.0.0/24
+```
+
+Typical additional member:
+
+```bash
+terrariumctl cluster token node2
+terrariumctl cluster join --token '<token>' --address 10.0.0.12:8443 --peer-cidr 10.0.0.0/24 --yes
+```
+
+Terrarium creates an OVN workload network named `terrarium-ovn` and points the
+default LXD profiles at it. Use at least three cluster members for a real
+quorum-tolerant setup; clustering does not make local ZFS storage magically
+shared.
+
+`lxdbr0` stays as the managed parent/uplink network. Host-side Traefik reaches
+published OVN workloads through Terrarium-managed LXD `proxy` devices bound to
+loopback, so it does not depend on direct host routing to private OVN instance
+addresses.
+
 ## terrariumctl Reference
 
 Top-level commands:
@@ -199,6 +233,11 @@ Top-level commands:
 | `terrariumctl reconfigure` | none | n/a | Re-runs the local Ansible reconciliation using the saved config. |
 | `terrariumctl config import` | none | n/a | Imports `/etc/terrarium/config.yaml` into the LXD dqlite-backed config store. |
 | `terrariumctl config export` | none | n/a | Recreates `/etc/terrarium/config.yaml` from the LXD dqlite-backed config store. |
+| `terrariumctl cluster status` | none | n/a | Shows LXD cluster state and the Terrarium OVN network. |
+| `terrariumctl cluster init` | required: `--member`, `--address`; optional: `--central-addresses`, `--peer-cidr` | network `terrarium-ovn`, parent `lxdbr0` | Enables LXD clustering on the first member and reconciles Terrarium OVN networking. |
+| `terrariumctl cluster token` | required: member name | n/a | Mints a single-use LXD cluster join token. |
+| `terrariumctl cluster join` | required: `--token`, `--address`; optional: `--peer-cidr`, `--yes` | storage pool `terrarium` | Joins the local node to an existing LXD cluster, exports shared Terrarium config, and reconfigures. |
+| `terrariumctl cluster ovn configure` | optional: `--central-addresses`, `--peer-cidr` | network `terrarium-ovn`, parent `lxdbr0` | Updates Terrarium OVN central member and peer firewall settings. |
 | `terrariumctl proxy sync` | none | n/a | Rebuilds Traefik dynamic config and Terrarium-managed UFW rules from LXC `user.proxy` labels. |
 | `terrariumctl mount add` | required: `protocol`, `hostPath`, `address`, `username`; optional: `-p/--password`, `--password-file`, `--seal` | password prompt, `uid=0`, `gid=0`, `file_mode=0660`, `dir_mode=0770`, `--seal=true` | Creates a managed host SMB/CIFS mount, stores credentials under `/etc/terrarium/mounts`, writes a managed `/etc/fstab` block, and mounts it immediately. |
 | `terrariumctl mount remove` | required: `hostPath` | n/a | Unmounts a Terrarium-managed host mount, removes its managed `/etc/fstab` block, and deletes its managed credentials file. |
@@ -410,7 +449,8 @@ Rules:
 - `udp://hostport:containerport` exposes a raw UDP port through Traefik.
 - Dynamic TCP/UDP host ports are also opened and closed in UFW automatically by the sync job.
 - Auth-protected published routes are backed by host-side `oauth2-proxy` instances managed automatically by `terrariumctl proxy sync`.
-- If the container does not have a global IPv4 address yet, the route is skipped until it does.
+- `terrariumctl proxy sync` also reconciles host-loopback LXD `proxy` devices for published container backends and points Traefik at those localhost targets.
+- If a backend proxy device cannot be reconciled, the sync fails without publishing a partially broken Traefik config.
 
 ## Development
 

ansible/roles/base/tasks/main.yml

@@ -99,6 +99,22 @@
     - "80"
     - "443"
 
+- name: Allow LXD cluster API from configured peer networks
+  community.general.ufw:
+    rule: allow
+    from_ip: "{{ item }}"
+    port: "8443"
+    proto: tcp
+  loop: "{{ terrarium_cluster_peer_cidrs | default([]) }}"
+
+- name: Allow OVN cluster database and overlay traffic from configured peer networks
+  community.general.ufw:
+    rule: allow
+    from_ip: "{{ item.0 }}"
+    port: "{{ item.1.port }}"
+    proto: "{{ item.1.proto }}"
+  loop: "{{ (terrarium_cluster_peer_cidrs | default([])) | product([{'port': '6081', 'proto': 'udp'}, {'port': '6641', 'proto': 'tcp'}, {'port': '6642', 'proto': 'tcp'}]) | list }}"
+
 - name: Enable UFW
   community.general.ufw:
     state: enabled

ansible/roles/lxd/defaults/main.yml

@@ -2,3 +2,11 @@ terrarium_lxd_snap_channel: latest/stable
 terrarium_lxd_auth_group: terrarium-management
 terrarium_lxd_oidc_groups_claim: groups
 terrarium_lxd_idp_group_state_path: "{{ terrarium_state_dir }}/lxd-idp-admin-group"
+terrarium_cluster_enabled: false
+terrarium_lxd_network_parent: lxdbr0
+terrarium_lxd_network_name: terrarium-ovn
+terrarium_lxd_parent_ipv4: 10.154.0.1/24
+terrarium_lxd_parent_dhcp_range: 10.154.0.2-10.154.0.199
+terrarium_lxd_parent_ovn_range: 10.154.0.200-10.154.0.254
+terrarium_ovn_central_addresses: []
+terrarium_ovn_local_address: "{{ ansible_default_ipv4.address }}"

ansible/roles/lxd/tasks/main.yml

@@ -5,13 +5,13 @@
   changed_when: false
 
 - name: Install LXD snap when missing
-  ansible.builtin.command: snap install lxd --channel={{ terrarium_lxd_snap_channel }}
+  ansible.builtin.command: /usr/bin/timeout --kill-after=15s 180s snap install lxd --channel={{ terrarium_lxd_snap_channel }}
   register: terrarium_lxd_snap_install
   until:
     - terrarium_lxd_snap_install.rc == 0
       or 'already installed' in (terrarium_lxd_snap_install.stdout | lower)
       or 'already installed' in (terrarium_lxd_snap_install.stderr | lower)
-  retries: 30
+  retries: 10
   delay: 10
   changed_when:
     - terrarium_lxd_snap_install.rc == 0
@@ -147,15 +147,103 @@
   delay: 10
   changed_when: false
 
+- name: Install OVN networking packages
+  ansible.builtin.apt:
+    name:
+      - openvswitch-switch
+      - ovn-central
+      - ovn-host
+    state: present
+
+- name: Resolve OVN database connection strings
+  ansible.builtin.set_fact:
+    terrarium_ovn_northbound_connection: "{{ terrarium_ovn_central_addresses | map('regex_replace', '^(.*)$', 'tcp:\\1:6641') | join(',') }}"
+    terrarium_ovn_southbound_connection: "{{ terrarium_ovn_central_addresses | map('regex_replace', '^(.*)$', 'tcp:\\1:6642') | join(',') }}"
+
+- name: Configure clustered OVN central service
+  ansible.builtin.template:
+    src: ovn-central.j2
+    dest: /etc/default/ovn-central
+    mode: "0644"
+  when:
+    - terrarium_ovn_central_addresses | length > 0
+    - terrarium_ovn_local_address in terrarium_ovn_central_addresses
+  register: terrarium_ovn_central_config
+
+- name: Enable OVN central on standalone or central members
+  ansible.builtin.systemd:
+    name: ovn-central
+    state: "{{ 'restarted' if terrarium_ovn_central_config.changed | default(false) else 'started' }}"
+    enabled: true
+  when: terrarium_ovn_central_addresses | length == 0 or terrarium_ovn_local_address in terrarium_ovn_central_addresses
+
+- name: Disable OVN central on non-central cluster members
+  ansible.builtin.systemd:
+    name: ovn-central
+    state: stopped
+    enabled: false
+  when:
+    - terrarium_ovn_central_addresses | length > 0
+    - terrarium_ovn_local_address not in terrarium_ovn_central_addresses
+
+- name: Enable OVN host service
+  ansible.builtin.systemd:
+    name: ovn-host
+    state: started
+    enabled: true
+
+- name: Configure Open vSwitch for Terrarium OVN
+  ansible.builtin.command:
+    argv:
+      - ovs-vsctl
+      - set
+      - open_vswitch
+      - "."
+      - "external_ids:ovn-remote={{ (terrarium_ovn_central_addresses | length > 0) | ternary(terrarium_ovn_southbound_connection, 'unix:/var/run/ovn/ovnsb_db.sock') }}"
+      - "external_ids:ovn-encap-type=geneve"
+      - "external_ids:ovn-encap-ip={{ (terrarium_ovn_central_addresses | length > 0) | ternary(terrarium_ovn_local_address, '127.0.0.1') }}"
+  register: terrarium_ovs_configure
+  until: terrarium_ovs_configure.rc == 0
+  retries: 12
+  delay: 5
+  changed_when: false
+
+- name: Configure LXD OVN northbound database connection
+  ansible.builtin.command: "/snap/bin/lxc config set network.ovn.northbound_connection {{ terrarium_ovn_northbound_connection }}"
+  when: terrarium_ovn_central_addresses | length > 0
+  changed_when: false
+
+- name: Configure LXD OVN parent network ranges
+  ansible.builtin.command: >-
+    /snap/bin/lxc network set {{ terrarium_lxd_network_parent }}
+    ipv4.dhcp.ranges={{ terrarium_lxd_parent_dhcp_range }}
+    ipv4.ovn.ranges={{ terrarium_lxd_parent_ovn_range }}
+  changed_when: false
+
+- name: Check whether Terrarium OVN workload network exists
+  ansible.builtin.command: "/snap/bin/lxc network show {{ terrarium_lxd_network_name }}"
+  register: terrarium_lxd_ovn_network
+  failed_when: false
+  changed_when: false
+
+- name: Create Terrarium OVN workload network
+  ansible.builtin.command: "/snap/bin/lxc network create {{ terrarium_lxd_network_name }} --type=ovn network={{ terrarium_lxd_network_parent }}"
+  register: terrarium_lxd_ovn_network_create
+  until: terrarium_lxd_ovn_network_create.rc == 0
+  retries: 12
+  delay: 5
+  when: terrarium_lxd_ovn_network.rc != 0
+
 - name: Pin LXD API to loopback
   ansible.builtin.command: /snap/bin/lxc config set core.https_address 127.0.0.1:8443
   changed_when: false
+  when: not (terrarium_cluster_enabled | bool)
 
 - name: Allow LXD bridge DHCP and DNS through UFW
   community.general.ufw:
     rule: allow
     direction: in
-    interface: lxdbr0
+    interface: "{{ terrarium_lxd_network_parent }}"
     proto: "{{ item.proto }}"
     port: "{{ item.port }}"
   loop:
@@ -167,7 +255,7 @@
   community.general.ufw:
     rule: allow
     route: true
-    interface_in: lxdbr0
+    interface_in: "{{ terrarium_lxd_network_parent }}"
     interface_out: "{{ ansible_default_ipv4.interface }}"
 
 - name: Disable LXD ACME certificate management

ansible/roles/lxd/templates/default-profile.yml.j2

@@ -7,7 +7,7 @@ description: Terrarium default container profile
 devices:
   eth0:
     name: eth0
-    network: lxdbr0
+    network: {{ terrarium_lxd_network_name }}
     type: nic
   root:
     path: /

ansible/roles/lxd/templates/lxd-preseed.yml.j2

@@ -1,10 +1,12 @@
 config:
   core.https_address: 127.0.0.1:8443
 networks:
-  - name: lxdbr0
+  - name: {{ terrarium_lxd_network_parent }}
     type: bridge
     config:
-      ipv4.address: auto
+      ipv4.address: {{ terrarium_lxd_parent_ipv4 }}
+      ipv4.dhcp.ranges: {{ terrarium_lxd_parent_dhcp_range }}
+      ipv4.ovn.ranges: {{ terrarium_lxd_parent_ovn_range }}
       ipv4.nat: "true"
       ipv6.address: none
 storage_pools:
@@ -24,7 +26,7 @@ profiles:
       eth0:
         type: nic
         name: eth0
-        network: lxdbr0
+        network: {{ terrarium_lxd_network_parent }}
       root:
         type: disk
         path: /

ansible/roles/lxd/templates/ovn-central.j2

@@ -0,0 +1 @@
+OVN_CTL_OPTS="--db-nb-addr={{ terrarium_ovn_local_address }} --db-nb-create-insecure-remote=yes --db-sb-addr={{ terrarium_ovn_local_address }} --db-sb-create-insecure-remote=yes{% if terrarium_ovn_local_address != terrarium_ovn_central_addresses[0] %} --db-nb-cluster-remote-addr={{ terrarium_ovn_central_addresses[0] }} --db-sb-cluster-remote-addr={{ terrarium_ovn_central_addresses[0] }}{% endif %} --db-nb-cluster-local-addr={{ terrarium_ovn_local_address }} --db-sb-cluster-local-addr={{ terrarium_ovn_local_address }} --ovn-northd-nb-db={{ terrarium_ovn_northbound_connection }} --ovn-northd-sb-db={{ terrarium_ovn_southbound_connection }}"

ansible/roles/lxd/templates/strict-profile.yml.j2

@@ -4,7 +4,7 @@ description: Terrarium stricter container profile without Docker-friendly nestin
 devices:
   eth0:
     name: eth0
-    network: lxdbr0
+    network: {{ terrarium_lxd_network_name }}
     type: nic
   root:
     path: /

ansible/roles/lxd/templates/terrarium-profile.yml.j2

@@ -7,7 +7,7 @@ description: Terrarium baseline container profile, kept as an explicit alias for
 devices:
   eth0:
     name: eth0
-    network: lxdbr0
+    network: {{ terrarium_lxd_network_name }}
     type: nic
   root:
     path: /

ansible/site.yml

@@ -7,6 +7,15 @@
     terrarium_config_path: /etc/terrarium/config.yaml
     terrarium_state_dir: /var/lib/terrarium
     terrarium_lxd_pool_name: terrarium
+    terrarium_cluster_enabled: false
+    terrarium_cluster_peer_cidrs: []
+    terrarium_lxd_network_parent: lxdbr0
+    terrarium_lxd_network_name: terrarium-ovn
+    terrarium_lxd_parent_ipv4: 10.154.0.1/24
+    terrarium_lxd_parent_dhcp_range: 10.154.0.2-10.154.0.199
+    terrarium_lxd_parent_ovn_range: 10.154.0.200-10.154.0.254
+    terrarium_ovn_central_addresses: []
+    terrarium_ovn_local_address: "{{ ansible_default_ipv4.address }}"
     terrarium_storage_mode: file
     terrarium_storage_source: ""
     terrarium_storage_size: ""
@@ -61,6 +70,14 @@
       terrarium_admin_group: "{{ terrarium_admin_group }}"
       terrarium_zitadel_admin_email: "{{ terrarium_zitadel_admin_email }}"
       terrarium_lxd_pool_name: "{{ terrarium_lxd_pool_name }}"
+      terrarium_cluster_enabled: "{{ terrarium_cluster_enabled }}"
+      terrarium_cluster_peer_cidrs: "{{ terrarium_cluster_peer_cidrs }}"
+      terrarium_lxd_network_parent: "{{ terrarium_lxd_network_parent }}"
+      terrarium_lxd_network_name: "{{ terrarium_lxd_network_name }}"
+      terrarium_lxd_parent_ipv4: "{{ terrarium_lxd_parent_ipv4 }}"
+      terrarium_lxd_parent_dhcp_range: "{{ terrarium_lxd_parent_dhcp_range }}"
+      terrarium_lxd_parent_ovn_range: "{{ terrarium_lxd_parent_ovn_range }}"
+      terrarium_ovn_central_addresses: "{{ terrarium_ovn_central_addresses }}"
       terrarium_storage_mode: "{{ terrarium_storage_mode }}"
       terrarium_storage_source: "{{ terrarium_storage_source }}"
       terrarium_storage_size: "{{ terrarium_storage_size }}"