linstor storage pool creation fail

[alcoralcor]

I can't find any similar issue.
This happened on incus 6.0 and 6.14
I use :

• OpenTofu 1.10.3
• ansible-core 2.18.1
• latest incus-deploy (b2bf217)
• python 1.13.1 (pyenv)

I follow the exact procedure but each time, I get the following error : "The satellite does not support the device provider LVM_THIN" on server01

TASK [lxc.incus.linstor : Create storage pool] *******************************************************************************************************************************
skipping: [server05] => (item=server01)
skipping: [server05] => (item=server02)
skipping: [server02] => (item=server01)
skipping: [server05] => (item=server03)
skipping: [server02] => (item=server02)
skipping: [server04] => (item=server01)
skipping: [server05] => (item=server04)
skipping: [server04] => (item=server02)
skipping: [server02] => (item=server03)
skipping: [server05] => (item=server05)
skipping: [server05]
skipping: [server02] => (item=server04)
skipping: [server04] => (item=server03)
skipping: [server03] => (item=server01)
skipping: [server02] => (item=server05)
skipping: [server02]
skipping: [server03] => (item=server02)
skipping: [server04] => (item=server04)
skipping: [server03] => (item=server03)
skipping: [server04] => (item=server05)
skipping: [server04]
skipping: [server03] => (item=server04)
skipping: [server03] => (item=server05)
skipping: [server03]
failed: [server01] (item=server01) => {"ansible_loop_var": "item", "changed": false, "cmd": ["linstor", "physical-storage", "create-device-pool", "--storage-pool", "incus", "--pool-name", "linstor-incus", "lvmthin", "server01", "/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_disk5"], "delta": "0:00:00.098983", "end": "2025-07-28 19:35:31.310855", "item": "server01", "msg": "non-zero return code", "rc": 10, "start": "2025-07-28 19:35:31.211872", "stderr": "", "stderr_lines": [], "stdout": "\u001b[1;31mERROR:\n\u001b[0m    The satellite does not support the device provider LVM_THIN", "stdout_lines": ["\u001b[1;31mERROR:", "\u001b[0m    The satellite does not support the device provider LVM_THIN"]}
changed: [server01] => (item=server02)
changed: [server01] => (item=server03)
changed: [server01] => (item=server04)
changed: [server01] => (item=server05)

PLAY RECAP *******************************************************************************************************************************************************************
server01                   : ok=74   changed=52   unreachable=0    failed=1    skipped=25   rescued=0    ignored=0
server02                   : ok=47   changed=34   unreachable=0    failed=0    skipped=47   rescued=0    ignored=0
server03                   : ok=47   changed=34   unreachable=0    failed=0    skipped=47   rescued=0    ignored=0
server04                   : ok=34   changed=22   unreachable=0    failed=0    skipped=58   rescued=0    ignored=0
server05                   : ok=34   changed=22   unreachable=0    failed=0    skipped=58   rescued=0    ignored=0

$ linstor storage-pool list
... does not list pool on server01

But :

$ incus project switch dev-incus-deploy
$ incus exec server01 bash
# linstor physical-storage create-device-pool --storage-pool incus --pool-name linstor-incus lvmthin server01 /dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_disk5
SUCCESS:
    (server01) PV for device '/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_disk5' created.
SUCCESS:
    (server01) VG for devices [/dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_disk5] with name 'linstor_linstor-incus' created.
SUCCESS:
    (server01) Thin-pool 'linstor-incus' in LVM-pool 'linstor_linstor-incus' created.
SUCCESS:
    Successfully set property key(s): StorDriver/StorPoolName
SUCCESS:
Description:
    New storage pool 'incus' on node 'server01' registered.
Details:
    Storage pool 'incus' on node 'server01' UUID is: 5f206d53-5705-49f9-bc9b-e3db32ea59cc
SUCCESS:
    (server01) Changes applied to storage pool 'incus'

$ linstor storage-pool list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node     ┊ Driver   ┊ PoolName                            ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                    ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ server01 ┊ DISKLESS ┊                                     ┊              ┊               ┊ False        ┊ Ok    ┊ server01;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server02 ┊ DISKLESS ┊                                     ┊              ┊               ┊ False        ┊ Ok    ┊ server02;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server03 ┊ DISKLESS ┊                                     ┊              ┊               ┊ False        ┊ Ok    ┊ server03;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server04 ┊ DISKLESS ┊                                     ┊              ┊               ┊ False        ┊ Ok    ┊ server04;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ server05 ┊ DISKLESS ┊                                     ┊              ┊               ┊ False        ┊ Ok    ┊ server05;DfltDisklessStorPool ┊
┊ incus                ┊ server01 ┊ LVM_THIN ┊ linstor_linstor-incus/linstor-incus ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server01;incus                ┊
┊ incus                ┊ server02 ┊ LVM_THIN ┊ linstor_linstor-incus/linstor-incus ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server02;incus                ┊
┊ incus                ┊ server03 ┊ LVM_THIN ┊ linstor_linstor-incus/linstor-incus ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server03;incus                ┊
┊ incus                ┊ server04 ┊ LVM_THIN ┊ linstor_linstor-incus/linstor-incus ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server04;incus                ┊
┊ incus                ┊ server05 ┊ LVM_THIN ┊ linstor_linstor-incus/linstor-incus ┊    49.89 GiB ┊     49.89 GiB ┊ True         ┊ Ok    ┊ server05;incus                ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

[stgraber]

I know we've seen some weirdness with Linstor on LVM recently and as a result we switched the Incus tests at least to use Linstor on top of ZFS instead which has so far been behaving for us.

[stgraber]

@luissimas any idea?

[luissimas]

This looks like some sort of race condition in the playbook. I'd assume we're trying to create the LVM PVs and VG via linstor physical-storage create-device-pool before Linstor knows that the current node supports LVM. If that's the case, we might add a check for the satellite's capabilities on the playbook.

@alcoralcor a couple of follow up questions:

1. Does running the playbook a second time after the first fail result in the same error?
2. Did you restart the linstor-satellite service on server01 before manually running linstor physical-storage create-device-pool ...?

[stgraber]

Ah, as in, we'd be installing and starting the satellite before the relevant LVM packages are installed?

[luissimas]

Yeah, I think that might be the case. We currently install all packages in a single task:

incus-deploy/roles/linstor/tasks/installation.yml

Lines 19 to 29 in 3e06839

	- name: Install linstor-satellite
	ansible.builtin.apt:
	name:
	- drbd-dkms
	- drbd-utils
	- linstor-satellite
	- lvm2
	- python3-setuptools
	- zfsutils-linux
	state: present
	when: '"satellite" in linstor_roles'

IIRC the linstor-satellite service gathers the information about supported storage back-ends only at startup. If for some reason the satellite service starts before LVM is properly setup on the host, we might end up in this situation. I'm not sure if that's possible tho, as I'm not very familiar with what guarantees apt gives us when installing multiple packages at the same time. But it seems reasonable to me that we could have a race condition between the LVM and Linstor services.

If that's the case, I think it would only be fixed with a restart of the linstor-satellite service, hence the follow up question number 2.

[alcoralcor]

I did not restart the linstor-satellite service ...

[stgraber]

It would depend on whether there is a dependency/recommend defined in the satellite packages, if there is then the order should be fine, if not, then the exact order is mostly random.

Given the satellite detects the supported backends on startup, it may make sense to split the package install into two, installing lvm2 and zfs before installing the rest, that should avoid the situation.

[alcoralcor]

I tried to apply a second time the deploy.yml playbook (without cleaning cache data/ directories) on a fresh deploy. It now creates the pool on server01.
(the script fails now later, on server01 - Set OVN NorthBound database, but I will look into this issue separately)

[luissimas]

Yeah, installing the packages in two different tasks might be the simplest solution if the issue is indeed a race condition.

But since @alcoralcor mentioned that the linstor-satellite service was not restarted I'm not so sure about my assumption that Linstor picks up the supported backends only at startup. Besides that, it might still be a race condition after all.

If the storage pool can be created by running the playbook a second time, then I'm quite confident that it's some sort of race condition.

[alcoralcor]

I added some debug to storage.yml to display services status before creating storage pools but I don't see any differences between hosts ... except that server01 has linstor-controller (running). All linstor-satellite.service are running