Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add linstor storage driver #1621

Draft
wants to merge 43 commits into
base: main
Choose a base branch
from
Draft

Add linstor storage driver #1621

wants to merge 43 commits into from

Conversation

luissimas
Copy link
Contributor

Description

This PR introduces a new linstor storage driver to enable provisioning of volumes backed by LINSTOR, as discussed in #564. Currently only the following core features are implemented:

  • Creating and deleting storage pools
  • Creating, deleting and attaching custom volumes
  • Creating and deleting instances with a linstor root volume

Other features such as snapshots, migrating volumes, optimized images, etc are planned to be handled in subsequent PRs.

For an easy way to deploy Linstor alongside Incus, check out lxc/incus-deploy#20.

Example

The following example shows some simple use cases that were tested while developing the core driver functionality. In this example we set the storage.linstor.controller_connection option and create a new linstor storage pool. We then create a virtual machine v1 using linstor as the storage pool for the root volume. We then create a custom block volume data, attach it to the virtual machine and write some data to it. Then, we delete the v1 instance and create a new v2 instance in another node. Finally, we verify that we can attach the data volume to the new instance and retrieve its data.

root@server01:~# incus config set storage.linstor.controller_connection=http://10.172.117.216:3370
root@server01:~# linstor resource-group list
╭──────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter  ┊ VlmNrs ┊ Description ┊
╞══════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2 ┊        ┊             ┊
╰──────────────────────────────────────────────────────╯
root@server01:~# incus storage list
+-------+--------+--------------------+---------+---------+
| NAME  | DRIVER |    DESCRIPTION     | USED BY |  STATE  |
+-------+--------+--------------------+---------+---------+
| local | zfs    | Local storage pool | 0       | CREATED |
+-------+--------+--------------------+---------+---------+
root@server01:~# incus storage create linstor linstor --target server01
Storage pool linstor pending on member server01
root@server01:~# incus storage create linstor linstor --target server02
Storage pool linstor pending on member server02
root@server01:~# incus storage create linstor linstor --target server03
Storage pool linstor pending on member server03
root@server01:~# incus storage create linstor linstor
Storage pool linstor created
root@server01:~# incus storage list
+---------+---------+--------------------+---------+---------+
|  NAME   | DRIVER  |    DESCRIPTION     | USED BY |  STATE  |
+---------+---------+--------------------+---------+---------+
| linstor | linstor |                    | 0       | CREATED |
+---------+---------+--------------------+---------+---------+
| local   | zfs     | Local storage pool | 0       | CREATED |
+---------+---------+--------------------+---------+---------+
root@server01:~# linstor resource-group list
╭──────────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter  ┊ VlmNrs ┊ Description                     ┊
╞══════════════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp    ┊ PlaceCount: 2 ┊        ┊                                 ┊
┊ incus         ┊ PlaceCount: 2 ┊        ┊ Resource group managed by Incus ┊
╰──────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus launch images:ubuntu/24.04 v1 --vm --storage linstor
Launching v1
root@server01:~# incus ls
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| NAME |  STATE  |         IPV4         |                      IPV6                       |      TYPE       | SNAPSHOTS | LOCATION |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
| v1   | RUNNING | 10.127.79.2 (enp5s0) | fd42:15af:690c:a4a3:216:3eff:fe41:d8bf (enp5s0) | VIRTUAL-MACHINE | 0         | server01 |
+------+---------+----------------------+-------------------------------------------------+-----------------+-----------+----------+
root@server01:~# linstor resource-definition list
╭───────────────────────────────────────────────────────────────────╮
┊ ResourceName        ┊ Port ┊ ResourceGroup ┊ Layers       ┊ State ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ incus-v1_block      ┊ 7000 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
┊ incus-v1_filesystem ┊ 7001 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
╰───────────────────────────────────────────────────────────────────╯
root@server01:~# linstor resource list
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName        ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-v1_block      ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block      ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block      ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:21 ┊
┊ incus-v1_filesystem ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
┊ incus-v1_filesystem ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:23 ┊
┊ incus-v1_filesystem ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus storage volume create linstor data --type block
Storage volume data created
root@server01:~# linstor resource-definition list
╭────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Port ┊ ResourceGroup ┊ Layers       ┊ State ┊
╞════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ 7002 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
┊ incus-v1_block           ┊ 7000 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
┊ incus-v1_filesystem      ┊ 7001 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
╰────────────────────────────────────────────────────────────────────────╯
root@server01:~# linstor resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:14:58 ┊
┊ incus-default_data_block ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-default_data_block ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-v1_block           ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block           ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block           ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:21 ┊
┊ incus-v1_filesystem      ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
┊ incus-v1_filesystem      ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:23 ┊
┊ incus-v1_filesystem      ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus exec v1 lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk
├─sda1   8:1    0  100M  0 part /boot/efi
└─sda2   8:2    0  9.9G  0 part /
root@server01:~# incus storage volume attach linstor data v1
root@server01:~# linstor resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:14:58 ┊
┊ incus-default_data_block ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-default_data_block ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-v1_block           ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block           ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:22 ┊
┊ incus-v1_block           ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:21 ┊
┊ incus-v1_filesystem      ┊ server01 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
┊ incus-v1_filesystem      ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:10:23 ┊
┊ incus-v1_filesystem      ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:10:24 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus exec v1 lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk
├─sda1   8:1    0  100M  0 part /boot/efi
└─sda2   8:2    0  9.9G  0 part /
sdb      8:16   0   10G  0 disk
root@server01:~# incus exec v1 bash
root@v1:~# mkfs.ext4 /dev/sdb
mke2fs 1.47.0 (5-Feb-2023)
Discarding device blocks: mdone
Creating filesystem with 2621440 4k blocks and 656640 inodes
Filesystem UUID: 894338ad-795f-49ba-893d-4fdd4d978f16
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
oCreating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: unt done

root@v1:~# mount /dev/sdb /mnt
root@v1:~# echo "Hello from $(hostname) at $(date)" >> /mnt/file.txt
root@v1:~# umount /mnt
root@v1:~#
exit
root@server01:~# incus stop v1 && incus rm v1
root@server01:~# linstor resource-definition list
╭────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Port ┊ ResourceGroup ┊ Layers       ┊ State ┊
╞════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ 7002 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
╰────────────────────────────────────────────────────────────────────────╯
root@server01:~# linstor resource list
╭──────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞══════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ Diskless ┊ 2025-02-01 00:14:58 ┊
┊ incus-default_data_block ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-default_data_block ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2025-02-01 00:14:59 ┊
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus launch images:ubuntu/24.04 v2 --vm --storage linstor --target server02
Launching v2
root@server01:~# linstor resource-definition list
╭────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Port ┊ ResourceGroup ┊ Layers       ┊ State ┊
╞════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ 7002 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
┊ incus-v2_block           ┊ 7000 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
┊ incus-v2_filesystem      ┊ 7001 ┊ incus         ┊ DRBD,STORAGE ┊ ok    ┊
╰────────────────────────────────────────────────────────────────────────╯
root@server01:~# linstor resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:14:58 ┊
┊ incus-default_data_block ┊ server02 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-default_data_block ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-v2_block           ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:39 ┊
┊ incus-v2_block           ┊ server02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:39 ┊
┊ incus-v2_block           ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:21:37 ┊
┊ incus-v2_filesystem      ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:42 ┊
┊ incus-v2_filesystem      ┊ server02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:21:40 ┊
┊ incus-v2_filesystem      ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:41 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus storage volume attach linstor data v2
root@server01:~# linstor resource list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName             ┊ Node     ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-default_data_block ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:14:58 ┊
┊ incus-default_data_block ┊ server02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-default_data_block ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:14:59 ┊
┊ incus-v2_block           ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:39 ┊
┊ incus-v2_block           ┊ server02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:39 ┊
┊ incus-v2_block           ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-02-01 00:21:37 ┊
┊ incus-v2_filesystem      ┊ server01 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:42 ┊
┊ incus-v2_filesystem      ┊ server02 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   Diskless ┊ 2025-02-01 00:21:40 ┊
┊ incus-v2_filesystem      ┊ server03 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-02-01 00:21:41 ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────╯
root@server01:~# incus exec v2 bash
root@v2:~# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda      8:0    0   10G  0 disk
├─sda1   8:1    0  100M  0 part /boot/efi
└─sda2   8:2    0  9.9G  0 part /
sdb      8:16   0   10G  0 disk
root@v2:~# mount /dev/sdb /mnt
croot@v2:~# cat /mnt/file.txt
Hello from v1 at Sat Feb  1 00:19:26 UTC 2025

@luissimas luissimas requested a review from stgraber as a code owner February 1, 2025 00:36
@github-actions github-actions bot added the Documentation Documentation needs updating label Feb 1, 2025
@luissimas
Copy link
Contributor Author

We expect to send integration tests validating the scenarios in the next few days. I think we'll probably need some Linstor setup on the testing environment as well.

@luissimas luissimas force-pushed the linstor branch 2 times, most recently from ea94519 to cf18bb1 Compare February 1, 2025 01:09
@stgraber
Copy link
Member

stgraber commented Feb 2, 2025

There's a bunch of static analysis failure around code styling/formatting that should be pretty easy to handle.

@luissimas luissimas force-pushed the linstor branch 4 times, most recently from 5723ea8 to 6d4dede Compare February 3, 2025 19:54
func (d *Daemon) getLinstor() (*linstor.Client, error) {
// Setup the client if it does not exist
if d.linstor == nil {
err := d.setupLinstor()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m seeing a small race condition leading to a double execution of setupLinstor:

  • getLinstor gets called until after the d.linstor == nil condition is checked
  • setupLinstor gets called before err := d.setupLinstor() gets called
  • setupLinstor gets called again by err := d.setupLinstor().

I’d put a synchronization here…

@luissimas luissimas force-pushed the linstor branch 5 times, most recently from 7c04d43 to 3d7e8c7 Compare February 5, 2025 21:22
@luissimas
Copy link
Contributor Author

Hey @stgraber just a quick update on the state of things. @winiciusallan, @bensmrs and I did a quick call about the current state of the implementation and the next steps. Here are the main takeaways:

  1. @bensmrs is working on the CI tests for the driver
  2. While looking at the tests, we've identified that the lack of support for storage buckets may be a block for merging this, at least from the test's perspective. Can you confirm that?
  3. @winiciusallan will look into how much work would it take to support storage buckets in the linstor driver
  4. Renaming volumes is most likely a block for merging. Our initial approach will be to use the volume ID in the Incus database as the Resource Definition name in LINSTOR. We'll need to check if the ID is available on all code paths that create the volumes. We also need to be aware of the 48 character limit on LINSTOR object names. @bensmrs is probably the best person to tackle this, as he is the most familiar with the Incus code base.
  5. We still need to figure out a good way to validate that all Incus nodes also run the linstor-satellite service and that the node names are consistent between Incus and LINSTOR
  6. I'll finish addressing @bensmrs's comments on this PR and then I'll check if it is possible to have multiple LINSTOR controllers managing the same set of satellites. I'm guessing that is not supported, but if it is, then we should set the controller connection string at the storage pool level rather than in a global config option

@luissimas luissimas force-pushed the linstor branch 3 times, most recently from c38398e to 120ae22 Compare February 6, 2025 11:17
@luissimas luissimas changed the title Add Linstor storage driver Add linstor storage driver Feb 6, 2025
@github-actions github-actions bot added the API Changes to the REST API label Feb 6, 2025
@luissimas luissimas marked this pull request as draft February 6, 2025 21:35
luissimas and others added 2 commits March 7, 2025 08:54
…hot volumes

Signed-off-by: Luís Simas <[email protected]>
Co-authored-by: Benjamin Somers <[email protected]>
Signed-off-by: Luís Simas <[email protected]>
Co-authored-by: Benjamin Somers <[email protected]>
@luissimas luissimas force-pushed the linstor branch 2 times, most recently from 5fca045 to a2e483d Compare March 7, 2025 12:33
@luissimas luissimas force-pushed the linstor branch 2 times, most recently from 8045125 to 672bdc5 Compare March 7, 2025 20:46
bensmrs and others added 22 commits March 7, 2025 18:27
Signed-off-by: Benjamin Somers <[email protected]>
Co-authored-by: Luís Simas <[email protected]>
Signed-off-by: Benjamin Somers <[email protected]>
Signed-off-by: Benjamin Somers <[email protected]>
…ng of volumes with snapshots

Signed-off-by: Luís Simas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Changes to the REST API Documentation Documentation needs updating
Development

Successfully merging this pull request may close these issues.

4 participants