Support creating and mounting Btrfs subvolumes

[laenion]

Feature Request

Btrfs is a supported file system for Ignition, but currently lacks support for several extended features. One of those features is subvolume support, which is used extensively on *SUSE distributions.

Desired Feature

In a first step Ignition should at least support mounting multiple existing subvolumes from the same device to be able to drop files on there during the files stage. This is currently failing because Ignition does not support using the same device name more than once.
Mounting a single subvolume per device is possible since #872 by using appropriate mount options.

In a second step it would be good if Ignition would also support creating additional subvolumes.

Other Information

This is related to and doing the opposite of #815, where multiple disks contain one Btrfs file system.

Example

Subvolume layout from an openSUSE installation:

ID 256 gen 78440 top level 5 path @
ID 258 gen 157397 top level 256 path @/var
ID 259 gen 157350 top level 256 path @/usr/local
ID 260 gen 157389 top level 256 path @/tmp
ID 261 gen 144350 top level 256 path @/srv
ID 262 gen 157352 top level 256 path @/root
ID 263 gen 156614 top level 256 path @/opt
ID 264 gen 157397 top level 256 path @/home
ID 265 gen 124813 top level 256 path @/boot/grub2/x86_64-efi
ID 266 gen 152932 top level 256 path @/boot/grub2/i386-pc
ID 267 gen 157367 top level 256 path @/.snapshots
ID 1038 gen 2151 top level 258 path @/var/lib/machines
ID 1448 gen 154571 top level 267 path @/.snapshots/283/snapshot

Currently it is not possible to mount more than one of these subvolumes besides the root file system.

[laenion]

I was wondering whether it would make sense to extend the path element.

To mount the /home subvolume one could use a syntax such as

 "device": "/dev/sda[/@/home]"

The brackets are escaped by udev, so there will be no conflict with existing path names (e.g. in a file system label), and this is the same syntax used by findmnt for SOURCE. This syntax would keep device as the primary key.

Creation of new subvolumes is more complex due to the fact that Btrfs supports different styles of subvolume layouts. Looking at the example from above we can see that most of the subvolumes are direct children of @ (which is not the root file system, but @/.snapshots/283/snapshot is). On the other hand e.g. @/var/lib/machines is a subvolume of @/var. Probably a distinction between relative and absolute paths (starting with a '/' or not in the brackets maybe?) would be enough, but in any case Ignition would have to make sure the parent is mounted to be able to create the subvolume...

Any thoughts?

[laenion]

Ping: Is there interest in this feature, or is extended Btrfs support out of scope?

[jdoss]

With Fedora's move to btrfs by default would this issue be in scope now?

[bgilbert]

I think this issue is in scope, but I suspect we should add dedicated config attributes for it rather than overloading existing ones.

@jdoss, it seems that Fedora change only affects desktop variants.

[cgwalters]

Exactly https://fedoramagazine.org/btrfs-coming-to-fedora-33/#comment-502620

[cmurf]

Btrfs is coming to Fedora 35 Cloud edition. I'm curious about a sort of "discoverable subvolumes spec" mimicking the discoverable partitions spec. The subvolume names could be self describing instead of using type guids. There's C API and python bindings provided by libbtrfsutil which might help with listing and creating subvolumes.

If the installation image were Btrfs, ignition could use the seed/sprout feature for replication installation: new fs uuid, preserves subvolume layout, and native compression if used; non-btrfs destinations are still possible but gain the benefit of source integrity checking with EIO on any detection corruptions.

[bgilbert]

Ignition supports compound primary keys. We could add a subvolume field to the filesystems section, and have config validation require that it be absent unless the format is btrfs. I think that's probably better than trying to put multiple data items into the device field.

Probably a distinction between relative and absolute paths (starting with a '/' or not in the brackets maybe?) would be enough, but in any case Ignition would have to make sure the parent is mounted to be able to create the subvolume...

Is the idea that a relative path would be relative to the default subvolume, and an absolute path would be relative to the top-level subvolume? Can we instead require that all paths be relative to the top-level subvolume?

In order of preference, I see a few options:

1. Only support subvolume paths relative to the top-level subvolume. Create subvolumes by mounting the top-level subvolume, then creating subvolumes in order of ascending path length.

2. Support subvolume paths relative to the top-level subvolume (which I'll call "top-relative") or to the default subvolume (which I'll call "default-relative"). At runtime, mount the top-level subvolume, look up the default subvolume, convert all default-relative subvolume paths to top-relative, fail if the same subvolume is specified multiple times, and create subvolumes in order of ascending path length.

3. Support both default-relative and top-relative subvolume paths, but don't try to disambiguate them. Do two creation passes, one for each type.

   I don't think this would work well for Ignition. To keep the config declarative, Ignition needs to ensure that the same object can't be referenced multiple times, since otherwise it would matter in what order the operations are performed. We could define an order (e.g. "absolute first, then relative") but we've generally avoided that approach so far.

Thoughts?

[bgilbert]

And I guess we'd have to reject label for subvolumes?

[laenion]

Ignition supports compound primary keys. We could add a subvolume field to the filesystems section, and have config validation require that it be absent unless the format is btrfs. I think that's probably better than trying to put multiple data items into the device field.

Agreed.

Probably a distinction between relative and absolute paths (starting with a '/' or not in the brackets maybe?) would be enough, but in any case Ignition would have to make sure the parent is mounted to be able to create the subvolume...

Is the idea that a relative path would be relative to the default subvolume, and an absolute path would be relative to the top-level subvolume?

Not necessarily the default subvolume, but whatever is used for the current root ('/') file system.

Can we instead require that all paths be relative to the top-level subvolume?

That sounds like a very reasonable approach, as everything will be reachable from the top-level subvolume (but not necessarily from the root file system).

In order of preference, I see a few options:

1. Only support subvolume paths relative to the top-level subvolume.  Create subvolumes by mounting the top-level subvolume, then creating subvolumes in order of ascending path length.

Thinking about this again this is probably also the expected solution from a user's perspective: At least when using snapshots for the root file system in most cases one probably doesn't want to have subvolumes of a snapshot.

And I guess we'd have to reject label for subvolumes?

Yes, btrfs subvolumes don't have labels themselves.

[bgilbert]

Okay, sounds good!

Not necessarily the default subvolume, but whatever is used for the current root ('/') file system.

That couldn't work anyway. Filesystem creation happens in the disks stage, before the OS mounts /sysroot, and Ignition doesn't know what filesystem will be mounted there.

[queeup]

Any news about this? I really love to have this.