Demonstrate a debian or arch base image

[cgwalters]

Let's do the work necessary to help push forward a Debian, Arch or other OS base image.

There are multiple components to this:

• store: Support importing images without /ostree ostreedev/ostree-rs-ext#674 will be extremely helpful
• how to workaround rpm dependency? coreos/bootupd#468 and bootupd in general and the bootloader handling is where we have some messy bits

I would actually like to just execute on coreos/bootupd#432 soon to close the iteration loop.

[LukeShortCloud]

If you need help testing anything, please let me know. I'm very interested in using bootc with my Arch Linux-based winesapOS project! I've put a lot work into it recently to support building OCI images.

[marcoh00]

I had some time on my hands this weekend and decided to have a look at building an Archlinux bootc container.
Good news is that it wasn't too hard to get a system that boots :-)
In case it helps, I'll do a writeup on what I've done and the roadblocks I faced (if only interested in roadblocks just scroll down!).

Base Image

I started with building an OSTree repository for a arch-based rootfs and used M1cha's archlinux-ostree tooling for this.

./arch-ostree build_builder_container
./arch-ostree build_rootfs_container --rootfs-containerfile examples/Containerfile.minimal --rootfs-tag localhost/archlinux:rootfs
ostree init --repo=$(pwd)/ostree
./arch-ostree --ostree-repo $(pwd)/ostree commit_rootfs_container --rootfs-tag localhost/archlinux:rootfs -- -b minimal

I did modify the Containerfile.minimal to include btrfs-progs, dosfstools, exclude grub and in a subsequent COPY/RUN installed additional packages for bootupd, bootc and some unholy package that is a compilation of various fedora boot files extracted from their rpms (more on that below).
It also copies a directory structure into /usr/lib/bootupd that I obtained from a fedora-bootc:41 image.

I could use the OSTree repo to generate a base image using bootc internals ostree-ext container encapsulate -l containers.bootc=1 -l ostree.bootable=1 --repo $(pwd)/ostree minimal containers-storage:localhost/archlinux:ostree.
For testing, I also created a derived image that just sets some simple root password so I could login after booting.

Custom packages

Maybe I find time to upload the PKGBUILDs if anyone's interested.
But they're certainly not ready for production in any way and just had the purpose to get something up and running ;-)

bootc: This is just a simple PKGBUILD looking like any other "rustapp-git" package.

bootupd: Same as bootc, but additionally creates a symlink from bootupd to bootupctl.

fedora-boot: This one cobbles together everything from Fedora's grub2-{common,pc-modules,tools-minimal,tools,efi-x64,pc} and shim-x64 RPMs.
Apart from that, you only need to make sure that the stuff that's going into /usr/sbin is moved to /usr/bin as /usr/sbin is a symlink in arch and otherwise this would conflict with filesystem.
I did not try this yet, but I think bundling everything together like that could make bootupd generate-install-metadata work in combination with a few hacks on the bootupd source code that I did (i.e. swapping the hardcoded rpm calls with calls to pacman).

Roadblocks

The following is pretty much arch-specific, however I'm sure a lot of aspects also apply to other distributions.
Also, I had pretty much zero trouble with bootc, all the hurdles I faced had something to do with either bootupd or boot configuration (I didn't look into who creates these yet, it didn't seem to be bootc, is it OSTree itself?).

• bootupd uses hardcoded RHEL/Fedora-specific parts everywhere.
  • /usr/sbin/grub2-install is /usr/bin/grub-install
  • generate-install-metadata expects everything to live in either /boot or the corresponding ostree directory when trying to query the package manager for version information. However, neither arch has packages putting files directly into boot, nor did I notice that any other distribution apart from Fedora does this.
  • Expecting to find a shim binary. I think most distributions do call it shimx64.efi so the hardcoded filename shouldn't be problematic. However in the case of Archlinux, setting up shim is (1) completely optional and (2) no signed shim packages are offered, so the wiki only explains how to use Ubuntu's shim and adding own keys via mok. As such, it would be handy if an existing shim file wasn't a strict requirement for running bootupd.
  • Currently bootupd has a strict requirement on rpm to generate metadata. When I first read this issue, I had some hope that implementing a pacman backend could be enough to make all of this work so I tried to implement that. If you're interested, I could contribute this, however it seems like there aren't any abstractions for different package management systems, so currently I just replaced the rpm code in my local branch. Unfortunately, I don't know how stable pacman's CLI output is. It certainly hasn't got RPM's capabilities that are currently used in bootupd, i.e. multiple calls are needed to get the package's build date. Also, the output seems to be locale-dependent (however setting LC_ALL did help). Moreover, I don't think all package management systems (looking at you, apt/dpkg!) have information on the build date? Maybe this could turn out to be problematic for other distributions?
• Whatever generates the bootloader configuration writes it into /boot/grub2 while Archlinux's grub searches inside /boot/grub.
• Archlinux GRUB doesn't support BLS so even if it could find the configuration it wouldn't work anyway. Maybe I should open an issue with arch and ask to add BLS support as it seems sensible to expect this.

Of course all of this is mitigated by using Fedora's boot-related packages, but that certainly is only a workaround to get it running at all.
Until now, I only tried to MBR-boot via qemu. I'm not sure if shim/grub check any signatures when not in secure boot mode, if they don't, then EFI could work, too. It wouldn't work with secure boot however (but nothing apart from self-signing and manually adding keys to shim would).

[cgwalters]

bootupd uses hardcoded RHEL/Fedora-specific parts everywhere.

Yes...sorry. There's some active discussion in coreos/bootupd#468 again about generalizing it.

Archlinux GRUB doesn't support BLS so even if it could find the configuration it wouldn't work anyway.

That's...unfortunate indeed, though it's probably worth noting that there's been some long lingering conflict and the BLS that grub implements is not the same as the BLS type 1 as documented.

Thanks for working on this! Bigger picture I was thinking about trying to make bootupd a soft dependency here. One thing that hurts is that bootloader bits are spread across both ostree and bootupd.

This all said I am sure the thing that would help the most is for us to support systemd-boot. That is a long, longstanding TODO on the ostree side but brings up its own complexities.

[frap129]

@marcoh00 How did you install the bootc container? bootc install?

I've been using ostree w/ Arch for a while now and would like to work towards migrating to bootc in the future. I've been trying to deploy an arch container from an exisiting bootc os (bluefin/ublue, deploying using bootc switch) to try avoid dealing with the bootupd issue for now. I've setup a minimal containerfile (https://github.com/frap129/arch-bootc) for working on this. bootc is able to deploy it but it fails to boot at the switch root stage. Falling back to the emergency shell also fails with "Failed to connect to system scope bus via local transport". This is in qemu setup w/ bios, not uefi, so secure boot shouldnt be an issue here.

At this point, I am wondering if I have misunderstood the purpose of your fedora-boot package. I assumed it was just to get bootupd to play nice during installation, and that using a bootc system w/ grub already installed would avoid it.

I should note that I am not encapsulating the container in an ostree commit first, since #887 is merged, which may also be problematic.

Edit: added that boot fails during switch root

Edit 2: The boot issue was because I copied the dracut config to enable the ostree hook to the wrong dir (my bad for copy+pasting and not double checking). The arch-bootc repo I linked above now houses a pretty minimal build setup for an arch-bootc base image. I have not attempted to deploy it via bootc install, but using bootc switch from an existing bootc OS works flawlessly.

[marcoh00]

@frap129 Hi, sorry for answering so late. I see you could figure out most of your issues by yourself?

Yes, I did install via bootc install, specifically the bootc install to-disk --via-loopback method. However, it needs bootupd to work - including all the trouble this brings along (which can be fixed by using a Fedora-like directory structure and packages).

[frap129]

Thanks for clarifying that. I was able to resolve all my issues, and I'm currently using bootc on one of my personal archlinux systems that was already set up using ostree.

As for the bootupd issue, the main problem with supporting archlinux seems to be that bootup expects that the grub and shim packages own their respective ".efi" files, which is only true of shim by nature of it being presigned. Given that bootc works and bootupd is really the only problem here, I'll be posting some thoughts on this in coreos/bootupd#468