systemd-remount-fs.service fails on f42 disk images with composefs enabled

[pcdubs]

Describe the bug

After enabling composefs in Fedora 42, systemd-remount-fs.service fails.

sudo systemctl status systemd-remount-fs.service
× systemd-remount-fs.service - Remount Root and Kernel File Systems
     Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: failed (Result: exit-code) since Thu 2025-03-06 16:49:35 UTC; 54s ago
 Invocation: d80bd8002b7a48ecac7fa630370e9734
       Docs: man:systemd-remount-fs.service(8)
             https://systemd.io/API_FILE_SYSTEMS
    Process: 795 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE)
   Main PID: 795 (code=exited, status=1/FAILURE)
   Mem peak: 1.5M
        CPU: 34ms

Mar 06 16:49:35 localhost.localdomain systemd-remount-fs[795]: /usr/bin/mount for / exited with exit status 32.
Mar 06 16:49:35 localhost.localdomain systemd-remount-fs[799]: mount: /: cannot remount /dev/vda3 read-write, is write-protected.
Mar 06 16:49:35 localhost.localdomain systemd-remount-fs[799]:        dmesg(1) may have more information after failed mount system call.
Mar 06 16:49:35 localhost.localdomain systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Mar 06 16:49:35 localhost.localdomain systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Mar 06 16:49:35 localhost.localdomain systemd[1]: Failed to start systemd-remount-fs.service - Remount Root and Kernel File Systems.
[root@localhost ~]# cat /usr/lib/ostree/prepare-root.conf
[composefs]
enabled = yes

This only happens on the disk images, installations by anaconda work as expected:

rpm-ostree status
State: idle
Deployments:
● fedora-iot:fedora/devel/x86_64/iot
                  Version: 42.20250306.0 (2025-03-06T16:41:33Z)
                   Commit: 11051ba0b7432a97f39ad197ffef7a96361d743400598d0ef60e880b2b778ca5
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944

  fedora-iot:fedora/devel/x86_64/iot
                  Version: 42.20250303.0 (2025-03-03T15:32:30Z)
                   Commit: 4252a6b67922938096981b0f581fb55d09cd396a43e110ca8209261b32c56383
             GPGSignature: Valid signature by B0F4950458F69E1150C6C5EDC8AC4916105EF944
[root@fitlet2 ~]# systemctl status systemd-remount-fs.service
● systemd-remount-fs.service - Remount Root and Kernel File Systems
     Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; enabled-runtime; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf, 50-keep-warm.conf
     Active: active (exited) since Thu 2025-03-06 14:23:43 EST; 30min ago
 Invocation: ad5378cf211e4c688069cd266763992c
       Docs: man:systemd-remount-fs.service(8)
             https://systemd.io/API_FILE_SYSTEMS
    Process: 959 ExecStart=/usr/lib/systemd/systemd-remount-fs (code=exited, status=0/SUCCESS)
   Main PID: 959 (code=exited, status=0/SUCCESS)
   Mem peak: 1.6M
        CPU: 28ms
[root@fitlet2 ~]# cat /usr/lib/ostree/prepare-root.conf
[composefs]
enabled = yes

OpenQA:
https://openqa.fedoraproject.org/tests/3275480#step/base_services_start/9

See:
osbuild/osbuild-composer#4593

[pcdubs]

From the iso install /etc/fstab:

# Updated by bootc-fstab-edit.service
UUID=81e257f8-7d45-4720-b181-5126f4c594a7 / ext4 defaults,x-systemd.device-timeout=0,ro 1 1

Changing the disk image to 'ro' fixed the failure on reboot

[pcdubs]

See: osbuild/bootc-image-builder#349

[miabbott]

There's a systemd generator shipped by bootc that dynamically generates a unit file to fix the /etc/fstab - bootc-dev/bootc#417

It's clear that this is running successfully for systems installed via Anaconda, but it don't understand why it is not running in the raw disk case.

Running the generator command manually on a "raw disk system" doesn't seem to do what I expected, which is creating a bootc-fstab-edit.service file:

[root@localhost ~]# bootc internals systemd-generator /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late

[root@localhost ~]# ls -lR /run/systemd/generator/
/run/systemd/generator/:
total 28
-rw-r--r--. 1 root root 342 Mar  6 21:59 -.mount
-rw-r--r--. 1 root root 445 Mar  6 21:59 boot-efi.mount
-rw-r--r--. 1 root root 541 Mar  6 21:59 boot.mount
-rw-r--r--. 1 root root 370 Mar  6 21:59 dev-zram0.swap
drwxr-xr-x. 2 root root 140 Mar  6 21:59 local-fs.target.requires
drwxr-xr-x. 2 root root  80 Mar  6 21:59 local-fs.target.wants
drwxr-xr-x. 2 root root  80 Mar  6 21:59 multi-user.target.wants
drwxr-xr-x. 2 root root  80 Mar  6 21:59 sockets.target.wants
-rw-r--r--. 1 root root 279 Mar  6 21:59 sshd-unix-local.socket
lrwxrwxrwx. 1 root root  37 Mar  6 21:59 sshd-unix-local@.service -> /usr/lib/systemd/system/sshd@.service
-rw-r--r--. 1 root root 306 Mar  6 21:59 sshd-vsock.socket
lrwxrwxrwx. 1 root root  37 Mar  6 21:59 sshd-vsock@.service -> /usr/lib/systemd/system/sshd@.service
drwxr-xr-x. 2 root root  60 Mar  6 21:59 swap.target.wants
drwxr-xr-x. 2 root root  60 Mar  6 21:59 systemd-zram-setup@zram0.service.d
-rw-r--r--. 1 root root 252 Mar  6 21:59 var.mount

/run/systemd/generator/local-fs.target.requires:
total 0
lrwxrwxrwx. 1 root root 10 Mar  6 21:59 -.mount -> ../-.mount
lrwxrwxrwx. 1 root root 17 Mar  6 21:59 boot-efi.mount -> ../boot-efi.mount
lrwxrwxrwx. 1 root root 13 Mar  6 21:59 boot.mount -> ../boot.mount
lrwxrwxrwx. 1 root root 46 Mar  6 21:59 ostree-remount.service -> /usr/lib/systemd/system/ostree-remount.service
lrwxrwxrwx. 1 root root 12 Mar  6 21:59 var.mount -> ../var.mount

/run/systemd/generator/local-fs.target.wants:
total 0
lrwxrwxrwx. 1 root root 49 Mar  6 21:59 systemd-fsck-root.service -> /usr/lib/systemd/system/systemd-fsck-root.service
lrwxrwxrwx. 1 root root 50 Mar  6 21:59 systemd-remount-fs.service -> /usr/lib/systemd/system/systemd-remount-fs.service

/run/systemd/generator/multi-user.target.wants:
total 0
lrwxrwxrwx. 1 root root 52 Mar  6 21:59 ostree-boot-complete.service -> /usr/lib/systemd/system/ostree-boot-complete.service
lrwxrwxrwx. 1 root root 51 Mar  6 21:59 ostree-finalize-staged.path -> /usr/lib/systemd/system/ostree-finalize-staged.path

/run/systemd/generator/sockets.target.wants:
total 0
lrwxrwxrwx. 1 root root 25 Mar  6 21:59 sshd-unix-local.socket -> ../sshd-unix-local.socket
lrwxrwxrwx. 1 root root 20 Mar  6 21:59 sshd-vsock.socket -> ../sshd-vsock.socket

/run/systemd/generator/swap.target.wants:
total 0
lrwxrwxrwx. 1 root root 17 Mar  6 21:59 dev-zram0.swap -> ../dev-zram0.swap

/run/systemd/generator/systemd-zram-setup@zram0.service.d:
total 4
-rw-r--r--. 1 root root 107 Mar  6 21:59 bindings.conf

[root@localhost ~]# find / -name bootc-fstab-edit.service
[root@localhost ~]#

Running bootc internals fixup-etc-fstab does work as expected:

[root@localhost ~]# bootc internals fixup-etc-fstab
Updated /etc/fstab to add `ro` for `/`
[root@localhost ~]# cat /etc/fstab 
# Updated by bootc-fstab-edit.service
UUID=d1da6a8b-2d66-4085-9e64-187624485bf1 / ext4 defaults,ro 1 1
UUID=4418bc09-61a7-41d4-85d3-8161fe522647       /boot   ext4    defaults        1       2
UUID=7B77-95E7  /boot/efi       vfat    umask=0077,shortname=winnt      0       2

[cgwalters]

This is at least the 12th distinct issue on this. Please see also e.g. ostreedev/ostree#3193 (comment)

It's clear that this is running successfully for systems installed via Anaconda, but it don't understand why it is not running in the raw disk case.

Because it was explicitly designed to only run in the Anaconda case because I am sympathetic honestly to that team being way under-resourced relative to their importance and trying to change how Anaconda writes /etc/fstab has a relatively high blast radius.

There is an outstanding PR that would change this somewhat, see bootc-dev/bootc#1113

Anyone is feel free to join and help there.

But the real fix just to reiterate for the 10th time is to switch everything mounting the rootfs to use rootflags= which includes all things generating disk images. The bootc generator is just a hack.

[cgwalters]

Hmm wait also now that I look this is the composefs-with-ostree case, not actually bootc. It's just that rpm-ostree pulling in bootc caused the generator to be present. What generated this disk image?

[miabbott]

The raw disk image is generated by osbuild/Image Builder in the Fedora infra

[cgwalters]

Right. So the Atomic Desktops always use Anaconda to do installs, so don't hit this.

I am fine taking a patch to bootc to loosen this requirement: https://github.com/containers/bootc/blob/190085d57e36968cf817f6f00042535d020897b5/lib/src/generator.rs#L11 - it'd be pretty easy.

And you're hitting this then because you're not specifically using bootc-image-builder, which is the documented and recommended path for bootc derivatives.

But the generic fix here is being tracked on the image builder side down here - see the linked issues from osbuild/bootc-image-builder#756 which would fix using the osbuild-but-not-bootc-image-builder container path.

---

Also bigger picture I am personally backing away from trying to do installs via "dd disk image" flows - it has a lot of problems. Anaconda as well bootc install-to-filesystem perform dynamic installation from a running host. But that said, there are use cases and the larger "we" probably do need to support them, I just wouldn't look at them as a default path.

[cgwalters]

Just to complete the picture here too, Fedora CoreOS derivatives never shipped an /etc/fstab on installed systems. But this project isn't inheriting from CoreOS either...so here we are, a bug that nothing else is hitting.