install-to-filesystem does not work when targeting LVM

[miabbott]

Host system is Fedora 39 Server.

Using the tier1 Fedora ELN image, I created a custom variant that had bootc installed from Copr:

FROM registry.gitlab.com/centos/cloud/sagano/fedora-boot-tier-1:eln
RUN curl -L -o /etc/yum.repos.d/bootc.repo https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/repo/fedora-rawhide/rhcontainerbot-bootc-fedora-rawhide.repo
RUN sed -i 's|\$releasever|rawhide|' /etc/yum.repos.d/bootc.repo
RUN rpm-ostree install bootc
RUN ostree container commit

Built it successfully with sudo podman build -t sagano-bootc-copr:latest -f Containerfile.sagano .

Tried install-to-filesystem:

$ image=localhost/sagano-bootc-copr:latest
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs
ERROR No filesystem uuid found in target root

Hmm...looks like it might be SELinux as the culprit:

$ sudo ausearch -m avc
----
time->Fri Nov  3 12:03:42 2023
type=AVC msg=audit(1699027422.008:673): avc:  denied  { nosuid_transition } for  pid=3425 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c54,c375 tcontext=system_u:system_r:install_t:s0:c54,c375 tclass=process2 permissive=0

[miabbott]

Similar failure using CentOS Stream 9 as the host.

$ cat Containerfile.sagano 
FROM registry.gitlab.com/centos/cloud/sagano/centos-boot-tier-1:stream9
# Not sur why I can't just use the copr repo, but it doesn't show as an
# enabled repo when I copy it to /etc/yum.repos.d ¯\_(ツ)_/¯
RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit

$ image=localhost/sagano-cs9-bootc-copr:latest                                                                                               
$ sudo podman run --rm --privileged --pid=host -v /:/target --security-opt label=type:unconfined_t "${image}" bootc install-to-filesystem --target-no-signature-verification --karg=console=ttyS0,115200n8 --replace=alongside /target
Mounting selinuxfs                                                
ERROR Inspecting filesystem /target: findmnt /target failed: ExitStatus(unix_wait_status(256))        
$ sudo ausearch -m avc
----
time->Fri Nov  3 16:07:22 2023
type=PROCTITLE msg=audit(1699042042.505:752): proctitle=2F746D702F2E746D70376174694E4800696E7374616C6C2D746F2D66696C6573797374656D002D2D7461726765742D6E6F2D7369676E61747572652D766572696669636174696F6E002D2D6B6172673D636F6E736F6C653D74747953302C3131353230306E38002D2D7265706C6163653D616C6F6E6773696465002F74617267
type=PATH msg=audit(1699042042.505:752): item=0 name="/lib64/ld-linux-x86-64.so.2" inode=34460089 dev=00:32 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:container_file_t:s0:c264,c980 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1699042042.505:752): cwd="/"
type=EXECVE msg=audit(1699042042.505:752): argc=6 a0="/tmp/.tmp7atiNH" a1="install-to-filesystem" a2="--target-no-signature-verification" a3="--karg=console=ttyS0,115200n8" a4="--replace=alongside" a5="/target"
type=SYSCALL msg=audit(1699042042.505:752): arch=c000003e syscall=59 success=yes exit=0 a0=55bdd5e15820 a1=55bdd5ddfca0 a2=55bdd5e16ca0 a3=8 items=1 ppid=35181 pid=35183 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4 comm=".tmp7atiNH" exe="/tmp/.tmp7atiNH" subj=system_u:system_r:unconfined_t:s0:c264,c980 key=(null)
type=SELINUX_ERR msg=audit(1699042042.505:752): op=security_bounded_transition seresult=denied oldcontext=system_u:system_r:unconfined_t:s0:c264,c980 newcontext=system_u:system_r:install_t:s0:c264,c980
type=AVC msg=audit(1699042042.505:752): avc:  denied  { nosuid_transition } for  pid=35183 comm="bootc" scontext=system_u:system_r:unconfined_t:s0:c264,c980 tcontext=system_u:system_r:install_t:s0:c264,c980 tclass=process2 permissive=0

[cgwalters]

RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm
RUN rpm-ostree install bootc.rpm
RUN ostree container commit

(this is unrelated to the bug, but what you really want is more like:

RUN curl -L -o bootc.rpm https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/centos-stream-9-x86_64/06595650-bootc/bootc-202311031759.g634ce23bce-1.el9.x86_64.rpm && \
  rpm-ostree install bootc.rpm && rm bootc.rpm && ostree container commit

This will avoid leaking the RPM (as a file) into the container image and ensure that unnecessary content is properly squashed.

[cgwalters]

Similar failure using CentOS Stream 9.

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy. (When you first said "Using c9s" I thought you meant for the host, not the container)

[cgwalters]

Hmm...looks like it might be SELinux as the culprit:

Right. We've been trying really really really hard to make this work. We have #83 which is related.

For #78 I finally just gave up and in that mode did a setenforce 0. (Background: for reasons similar to this, Anaconda has always run in setenforce 0 mode)

Can you check that setenforce 0 beforehand works? Bear in mind that that this command is effectively replacing the OS, and setenforce 0 is a transient operation for the soon-to-go-away OS.

It's really tempting to do that unconditionally but I'm trying to Do It Right as much as possible. I'm not yet sure why it works in some paths but not others.

cc @rhatdan

[cgwalters]

Oh actually this has nothing to do with selinux...we're already detecting this situation and running setenforce 0 internally already.

The problem is we don't support LVM. One immediate issue seems to be that when executed from a container with a bind mount to the target root, the findmnt tooling gets confused and doesn't manage to introspect the UUID in this case.

I think what we need to do here is two things to fix the baseline error:

• Add install-to-host-filesystem-alongside (as an effective alias for install-to-filesystem --replace=alongside /target) which would have improved ergonomics
• Enter the host mountns when gathering data like from findmnt - here we know to do that instead of writing to an arbitrary filesystem as we have with /target (although I guess we could detect the case where it's the host root)

But that's just preparatory work.

I think we'll also need to pick up kargs like rd.lv* etc., and properly honor root=/dev/mapper/root.

[rhatdan]

Can't we run the tool that is attempting to install as install_t or unconfined_t? Is this bootc? We could just transition to an unconfined role. Looks like there is a type

install_exec_t

Which transition to install_t, which is allowed to do anything including put labels that don't exist yet, I believe.

[miabbott]

The failure is totally unrelated to the bootc-in-container (or the OS version of the container), it's about the host SELinux policy. (When you first said "Using c9s" I thought you meant for the host, not the container)

I amended my comment to indicate I was using a c9s host, even if it is still an LVM issue.

[henrywang]

@cgwalters Hi! I run into the same issue on ARM VM in beaker. ARM VMs in beaker are deployed as LVM when install CentOS Stream 9.

[root@ampere-mtsnow-altramax-02-vm-15 /]# df -h
Filesystem                                                Size  Used Avail Use% Mounted on
devtmpfs                                                  4.0M     0  4.0M   0% /dev
tmpfs                                                      17G  100K   17G   1% /dev/shm
tmpfs                                                     6.8G  8.8M  6.8G   1% /run
/dev/mapper/cs_ampere--mtsnow--altramax--02--vm--15-root   37G  3.7G   33G  11% /
/dev/vda2                                                 960M   39M  922M   5% /boot
/dev/vda1                                                 599M  4.0K  599M   1% /boot/efi
/dev/mapper/cs_ampere--mtsnow--altramax--02--vm--15-home   18G  159M   18G   1% /home
tmpfs                                                     3.4G     0  3.4G   0% /run/user/0

[shi2wei3]

@cgwalters Do you think we need to escalate this issue a little bit? Azure RHEL on-demand images are using LVM. And I just tried to do bootc install on a RHEL9.3 public image on Azure, which lead to exactly the same problem here.

[cgwalters]

Ah, good find. So...I don't think we need to escalate this right now, let's just document the places that it works and the limitation on LVM.

The real fix for the "in-place" will be #78 which will default to rewriting the filesystem to not be LVM (i.e. it will default to what the container image specifies).

(We still do want to support configuring LVM in this scenario of course, but that gets into its own trickiness)