Symlink from /etc/<dir> to /usr/etc/<dir> breaks install to-filesystem

[rainest]

I've installed an RPM that includes a symlink from a path under /etc to a path under /usr/etc. In the container image:

# ls -l /etc/ | grep podman
lrwxrwxrwx 1 root root       21 Nov 12  2023 podman_hpc -> ../usr/etc/podman_hpc

# bootc --version
bootc 1.1.2

Attempting to run bootc install to-filesystem with this image (Containerfile) fails. Using https://github.com/osbuild/bootc-image-builder, the error is:

# podman run --rm -it --privileged --pull=newer --security-opt label=type:unconfined_t -v ./output:/output -v /var/lib/containers/storage:/var/lib/containers/storage quay.io/centos-bootc/bootc-image-builder:latest --local --rootfs ext4 localhost/broken:4
...
ERROR Installing to filesystem: Creating ostree deployment: Pulling: Importing: Parsing layer blob sha256:0c79447a149538e009ebe92da56b5fb54c07de502c82f69683cf8ac5136e6ac9: **
OSTree:ERROR:src/ostree/ot-main.c:324:ostree_run: assertion failed: (success || error): Processing tar: Failed to commit tar: ExitStatus(unix_wait_status(134))
Traceback (most recent call last):
...
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]4ce67502d3e958860e4c9541e4cd375da6a9833d43df3d6389f8eaf33a22dca4', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'localhost/broken:4', '/run/osbuild/mounts']' returned non-zero exit status 1.

log.txt is the complete log.

This comes from ostree-ext/src/container/store.rs and ostree-ext/src/tar/write.rs AFAICT.

Building from an image without the link (Containerfile) succeeds.

This looks a bit similar to #637 and #856, but with a symlink rather than a hard link.

[cgwalters]

I've installed an RPM that includes a symlink from a path under /etc to a path under /usr/etc.

It's not about the symlink, it's about having both /etc and /usr/etc at all, it's not supported....
OK well, surprisingly right now it does work but this was not intentional and I'm inclined to make it a hard error.

What are you putting in /usr/etc?

(The semantics right now apparently are that we end up effectively silently "merging" the two but again, not something I want to try to support)

OSTree:ERROR:src/ostree/ot-main.c:324:ostree_run: assertion failed: (success || error)

Hmm fun, that's a different bug where we apparently aren't setting an error correctly.

[cgwalters]

Completely unrelated to this bug, but fwiw rpm directly supports fetching URLs so the reproducer boils down to

FROM quay.io/fedora/fedora-bootc:42
RUN rpm -ivh --nodeps https://github.com/NERSC/podman-hpc/releases/download/v1.1.0/podman-hpc-1.1.0-1.x86_64.rpm

(And if you do want to fetch URLs, curl is part of the base image, I don't know why people turn to wget; but if you do do it that way you should also remember to rm the RPM afterwards so it's not pointlessly taking up space)

[cgwalters]

And sorry so I now understand I (or someone) needs to go talk to the https://github.com/NERSC/podman-hpc folks about this, it's not something you specifically are doing.

[antheas]

Universal blue also used to use both /usr/etc and /etc for two reasons: (1) using /usr/etc when making a package allows overriding the /etc files of another package due to how rpm-ostree works, (2) confusion and user error

We found that out when building rechunk and quirked it

I think its done enough where it might be warranted to quirk it and throw a warning instead.

[cgwalters]

(1) using /usr/etc when making a package allows overriding the /etc files of another package due to how rpm-ostree works

It was really never intended for that. Do you have references to where this hack was used so we can track them? When doing container layering in particular it's totally fine to overwrite files owned by packages, especially for /etc it's just expected. If this is correct then I think the fix here is to just stop using RPM for those files.

[antheas]

Here are some of the PRs that fixed that hack when we switched to rechunk
ublue-os/ucore#183
ublue-os/bluefin#1580
ublue-os/config#310

Currently, the only place it is used is the policy.json file due to (1)
ublue-os/config#311

[cgwalters]

@rainest FYI, https://github.com/NERSC/podman-hpc/releases/tag/v1.1.1 includes a fix on their side for this. Looks like that got rolled up into the current latest in https://github.com/NERSC/podman-hpc/releases/tag/v1.1.3