Convergence with app container images

[jlebon]

Currently, bootable containers are a special thing that are for the most part built with rpm-ostree. Let's track here what it would take to make it less special.

First, we need to agree on what the end goal should be. I'm picturing something like this:

FROM quay.io/fedora/fedora-minimal:41
RUN dnf install -y @bootable
COPY unpackaged/ /
RUN bootc build commit

The @bootable group would pull in kernel, systemd, bootc, etc.... In practice, we'd probably need to enable e.g. --allowerasing to be able to swap out e.g. fedora-release-container, but maybe that can live in dnf somehow.

The custom base image flow then also converges with custom app base images. It works today to build a container image from dnf install --installroot rootfs -y ... (though I'm not sure I've seen it done yet as part of a multi-staged container-native build but I don't see why that wouldn't work). So then whether you're building a custom base (app|bootable) image just depends on whether you pull in @bootable.

Known tasks:

• Support no /ostree repo (store: Support importing images without /ostree ostreedev/ostree-rs-ext#674).
• Drain rpm-ostree postprocess logic (e.g. nss-altfiles, tmpfiles.d) into... various places. bootc and dnf mostly. Needs to be broken down into smaller tasks. (postprocess: Split up, improve docs coreos/rpm-ostree#5230, Add bootc build commit #216).
• Model rpm-ostree treefiles/fedora-bootc tiers as comps groups (I think there's a ticket somewhere for this?).

[cgwalters]

Thanks for filing this. There's a whole lot of related threads but...my take is that most of this is actually on the operating system side.

I would be totally fine having improved generic guidance here though of course.

Support no /ostree repo (ostreedev/ostree-rs-ext#674).

Landed as #887

Model rpm-ostree treefiles/fedora-bootc tiers as comps groups (I think there's a ticket somewhere for this?).

This clearly has nothing to do with bootc, right? But yes, we tried this in https://gitlab.com/fedora/bootc/base-images/-/merge_requests/91

So that just leaves:

Drain rpm-ostree postprocess logic (e.g. nss-altfiles, tmpfiles.d) into... various places. bootc

I think nss-altfiles I would leave as orthogonal to bootc. It's not required strictly speaking and people starting fresh could probably get far by combining just systemd sysusers and JSON user records.

The thing to debate I think is tmpfiles.d which I think is already tracked by #216 right?

[cgwalters]

dnf install --installroot rootfs

Tangential but: The problem is dnf doesn't set up key API filesystems on its own like dev and proc and such and that gets done by wrapper tools (such as mock) and while you can install a lot of packages without them, it's a pretty hostile environment to not have /dev/null and the /proc/self/fd links etc. I did https://gitlab.com/fedora/bootc/base-images-experimental/-/blob/main/build/dnf-installroot?ref_type=heads but it'd probably make sense to put this directly in dnf, it's not a lot of code.

[jlebon]

Support no /ostree repo (ostreedev/ostree-rs-ext#674).

Landed as #887

OK right, this is obvious in retrospect since in the custom base image flow, the final image doesn't have a repo.

Thanks for filing this. There's a whole lot of related threads but...my take is that most of this is actually on the operating system side.

I would be totally fine having improved generic guidance here though of course.

Yeah, the Fedora-specific instance of this is probably better tracked in https://gitlab.com/fedora/bootc/tracker/. I'm fine tracking that there if you'd prefer. Though I guess the high level idea of converging with app containers makes sense to track here?

[cgwalters]

Yeah, the Fedora-specific instance of this is probably better tracked in https://gitlab.com/fedora/bootc/tracker/. I'm fine tracking that there if you'd prefer.

Moved to https://gitlab.com/fedora/bootc/base-images/-/issues/56

Though I guess the high level idea of converging with app containers makes sense to track here?

It doesn't seem to me that bootc has a strong opinion on this really and it's mostly a lot of legacy in Fedora derivatives that are problems. I mean ultimately bootc doesn't really know if you derived from an "app" container or not, it just sees OCI layers. We don't have a blessed build system. So...tenatively closing because I think the other existing (and newly filed) trackers cover the finer details but obviously feel free to reopen if you disagree!

[travier]

The composefs-rs integration work should make this use case easier: #1190

[cgwalters]

The composefs-rs integration work should make this use case easier: #1190

Can you elaborate? How so?

[travier]

The composefs-rs example container images are currently built this way. It's not efficient (no layer splitting), complete (no post processing like rpm-ostree does), etc. but it's a similar idea.

[cgwalters]

It's not [...] complete (no post processing like rpm-ostree does)

Of course. That's what's under debate, it has all the bugs that the current Fedora bootc base image builder is papering over, like /var/lib/selinux -> /etc/selinux etc. I am not aware of any special logic in composefs-rs that somehow makes this better, are you? Other than simply trying not to use the existing Fedora bootc ecosystem and ignoring the bugs right?

[cgwalters]

There's just a big grab bag of things here, but one thing that is probably not immediately obvious to folks is that all of those tweaks also apply to rpm-ostree package layering and are pretty load bearing in many cases. That's the whole "unified core" thing, and there's still a lot of usage of package layering of course. So again don't get me wrong, I am all in favor of chipping away at this debt. But that's one of several reasons why the base image builds still use rpm-ostree.

[travier]

I agree, there isn't anything specific fixing those problems in composefs-rs. The main item was that it does not require the presence of an ostree repo in the container image but that has indeed also been done in bootc in #887. Also agree with the next steps in https://gitlab.com/fedora/bootc/base-images/-/issues/56.

[cgwalters]

I agree, there isn't anything specific fixing those problems in composefs-rs.

OK cool, we're in agreement. I did think it was worth drilling into this to be sure I wasn't missing something!

The main item was that it does not require the presence of an ostree repo in the container image but that has indeed also been done in bootc in #887.

Yes (for a while now) but I totally understand having missed this or not considered it because the fedora base image obviously still uses that.

I filed https://gitlab.com/fedora/bootc/base-images/-/issues/58 (with another linked issue) to track dropping it but yeah, the timestamp/reproducibility/rechunking subthread here is huge.