using bootc install-to-filesystem

[cgwalters]

This relates to #4

• We had some general agreement to support bootc install-to-filesystem; this will help long term with things like tracker: logically bound app images containers/bootc#128
• bootc install-to-filesystem should also grow support for being provided the base container image externally (e.g. cached in osbuild); we know this is needed for offline ISO installs too. This ties with the above for the lifecycle bound app/infra containers
• We can't drop the osbuild/ostree stages because not every case will use bootc in the near future
• Agreement that for the ISO/installer case any customization (embedded kickstarts, but also which installer) would likely live external to the container (blueprint or equivalent)

[cgwalters]

bootc install-to-filesystem should also grow support for being provided the base container image externally

Digging in, this is messier than I thought. Still possible, but @ondrejbudai can you state more precisely the concern you had with having bootc install from the running container?

ISTM that in general going forward we'll want to support running images cached in the infrastructure, which will drive us towards using containers-storage most likely, as opposed to e.g. the dir transport. And if we do that, ISTM it's just simpler to keep bootc doing exactly what it's doing today in fetching from the underlying store as opposed to having something else push content in, right?

[achilleas-k]

Just to clarify, because there are two ideas here that sound very similar but are probably unrelated:

• The issue you're talking about is with the idea of having bootc run from outside the base container image when running install-to-filesystem. So the idea of having it do bootc install-to-filesystem <container ref> <filesystem path> (or bootc install-to-filesystem oci-archive:/path/to/container.tar /path/to/tree for example), from a host machine would require too much work.
  • This is as opposed to running podman run -v/path/to/tree:/target <container ref> bootc install-to-filesystem /target, from the host, which is how it currently works.
• This issue is not related to running bootc from a container that is in an "offline" storage format like an archive, right? So we can still do podman run -v/path/to/tree:/target oci-archive:/path/to/container.tar bootc install-to-filesystem /target? Which will probably work fine in osbuild. My concern, as we discussed yesterday, is that we're putting a few too many layers of containers/namespaces here which make it hard to predict some details, but might be okay. I think it's time we actually tried this and see what we get.

If I'm understanding everything correctly (and if I'm remembering everything from yesterday's conversation), @ondrejbudai's idea to mount the container and run it in bwrap is the alternative to this, but like you said, bootc wont like that as it makes some container-specific assumptions.

[ondrejbudai]

I would actually combine #1 with mounting the container.

1. Mount the container and chroot into it (in osbuild terms, construct a buildroot by "exploding" the container, and use this a build pipeline for the following steps)
2. Partition a disk file using tools from inside the container
3. Mount the disk file to /target
4. Somehow get the container image in the oci format to e.g. /source/container.tar
5. Run bootc install-to-filesystem --source oci-archive:/source/container.tar --target /target

Note that I do have a slight preference for passing a whole container storage instead of an oci archive.

[cgwalters]

Just to level set, this today is sufficient to generate a disk image:

$ truncate -s 20G /var/tmp/foo.disk
$ losetup -P -f /var/tmp/foo.disk
$ podman run --rm --privileged --pid=host --security-opt label=type:unconfined_t quay.io/centos-bootc/fedora-bootc:eln bootc install --target-no-signature-verification /dev/loop0
$ losetup -d /dev/loop0

[cgwalters]

Backing up to a higher level, I think there are basically two important cases:

• Generating a disk image from a container image stored in containers-storage: notably this is the most obvious flow in podman-desktop on Mac/Windows. Copying that into a dir or oci-archive is just an unnecessary performance hit.
• Generating a disk image from a container in a remote registry: this will happen in many production build flows. It seems simplest then if we try to unify this with the first case by always pulling into containers-storage, right?

[cgwalters]

Also bootc-dev/bootc#215 can't work until bootc-image-builder starts using bootc.

[achilleas-k]

Backing up to a higher level, I think there are basically two important cases:

* Generating a disk image from a container image stored in `containers-storage`: notably this is the most obvious flow in podman-desktop on Mac/Windows.  Copying that into a `dir` or `oci-archive` is just an unnecessary performance hit.

Which phase of the build is this referring to? If it's about having the stage in osbuild use the host containers-storage directly, I think the performance hit isn't entirely unnecessary but gives us the caching and reproducibility guarantees that we get with osbuild. These aren't directly relevant to the current use case (running it all in an ephemeral container), but I'm also thinking about the whole disk image built use a case more generally (using the same code and flow in the service).
Or is this just about having the osbuild containers cache be itself a containers-storage? That's definitely an idea I'd like to explore.
If we're talking about having a convenient way of using the host's containers-storage in the bootc-image-builder container, I think that's a lot simpler.

* Generating a disk image from a container in a remote registry: this will happen in many production build flows.  It seems simplest then if we try to unify this with the first case by always pulling into `containers-storage`, right?

Generalising any solution to both cases would be preferable, I agree.

[achilleas-k]

the caching and reproducibility guarantees that we get with osbuild

Thinking about this a bit more, I realise my hesitation is mostly around modifying the caching model substantially but now I'm thinking there's a good way to do this with a new, different kind of source. A containers-storage source could use the host container storage as its backend directly and pass it through to the stage.

The one "unusual" side effect would be that osbuild would then have to pull a container into the host machine's containers-storage, which I guess is fine (?). But what happens if osbuild, running as root, needs to access the user's storage? What if it writes to it?

[cgwalters]

But what happens if osbuild, running as root, needs to access the user's storage? What if it writes to it?

One thing that can occur here is that a user might be doing their container builds with rootless podman; so when they want to go make a disk image from it we'd need to copy it to the root storage. Things would seem to get messy to have a root process with even read access to a user storage because there's locking involved at least.

[achilleas-k]

so when they want to go make a disk image from it we'd need to copy it to the root storage

I think this makes sense. I'd want to make it explicit somehow that osbuild is doing this. It's one thing to write stuff to a system's cache when building images with osbuild (or any of IB-related projects), it's another thing to discover that your root container store now has a dozen images in it from a tool that some might think of as unrelated to "container stuff".

[achilleas-k]

Pinging @kingsleyzissou here since he's working on this.

[cgwalters]

Which phase of the build is this referring to? If it's about having the stage in osbuild use the host containers-storage directly, I think the performance hit isn't entirely unnecessary but gives us the caching and reproducibility guarantees that we get with osbuild.

I'm not quite parsing this (maybe we should do another realtime sync?) - are you saying using containers-storage is OK or not?

Backing up to a higher level, I think everyone understands this but I do want to state clearly the high level tension here because we're coming from a place where osbuild/IB was "The Build System" to one where it's a component of a larger system and where containers are a major source of input.

I understand the reasons why osbuild does the things it does, but at the same time if those things are a serious impediment to us operating on and executing containers (as intended via podman) then I think it's worth reconsidering the architecture.

These aren't directly relevant to the current use case (running it all in an ephemeral container), but I'm also thinking about the whole disk image built use a case more generally (using the same code and flow in the service).

It's not totally clear to me that in a service flow there'd be significant advantage to doing something different here; I'd expect as far as "cache" fetching images from the remote registry each time wouldn't be seriously problematic. For any cases where it matters one can use a "pull-through registry cache" model.

Or is this just about having the osbuild containers cache be itself a containers-storage? That's definitely an idea I'd like to explore.

That seems related but I wouldn't try to scope that in as a requirement here. Tangentially related I happened to come across https://earthly.dev/ recently which deeply leans into that idea. At first I was like the "Makefile and Dockerfile had a baby" was kind of "eek" but OTOH digging in more I get it.

[LorbusChris]

Backing up to a higher level, I think there are basically two important cases:

* Generating a disk image from a container image stored in `containers-storage`: notably this is the most obvious flow in podman-desktop on Mac/Windows.  Copying that into a `dir` or `oci-archive` is just an unnecessary performance hit.

* Generating a disk image from a container in a remote registry: this will happen in many production build flows.  It seems simplest then if we try to unify this with the first case by always pulling into `containers-storage`, right?

Coming from the OpenShift/OKD side, I think ideally the tool for ostree container to disk image conversion can be run independently of osbuild, i.e. it can also be wrapped by other pipeline frameworks such as prow, tekton, argo workflows, and even jenkins for any kind of CI/CD or production build.

Agreeing on keeping the container images in containers-storage everywhere seems fine to me.