tracker: logically bound app images

[cgwalters]

Logically bound images

Current documentation: https://containers.github.io/bootc/logically-bound-images.html

---

Original feature proposal text:

We should support a mechanism where some container images are "lifecycle bound" to the base bootc image.

A common advantage/disadvantage of the below is that the user must manage multiple container images for system installs - e.g. for a disconnected/offline install they must all be mirrored, not just one.

In this model, the app images would only be referenced from the base image as .image files or an equivalent.

This contrasts with physically bound images.

bootc logically bound flow

bootc upgrade follows a flow like:

• fetch new base image
• Read its root filesystem, discover logically bound images
• Pull those images into a new bootc-owned container storage (xref Introduce bootc-owned container store, use for bound images #724 )
• Garbage collect images which are not referenced by any root (e.g. pending/current/rollback).

Current design: symlink to .image or .container files

Introduce /usr/lib/bootc/bound-images.d that is symlinks to .image files or .container files.

Pros:

• Straightforward to implement
• The admin only needs to bump a :sha256 digest in one place to update

Cons:

• Handling systemd specifiers is tricky, we will error out on them
• No separation of concerns: an .image file is intended to pull images not be parsed by an external tool for a separate purpose.
• Updates to Quadlet may break the process and/or add (potential) continuous maintenance burden for bootc (i.e., "chasing/reimplementing new features").
• Forces users to use Quadlet even if they have no use for pulling images under systemd.

Note: we expect the .image files to reference images by digest or immutable tag. There is no mechanism to pull images out of band.

Other alternatives considered

New custom config file

A new TOML/usr/lib/bootc/bound-images.d, of the form e.g. 01-myimages.toml:

images = ["quay.io/testos/someimage@sha256:12345", "quay.io/testos/otherimage@sha256:54321"]
authfile = "/etc/containers/my-custom-auth.json"

Pros:

• Easy to just list multiple images vs one image per .image file
• TOML format is used by other bootc tooling and some of the container config formats

Cons:

• New file format relating to container images
• May need in the general case to support many of the existing options in .image files
• The admin will need to bump a :sha256 digest in two places to update in general (both in a .container or .image and the custom .toml here)

Parse existing .image files

Pros:

• Well known
• Spec -> pull translation exists
• Existing spec handles most image pull fields

Cons:

• Would need to extend the spec to include a new bootc=bound or equivalent opt-in
• Handling specifiers is tricky
• Implentation complicated wrt managing systemd

What would happen under the covers here is that bootc would hook into podman and:

• disallow GC of these images even if the unit isn't running (for all deployments)
• Fetch new images (from the new base container image) on bootc upgrade

---

TODO:

• docs
• CI test
• PR to fedora bootc examples
• Ensure compatibility with bootc-image-builder bound images fail in bootc-image-builder #715
• install path with bootc install to-filesystem - simple scenario w/out pull secret?
• install path w/pull secret embedded in bootc image? podman pull happens from bootc container
• install path w/bootc-image-builder where it pre-pulls images, demonstrated e2e w/konflux, we probably need to enable a model where bound images are provided in a mirror location or OCI directory

[alexlarsson]

In the automotive world we often think of containers as two possible things. Either they come with the system, and are updated atomically with it, or they are separately installed. They way we expect this to work is for the system ones to be installed in a separate image store that is part of the ostree image. And then the "regular" containers will just be stored in /var/lib/container.

The automotive sig manifests ship a storage.conf that has:

[storage.options]
additionalimagestores = [   /usr/share/containers/storage ]

Then we install containers in the image with osbuild like:

      - type: org.osbuild.skopeo
        inputs:
          images:
            type: org.osbuild.containers
            origin: org.osbuild.source
            mpp-resolve-images:
              images:
                - source: registry.gitlab.com/centos/automotive/sample-images/demo/auto-apps
                  tag: latest
                  name: localhost/auto-apps
        options:
          destination:
            type: containers-storage
            storage-path: /usr/share/containers/storage

[alexlarsson]

This was part of the driver for the need for composefs to be able to contain overlayfs base dirs (overlay nesting). Although that is less important if container/storage also uses composefs.

[rhatdan]

I love the idea of additonal stores for this.

[vrothberg]

Quadlet supports .image files now which can be directly referenced in .container files. Maybe that's a way to achieve a similar effect.

The .image files don't yet (easily) allow for pulling into an additional store, but this could be a useful feature.

Cc: @ygalblum

[cgwalters]

Then we install containers in the image with osbuild like:

So IMO this issue is exactly about having bootc install and bootc update handle these images. Because as is today, needing to duplicate the app images in an osbuild manifest is...unfortunate. With this proposal, when osbuild is making a disk image, it'd use bootc install internally to the pipeline, and we wouldn't need to re-specify the child container images out of band of the "source of truth" of the parent image.

[alexlarsson]

Then we install containers in the image with osbuild like:

So IMO this issue is exactly about having bootc install and bootc update handle these images. Because as is today, needing to duplicate the app images in an osbuild manifest is...unfortunate. With this proposal, when osbuild is making a disk image, it'd use bootc install internally to the pipeline, and we wouldn't need to re-specify the child container images out of band of the "source of truth" of the parent image.

I understand that, and I merely pointed out how we currently do it in automotive, not how it would be done with bootc.

Instead, what I propose is essentially:

Dockerfile:

FROM bootc-base
RUN podman --root /usr/lib/containers/my-app pull quay.io/my/app
ADD my-app.container /etc/containers/systemd

my-app.container:

[Container]
Image=quay.io/my/app
PodManArgs=--storage-opt=overlay.additionalimagestore=/usr/lib/containers/my-app

And then you have an osbuild manifest that just deploys the above image like any normal image.

Of course, instead of open-coding the commands like this, a tool could do the right thing automatically.

You might also want the tool to tweak the image name in the quadlet to contain the actual digest so we know that the exact right image version is used every time.

[alexlarsson]

Its also interesting to reflect on the composefs efficiency in a setup like this.

If we use composefs for the final ostree image, we will get perfect content sharing, even if each of the individual additional-image-stores use its own composefs objects dir. Even if no effort is made to try to share object files between image store directories. Because all the files will eventually be deduplicated as part of the full ostree composefs image.

In fact, we will even deduplicate files between image stores that use the traditional overlayfs or vfs container store formats.

[alexlarsson]

In fact, maybe using vfs backend is the right approach here? It is a highly stable on-disk format, and its going to be very efficient to start such a container. And we can ignore all the storage inefficiencies, because they are taken care off by the outer composefs image.

[ygalblum]

my-app.container:

[Container]
Image=quay.io/my/app
PodManArgs=--storage-opt=overlay.additionalimagestore=/usr/lib/containers/my-app

Just wanted to note that --storage-opt is a global argument. So, the key to use is GlobalArgs instead of PodmanArgs.

[alexlarsson]

I wonder if we should tweak the base images to have a standardized /usr location for additional image store images.