Various feedback

[cgwalters]

This project fills a real need. I am not totally proud of the "face" that we present with https://gitlab.com/fedora/bootc/examples.

At the same time though it's intentional: that Dockerfile-oriented examples is to emphasize that familiarity for users exposed to that ecosystem.

Requirements:

• bootc (the project) will happily accept and boot any OCI images that conform to the basic layout requirements (of which there are not much). Dockerfile is not the only build system to build OCI images.

Strongly held opinions

Execute via RUN first

I think we will get the most value out of supporting declarative code executed inside RUN invocations in Dockerfile instead of trying to generate them. There's a huge amount of value to this approach because it Just Works with the whole ecosystem that can consume Dockerfile (including podman build on e.g. MacOS which has all this support for remotes, multi-arch etc.).

Wrapping podman build or equivalent creates a whole host of questions for how these features get exposed and used, and also opens questions around how versioning for the build instructions (modules, etc.) works.

Avoid inventing new formats unless necessary

The world is full of wrappers, especially for example custom (mostly YAML) wrappers for lists of RPM/deb packages. But we also have blueprint and kickstart and rpm-ostree treefiles and Kiwi XML and comps and...the list goes on and on.

I would prefer to pick up something of the shape of https://gitlab.com/fedora/bootc/osbuild-cfg which is strongly oriented towards consuming one existing format (blueprints) but I think it would make sense to learn kickstart too.

We have customers that are right now porting from Kickstart to Dockerfile, and it would really help the story if they could literally just move bits of kickstart instead of rewriting it.

Actually also another big one in this space is butane.

Drive declarativity as close to the source as possible

Really dnf should learn a declarative format instead of having endless variations created. libpkgmanifest might become that (though that one is designed to interact with lockfiles)

But for example on a different topic of kernel arguments, we handle that directly in bootc in a declarative, drop-in way.

Taking the ssh module as an example...I think it would ultimately make sense to have something more authentication-oriented shipped in the OS own something like this. It's of course really hard to pick what that is.

(Also osbuild-cfg has a different but related version of this ssh key handling)

Moving forward

I totally agree with the goals of fab (and thanks for starting it!), I just hope we can agree on some of these principles and where to go from here.

(On osbuild-cfg, it was just a prototype spike, if someone wants to do something like it but doesn't know Rust and wants to use Python or something, fine by me...)

[cgwalters]

Edit, forgot one more strongly held opinion:

Multi stage builds are good

In order to generate smaller systems, we really want to support separating the build time from runtime. With bootc it's now possible to create and maintain a RHEL system that doesn't have dnf installed on the client side at all - or other even larger things.

Ideally key modules that we create here are supported in this way. And we do know how to do that with things like ssh (the osbuild-cfg blueprint module could write to a target root or generate a rootfs fragment).

Of course implementing this in general requires each "module" or config knob to support it.

[kwozyman]

Thank you very much for taking the time to reply in such a detailed and comprehensive fashion, this means a lot!

Execute via RUN

First, I totally agree that having wrappers upon wrappers for config (and other things) is somewhat cumbersome and can become tedious and frankly that's why I don't particularly enjoy using blueprints. Whenever I'm forced to write a blueprint, I have to either lookup the syntax or reuse one of the blueprints I've written in the past, which is no way different than reusing a Containerfile snippet.

You mention using a RUN statement. How would that look like? Something like RUN fab module ssh-config perhaps? An immediate issue I see with that is at this point fab (or something akin to it; for simplicity, I will refer to fab moving forward, but that includes other similar tooling as well) needs to be embedded in the base image. An easy way out of this would be to include this configuration tool in bootc proper as that would always be included in all the base images. Another easy out would be to copy fab and it's dependencies to the base image at build time and maybe clean after. Inclusion in bootc proper would probably bloat it a bit too much for our liking? The copy/remove of a configuration tool seems like unnecessary busywork. Unless bootc would pull fab and it's deps under the hood and handle the setup and cleanup? That could decouple bootc proper from fab and still keep it clean.

Avoid inventing new formats unless necessary

Regarding the use of blueprints, I see one big disadvantage -- lack of customization. Let's say we brush aside the need to know blueprint format, it still only fixes the issue of generic OS customization, but not the core issue of allowing users to mix and match their own "modules". In the end, OS customization is just somewhat cumbersome, but would not impede the users too much. However, if they would want to offer various images with or without their own "modules" (features), the issue of a complex CI/CD the users would need to built would still be very much present.

Actually, the more I think about this, the more I get to the conclusion that there is room for both something like osbuild-cfg that would apply a blueprint at build time, but also for something like fab that would allow composition of multiple standardized Containerfiles. Perhaps a special "blueprint" module could fill this role.

Drive declarativity as close to the source as possible

No actual comments on this one, except I agree. However, there will always be "modules" that need their own engineering, especially if we think outside of just OS configuration and lean into the scenarios of various users trying to ship a host of images with various features enabled or disabled.

Moving forward

I'm happy to start implementing things once we decide on how things should look. Regarding the programming language, I've used Python for prototyping and there is also a go version in a branch, at the suggestion of Sam from flightctl. Personally, I'm quite familiar with Py but I would not put aside learning Rust if there is a need to.

[cgwalters]

An immediate issue I see with that is at this point fab (or something akin to it; for simplicity, I will refer to fab moving forward, but that includes other similar tooling as well) needs to be embedded in the base image.

Strictly speaking, no - it could be installed, then used - and also removed thereafter. Obviously though the level of "blessing" goes up significantly for tooling in the default fedora base image (especially if it's also in examples). But I also am not sure I ever see there being one blessed tool even in the Fedora-bootc derivative ecosystem.

An easy way out of this would be to include this configuration tool in bootc proper as that would always be included in all the base images.

That clashes directly with the first (and only) Requirement listed above right? I don't think it's necessary for us to improve the situation.

something like osbuild-cfg that would apply a blueprint at build time,

Note that while blueprint today doesn't have includes, kickstart does. But even without that it doesn't have to be "a" blueprint, it can also be:

COPY blueprints/ /run/blueprints/
RUN osbuild-cfg /run/blueprints/*

And that alone adds a decent first level of composition. (But yes, not as much I would say as having a tool directly process a hierarchy)

the core issue of allowing users to mix and match their own "modules"

This is a pivotal question for sure. Kickstart and blueprint were not quite designed with that level of extensibility in mind, they're more things provided by the OS vendor.

I am not at all sure that we need to have the same declarative language for "base operating system stuff" versus what e.g. "mid level" app teams might want/need.

That said again, I think there isn't a strict need to define a "module" for any change a user could want to make to the operating system or app!

Again we have RUN update-crypto-policies --set=future and COPY someconfig.conf /usr/lib/systemd/system/foo.conf.d/frob-foo.conf as fundamental baselines that compose just fine overall.
Would we add a "wrapper" module for update-crypto-policies? To me the answer is "probably not by default".

To state this another way, sometimes I think the outcome here is improving underlying tools and config files.

[kwozyman]

Strictly speaking, no - it could be installed, then used - and also removed thereafter. Obviously though the level of "blessing" goes up significantly for tooling in the default fedora base image (especially if it's also in examples). But I also am not sure I ever see there being one blessed tool even in the Fedora-bootc derivative ecosystem.

You're right and I was more "thinking out loud" then proposing a strict path with this.

That clashes directly with the first (and only) Requirement listed above right? I don't think it's necessary for us to improve the situation.

I don't think that it does? Or maybe I did not understand correctly. In the end, including a configuration tool in bootc does not change the underlying structure of the container image, nor does it prevent the current configuration method (i.e. Containerfile snippets) to work. But much like the point above, it was more forward thinking on my part, in the end I am also a large supporter of the Unix philosophy of having distinct tools that can be chained together.

And that alone adds a decent first level of composition. (But yes, not as much I would say as having a tool directly process a hierarchy)

I believe for a "first try" this level hierarchy is good enough, let's not over engineer. In my opinion, already having these different configurations being short one-liners in the Containerfile is a very good improvement.

To state this another way, sometimes I think the outcome here is improving underlying tools and config files.

This is a completely reasonable point and I completely agree.

I am not at all sure that we need to have the same declarative language for "base operating system stuff" versus what e.g. "mid level" app teams might want/need.

I believe this one circles back to one of the initial problems I am trying to solve with fab -- maintaining multiple different flavors of bootc images with multiple optional modules. Or simply put, the complexity that comes out of a lack of "INCLUDE" in the Containerfile syntax.

This particular problem can be solved in multiple ways and is strictly speaking not insurmountable (a somewhat complex CI/CD pipeline comes to mind, simply copy/pasting the optional pieces of a Containerfile etc), but I feel like already providing some tool to ease this would help a lot. If we think even of classic operating systems, there is always some mechanism to provide this to various degrees -- for example, package managers almost always have groups of packages that can be defined and the user can choose to install any combination of those and those groups are defined somewhere above in the hierarchy, at the repository level. It would be somewhat cumbersome to have to define those groups for every installation (build in the bootc case). This is just an example, but a part of a Containerfile that would provide this type of optional feature can be a combination of packages, static files and all that can brought together in a Containerfile.

To summarize, it seems there are three cases we're talking about here:

1. generic OS configuration -- this would be provided by a blueprint and processed by fab a build time.
   a. a variant of this could be support for Kickstart files as well
   b. is there a need to support both blueprints and kickstarts?
   c. if we open the Kickstart discussion, what do we do about disk configuration (partitioning)?

2. improving and adapting underlying OS tools -- this is outside of the scope of this tool and is simply an iterative process in other projects

3. flavor-specific or non-generic "modules" that ease composition of multiple choices images -- this would somehow fix lack of an include verb in Containerfile syntax and would greatly help to decrease the complexity of build pipelines.
   a. I would say point 1. is a subset of these, but a very important one that warrants it's own separate implementation
   b. I also recognize there are other solutions to this, but in my opinion, they would just increase complexity when part of the goal of bootc is to simplify OS image building

[cgwalters]

I've now split this off upstream in https://gitlab.com/fedora/bootc/tracker/-/issues/68 with how I'm thinking of things.

[kwozyman]

Finally, I had some time to look at this in more detail. I've implemented Kickstart support via the kickstart subcommand (i.e. fab kickstart --help).

I've also looked a bit into running Anaconda proper (or a fork) in the container, but things are not as straightforward (it requires DBus support) so I'm implementing the actual execution of the ks commands. I don't think this is a huge hindrance, as the ks commands that make sens in a bootc environment ar not so many or so complicated.

If you agree with this approach, I'd like to continue implementing the kickstart commands that make sense for bootc and maybe even look into how blueprints could be supported as well.

Please let me know what you think.