Investigate usage of Rechunk

[fiftydinar]

It can fix some OCI image issues with permissions & SELinux, while nicely lowering the image size.

https://github.com/hhd-dev/rechunk

Bazzite, Bluefin & Aurora are already experimenting with this.

Would be smart to wait & see how Rechunk will refine over time, while Ublue images use it.

[prydom]

Working example using OSTree's built-in rechunk feature here that I've been using for months:

https://github.com/prydom/my-ostree-build/blob/ed2b05b29bac9dc0152862f198f166b63380c044/.github/workflows/build.yml#L108-L129

What I currently do is use bluebuild to push the raw image (gzip compressed), then pull with ostree, recommit to relabel, rechunk it (ostree container-encapsulate does not support zstd), then re-compress it to non-chunked zstd (I was running into bugs with chunking), then sign the final image.

It would be nice if some of those steps were integrated together to save time/bandwidth.

[prydom]

Ah, looking at the rechunk repo @hhd-dev and I came up with the same thing independently - I did it to solve SELinux problems with libvirt and increase layer reuse. Consider my example above an alternative implementation using the same primitives - I added it back in April.

[prydom]

https://github.com/hhd-dev/rechunk/blob/master/1_prune.sh does some very invasive and potentially problematic, when used across distros, actions with regards to permissions that ostree-rs-ext handles/applies more safely by using the ostree metadata during ostree checkout. I will report that to the upstream project.

For example, I do very few of the workarounds found in prune (only similar truncating of /etc/passwd and /etc/group and the /usr/etc->/etc/ merge) and have had no issues with booting or permissions.

[prydom]

I had a good discussion with the maintainer of the "rechunk" repo at hhd-dev/rechunk#4.

We discussed the pros and cons of using ostree-rs-ext to extract the post-processed rootfs vs their approach of quirking the permissions and exattrs from a copy of the container image from podman mount. The latter approach requires rootful permissions on builders and the former approach is the only way to maintain xattrs from a base image. However there is a concern about doubling build space requirements when using the former approach, as currently ostree-rs-ext doesn't support unified storage with other containers/storage based tools (e.g. podman). It may be worth waiting for something like bootc-dev/bootc#20 to land or possibly code from bootc-dev/bootc#215 could be adapted to obtain a rootfs directory that is appropriate for rechunking and avoid an additional copy of the container image being created. We also discussed upstream issue ostreedev/ostree-rs-ext#655 which prevents OCI layered xattr changes from being applied (which may not impact bluebuild if re-chunking is always the last stage).

Also note that https://github.com/hhd-dev/rechunk uses its own chunking algorithm distinct from https://github.com/ostreedev/ostree-rs-ext/blob/main/lib/src/container/encapsulate.rs. It may be worth evaluating if the techniques from @hhd-dev 's repo are appropriate to contribute upstream.