os: Manage OS scripts and config files using Forklift

[ethanjli]

Motivation

Currently, various files are copied into place by the distro setup scripts:

• systemd unit files
• shell scripts executed by systemd services
• /etc config files
• /boot/config.txt (and/or snippets of it)
• $HOME/.local/etc config files
• local static binaries such as machine-name

This makes them hard to develop (because the develop/test iteration loop involves re-running the distro setup scripts on a fresh SD card image each time), and it also makes them hard to upgrade with integrity (because the SD card must be re-flashed to apply new changes). If we could manage such files with Forklift, then it would be easier iterate on these things, and to deploy them.

Goals

• Reduce the OS-related work done by the distro setup scripts purely to installing a few Debian packages and setting up Forklift; application-related work is covered by other tasks under os/release: Make it safe, simple, and easy for users to apply software updates #133 and is a non-goal for this issue.

Steps

• Manage shell scripts in Forklift
• Mount /etc and /usr as overlay filesystems, each with an intermediate layer managed by Forklift (so that Forklift can deliver arbitrary files into /etc and /usr at startup, and change which files it delivers in an atomic way)
• Manage new systemd units and overrides with Forklift
• Manage overrides for existing systemd services with Forklift
• Manage arbitrary files in /etc and /usr with Forklift
• Manage static binaries with Forklift

Unresolved Questions

• Can we use Forklift to download newer versions of Forklift and bootstrap the selection of a newer version during boot? (UPDATE: moved to toolchain: Enable bootstrapping into a different Forklift version forklift-run/forklift#237)
• Can we use Forklift (perhaps with some scripts as hooks before/after/during Forklift operations) to manage /boot/config.txt? (UPDATE: moved to host/files: Enable managing bootloader/kernel files used before early boot forklift-run/forklift#241)

Implementation History

• forklift-run/forklift@c1bdfad (on the main branch of the repo) added an easy way to resolve the location of a file provided by a Forklift package referenced by a Forklift pallet
• Add scripts and local config files from the PlanktoScope distro PlanktoScope/device-pkgs#5 and Bump device-pkgs repo, add deployments for new packages PlanktoScope/pallet-standard#5 and os/pallet: Move some scripts and local files over into management by forklift #350
• Provision dhcpcd config file PlanktoScope/device-pkgs#6 and Provision dhcpcd config file PlanktoScope/pallet-standard#6 and Add support for usb0, eth1, and wlan1 interfaces #351
• Stage pallets before applying them forklift-run/forklift#172 and Enable exporting files which can be combined with OS files via an overlay filesystem forklift-run/forklift#178
• Migrate OS config files into the Forklift pallet #397
• Allow exporting files from HTTP URLs by downloading them forklift-run/forklift#195 and Preserve source file perms in http-archive file exports forklift-run/forklift#200 and Add support to download OCI images and export files from them forklift-run/forklift#206
• Deploy machine-name via Forklift with pallet-standard rather than the OS setup scripts #411

[ethanjli]

Following up on comments from #133:

• It would be worth investigating whether we can use chezmoi as a library to manage both homefiles and etcfiles (from os/release: Make it safe, simple, and easy for users to apply software updates #133 (comment)) - and maybe even also for managing binaries
• For software which only needs to run as a systemd service, it would be worth investigating whether we can get systemd portable services (provided by the systemd-container apt package) working, and whether we could manage them using Forklift (perhaps using https://github.com/coreos/go-systemd?). One key question is whether a portable service can modify files outside its root filesystem.
• For software which also need to be made available as a binary to run (which is maybe just docker), it would be worth investigating whether we can use systemd-sysext and systemd-confext (os/release: Make it safe, simple, and easy for users to apply software updates #133 (comment) and os/release: Make it safe, simple, and easy for users to apply software updates #133 (comment)), at least after we upgrade from Raspberry Pi OS bullseye to Raspberry Pi OS bookworm (os/platform: Add software support for Raspberry Pi 5 #322)

[ethanjli]

Apparently systemd-confext makes /etc read-only until after the extension is unmounted (because the config extensions are mounted over /etc). I think we really should try to keep /etc mutable for tinkering, so this rules out systemd-confext as something that might be useful for us. Any config files should instead be added using systemd-sysext to a directory inside /usr.

[ethanjli]

One potential simple yet flexible approach for putting arbitrary files in arbitrary locations on the filesystem in an atomic way could be to use bind mounts (maybe with systemd, e.g. as in https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdBindMountUnits) and/or OverlayFS (maybe with systemd, e.g. as in https://bootlin.com/blog/systemd-read-only-rootfs-and-overlay-file-system-over-etc/, which discusses being able to add/modify systemd services using the OverlayFS)