key composefs/transient handling off label

[cgwalters]

Per discussion related to bootc-dev/bootc#372 we want to allow configuring non-/usr content in a derived image and hence we need to stop parsing it just from the base commit.

So one strawman here is to parse a label in the metadata.

Another entirely different bigger path is to rework things to always operate on the squashed image like bootc-dev/bootc#215 does (or just target that)

---

Also because users may be relying /usr/local -> /var/usrlocal right now to store data we cannot automatically switch what that does if composefs happens to be enabled...at least for FCOS and existing ostree users.

[jlebon]

Just going to summarize what I remember from a sync @cgwalters and I had related to this and bootc-dev/bootc#372.

• In a composefs world, it's fine to have non-/usr top-level content. /usr is no longer special and there's no read-only bind mount just for it.
• We keep defaulting to a read-only root (default composefs behaviour).
• Trying to make things work in a non-composefs setup is out of scope.
• We'll add an rpm-ostree knob to make /opt and /usr/local be directories instead of symlinks. They need to remain symlinks for compatibility, but for the new base images, we should have them default to directories.
• We don't need state overlays or transient root configurable in the base compose. We'll focus on ensuring that enablement can happen in derived container builds instead (which is this ticket).
• In the case of top-level non-/usr content that also needs mutability, some choices in the derived build are:
  1. Add the required symlinks to /var/... strictly where mutability is needed. This option in theory is the best one since it aligns with the r/o default, but requires knowledge of where to place the symlinks (could have e.g. docs for common packages).
  2. Enable transient root by editing the prepare-root.conf (or short-term, a label on the container, as described in this ticket).
  3. Enable state overlays on the relevant dirs by enabling instances of ostree-state-overlay@.service (e.g. on ostree-state-overlay@opt.service or ostree-state-overlay@usr-local.service).

[cgwalters]

related PR in coreos/rpm-ostree#4850