Add factory reset capability

[bgilbert]

Under the immutable infrastructure model, we encourage users to handle config changes by reprovisioning their systems, rather than using configuration management or manually modifying the running system. This is okay in virtualized setups or in the cloud, and is also okay on bare metal where PXE infrastructure or IPMI-assisted ISO boot is available. But on single-node bare metal setups without external infrastructure, manual reinstall is impractical. (See e.g. this discussion.)

For this use case, it might help to have a command which receives an Ignition config to use for reprovisioning, copies it into /boot, sets first-boot kargs, and reboots. Then the initramfs, before running the usual first-boot steps, would need to restore the system to a pristine state. That process might look a lot like #94, or might look more like clearing /var and resetting /etc from /usr/etc. If those are in non-standard locations, perhaps the user-mode command could add kargs which tell the initramfs where to find them.

Because we'd be deleting user customizations before rerunning Ignition, this approach wouldn't[*] get into unsupported territory of using Ignition for configuration management. It would slightly complicate OS maintenance, though. At present we can freely change the Ignition initrd glue in ways that are incompatible with old installs (e.g. by changing first-boot kargs) because old installs will never run Ignition again. Supporting factory reset of old nodes would presumably add some additional constraints.

[*] Presumably it'd be infeasible to delete every possible user customization, so that statement might be optimistic.

[bgilbert]

Discussed in the community meeting today.

• If we committed the Ignition config to the ostree repo, that could make this a bit easier.
• coreos-installer used to be in the initramfs, which technically allowed every system to reinstall itself outright (assuming network access). We could investigate running the new coreos-installer from the initramfs again, but it requires lsblk, udevadm, and gpg.
• We've discussed splitting the live PXE root squashfs out to a separate stage2 (Publish the initrd and rootfs images separately for PXE booting #390) which could be useful here: assuming network access, the existing installed kernel and initrd could boot themselves into a live PXE image and rerun coreos-installer from there. That doesn't let air-gapped systems reprovision without external assistance, though.
  • Assuming network access, we could also fetch the image before the reboot and stash it somewhere on disk. But if the install fails and overwrites the image in the process, you don't get a second chance.
  • To solve the air-gapped case, we could: 1) before reboot, assemble a live squashfs from the ostree we already have, 2) from the initramfs, mount the old root filesystem and copy the squashfs into RAM, 3) create a temporary partition at the end of the disk and copy the squashfs into it for safety, 4) run coreos-installer, and if it fails, recreate /boot and the safety partition, 5) reboot into the new system. That's complex though.
• We didn't explore it in the meeting, but rerunning coreos-installer is not mandatory if there's a good way to delete user customizations from the existing system. We probably won't be able to get the system to 100% pristine state though.

There was a general sense that the feature would be nice, but there was also concern about implementation complexity depending on the approach taken.

[jlebon]

Note with the stage 2 approach, as long as the stage 2 binary is the initrd we append to the base one, it should contain both the osmet file and the root squashfs, so it should be possible to do the re-install completely offline.

[jlebon]

Related OSTree issue: ostreedev/ostree#1793

[cgwalters]

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

(This all said I think a lot of cases are going to be happy enough with the "ostree level" factory reset and not "wipe and replace disk image", particularly because the former can be made transactional)

[cgwalters]

As is today, it will work in many basic scenarios to just do:

$ unshare -m /bin/sh -c 'mount -o remount,rw /boot && touch /boot/ignition.firstboot'
$ reboot

A much stronger version would look like touch /run/factory-reset and we go back into the initramfs at shutdown time and do e.g. rm /var/* -rf && rsync -rlv --delete /usr/etc/ etc/.

The strongest version here of course is re-fetching and re-imaging the target disk from the initramfs, but that's even more involved.

[jlebon]

Also, if we shipped the .osmet file in e.g. /sysroot/fcos.osmet, then we could always support a factory reset back to the aleph image. If we wanted to support reset back to the current ostree commit we'd need a tool to fetch it out of band.

Truly shipping the osmet file in the image is impossible because it's a circular dependency. So it would probably instead require coreos-installer to propagate it there manually at install time. But the problem is that the osmet file is tied to the OSTree commit of the aleph version, so if the node upgraded at all, we would need to fetch the aleph OSTree commit. One thing we could easily do is have coreos-installer add a ref to it it's not GC'ed, but then every node is paying that storage cost.