Description
This is split out from discussions that started in #656.
To summarize the problem statement:
- The root issue is that Ignition today does not require a config. So to do its job, it needs to be able to query whether there is a config. In the case of user-data over the network for example, it can simply wait until networking comes online and
GETthe URL. If it's a 404, it can move on. - There is no surefire way to know when the kernel has finished discovering all attached storage devices. This intuitively makes sense because many types of storage device are designed to be hotpluggable, and the architecture reflects this. Additionally, storage device hierarchy may be complex in some environments, resulting in a long discovery phase as more nested devices come online. The
fw_cfgdevice works around this because it's memory-mapped, but it's only available on a subset of architectures. - Therefore, in the case of an Ignition config stored on a storage device, Ignition cannot tell the difference between "config was not provided" or "config was provided but the device hasn't yet come online".
- It's also worth noting a lot of this pain also comes about from wanting to avoid extra supporting tools for launching VMs.
One suggestion then is to change the semantics of Ignition slightly on QEMU: we don't require an Ignition config, but we require the device through which we can determine whether an Ignition config was provided. So e.g. we can require a CD drive, even if there is no "CD" inserted (not actually suggesting this though, see below). Then we can afford to just wait forever until whatever device we want shows up. In a way, this is consistent with waiting until networking comes up.
Note that unlike most cloud services that have a metadata service for SSH keys, on QEMU there is no other way to configure the guest anyway. Therefore, while there might be some use cases for booting a QEMU guest without a config, they're likely very rare. (And again, note that this change doesn't make Ignition technically require a config, but simply the device itself.)
So, what's the best way to implement this? Three primary criteria:
- it should be supported across architectures
- it should be easily identifiable as "the Ignition device" from the guest side
- it should be the minimal amount of set up/hassle for users
So for example, while virtio-serial is AFAIK supported across all arches, and can have custom names, it would require a shim on the host side to communicate with (unless something like suggested in #656 (comment) is implemented).
A CD drive would also work, but it requires users to create an ISO first, which is annoying.
What I'm playing with right now is a virtio-blk device with a serial of ignition:
host$ echo '{"foo": "bar"}' > config.ign
host$ cosa run -- -drive file=config.ign,if=none,format=raw,readonly=on,id=ignition -device virtio-blk,serial=ignition,drive=ignition
guest$ cat /dev/disk/by-id/virtio-ignition
{"foo": "bar"}
This more or less fits all three criteria by not requiring much preparation, being supported across architectures, and being easily identifiable from the guest side. The CLI is clearly less elegant than -fw_cfg, though ideally you'd be copy-pasting it anyway.
Of course, if you have other candidate devices that fit the criteria, definitely comment!