The right abstraction layer for VMs #47
Description
What
We need to figure out the best layer at which to implement tyche support for VMs.
The options are:
- Implement our own loader/emulator.
- Instrument/modify existing emulators (e.g., firecracker, qemu).
- Extend KVM with a new hardware backend (e.g., to replace kvm's smv or vmx on x86).
Why
Each of the three alternatives comes with its own trade-offs.
Alt-1: Implement our own loader/emulator
pros:
- No need to integrate with existing code.
- Implement only the features we are interested in.
cons:
- Lots of work to do for device emulation.
- Does not integrate well within existing OS (e.g., Linux).
- Limited outreach, i.e., it is less convincing if we say we have our own emulator.
Alt-2: Modify existing emulators
pros:
- user-space so easier to debug/modify.
- already has all the code logic for emulation/creation of devices.
- More convincing argument that we support confidential VMs.
- Small API with KVM.
cons:
- Some of these implementations are really tied to KVM.
- They delegate a lot of the logic to KVM (e.g., for interrupts and how they integrate with Linux).
- Requires a lot more "hypervisor" logic inside tyche to reach the same level of abstraction as kvm. The IRQ logic is tightly integrated to Linux's fds and management/routing of interrupts and events. It is not really possible to do the same in tyche at the monitor level, and reinjecting the interrupts will basically lead to process termination if linux is not able to understand the context of these interrupts. We could implement handlers ourselves and route the vectors all the way back to them, i.e., do a user-space IRQCHIP implementation (@YuChen-Qian said that qemu and nitro do that), but it's a lot of work.
Alt-3: Tyche as a hardware platform and KVM backend
pros:
- All we need to do is replace vmcs/vmx specific operations with tyche ones.
- The layer of abstraction is closer to the hardware and thus a bit simpler.
- Qemu and firecracker should work mostly out of the box (at least for just regular VMs, confidential ones might need a bit of modification).
- Tyche remains simple.
cons:
- There are a lot of functions for a kvm backend, but I hope to be able to reuse most of the ones defined for x86 vmx.
- debugging driver-space code is a bit more complicated.
Adrien's opinion
After having a close look at KVM's configuring interrupts, I think alternative 3 is the most viable one.
It should provide us with a low yet simple enough interface for which we can definitely implement support in Tyche.
If this approach works, we should theoretically be able to use qemu and firecracker out-of-the-box, at least to run sandboxed (non-confidential) VMs.
a. What do you guys think?
b. Should we explore several alternatives in parallel?
c. Do you see another alternative?