vfkit,krunkit: Support nested virtualization

[nirs]

Summary

On macOS 15+ with M3, M4, and M5 CPUs, both vfkit and krunkit support nested
virtualization via the --nested flag. This allows running KVM-based workloads
(e.g. KubeVirt VMs) inside a minikube cluster on Apple Silicon — similar to how
this already works on x86_64 with the KVM driver.

minikube should:

1. Accept a --nested flag and pass it through to vfkit/krunkit
2. Include the missing kernel configs in the aarch64 ISO to support nested VM workloads

Background

• Both vfkit and krunkit use Apple's Hypervisor.framework, which supports nested
  virtualization on M3+ with macOS 15+
• vfkit added --nested in PR cmd: add support for nested virtualization crc-org/vfkit#327 (June 2025), closing Add support for nested virtualization crc-org/vfkit#279
• krunkit added --nested in krunkit 0.2.0; Podman enables it by default on
  supported hardware (machine: enable nested virt on libkrun by default containers/podman#25922)
• Lima also added support for nested virt with both vfkit and krunkit
  (krunkit: enable support for nested virtualization lima-vm/lima#4658)
• Requires Apple Silicon M3+ with FEAT_NV2 (ARMv8.4 Nested Virtualization)
• Requires macOS 15 (Sequoia) or later
• On x86_64, minikube already supports running KubeVirt VMs inside the cluster
  using the KVM driver with nested virtualization

Missing kernel configs in the aarch64 ISO

The aarch64 ISO kernel (6.6.95) already has CONFIG_VIRTUALIZATION=y and
CONFIG_KVM=y, which is the primary requirement — the guest kernel uses KVM
normally while macOS Hypervisor.framework handles the nesting transparently.

However, comparing with the x86_64 defconfig, the aarch64 defconfig
(deploy/iso/minikube-iso/board/minikube/aarch64/linux_aarch64_defconfig) is
missing configs needed for running VM workloads like KubeVirt:

Config	x86_64	aarch64	Purpose
CONFIG_KVM	=m	=y	KVM core — present on both
CONFIG_VHOST_NET	=m	missing	In-kernel virtio-net backend — critical for VM networking
CONFIG_VHOST_VSOCK	=m	missing	VM socket communication (host ↔ guest)
CONFIG_VIRTIO_VSOCKETS	=m	missing	vsock transport

These should be added to linux_aarch64_defconfig:

CONFIG_VHOST_NET=m
CONFIG_VHOST_VSOCK=m
CONFIG_VIRTIO_VSOCKETS=m

Driver changes

Both the vfkit and krunkit drivers need to pass --nested to their respective
hypervisors when requested.

vfkit (pkg/drivers/vfkit/vfkit.go): pass --nested in startVfkit().
This follows the same pattern as the existing --rosetta flag (line 342).

krunkit (pkg/drivers/krunkit/krunkit.go): pass --nested in
startKrunkit(), appending it to the args slice (line 242).

minikube should expose this as --nested (defaulting to false), for example:

minikube start --driver=vfkit --nested
minikube start --driver=krunkit --nested

Note: Podman enables nested virt by default on supported hardware
(containers/podman#25922). We may want to consider the same approach in a
follow-up, but starting with an explicit opt-in flag is safer.

Kernel version analysis

The current aarch64 ISO uses Linux 6.6.95 (LTS). There is active development on
ARM64 nested KVM in kernels 6.8–6.16 (kvm-arm.mode=nested, FEAT_ATS1A, etc.),
but this work is all host-side — it's for when Linux KVM itself is the L0
hypervisor and needs to allow its guests to also run KVM.

In the vfkit/macOS scenario, the nesting layers are:

Level	Role	What handles nesting
L0	macOS Hypervisor.framework	Apple — via vfkit/krunkit --nested flag
L1	minikube ISO (guest kernel)	Just needs regular CONFIG_KVM=y
L2	KubeVirt VM	Uses KVM inside L1

The L1 guest kernel doesn't know or care that it's nested — it sees hardware
virtualization extensions exposed by macOS and uses KVM normally. Both 6.6 LTS
and 6.12 LTS have identical CONFIG_KVM support for this use case.

No kernel upgrade is needed for this feature. The missing kernel configs
(CONFIG_VHOST_NET, CONFIG_VHOST_VSOCK, CONFIG_VIRTIO_VSOCKETS) need to be
added regardless of kernel version.

Note: Linux 6.16+ with kvm-arm.mode=nested would only be needed for a
different scenario — running minikube with the KVM driver on an ARM64 Linux
host and nesting VMs inside. That is not what this issue covers.

Requirements

• macOS 15 (Sequoia) or later
• Apple Silicon M3, M4, or M5
• vfkit with --nested support (v0.6.0+), or krunkit 0.2.0+

Related issues

• Add support for nested virtualization crc-org/vfkit#279 — Add nested virtualization support to vfkit
• Enable nested KVM containers/libkrunfw#50 — Enable nested KVM in libkrunfw
• machine: enable nested virt on libkrun by default containers/podman#25922 — Enable nested virt on libkrun by default
• krunkit: enable support for nested virtualization lima-vm/lima#4658 — krunkit: enable support for nested virtualization

[nirs]

@cfergeau anything missing?

[cfergeau]

@cfergeau anything missing?

I don’t know what the full kernel config has, but I’d add as many virtio features as possible if there are config options for them (virtiofs, virtio-serial/console, virtio-blk, virtio-rng, virtio-balloon, …)