Support shared network

[nirs]

It would be useful to have a shared network like lima:shared via socket_vmnet.

Requirements:

• Host can access VM via VM IP (without port forwarding)
• VM can access other VMs
• Good performance

Use case: create multiple k8s clusters for testing managed by OCM. This can be used to test application running on OpenShift and ACM on a laptop. This is possible with minikube and libvirt driver on Linux, and we want to make it possible to do the same using new vfkit driver in minikube.

Looks like gvisor-tap-vsock cannot provide this: containers/gvisor-tap-vsock#383

It seems that the key to get this working is using VZBridgedNetworkDeviceAttachment which is not exposed by vfkit. I understand that his requires code signing with com.apple.vm.networking entitlement, so it may not work for pre-built executables, but can we get this working by building and code signing the executable locally? Our use case is developers and building and code-signing can be automated.

[cfergeau]

This was discussed in #144
The main issue with this is indeed the com.apple.vm.networking requirement.
If your signing keys can enable this entitlement, you can give a try to https://github.com/cfergeau/vfkit/tree/bridge
I haven't been able to test this myself.

[nirs]

Thanks! I think this code is valuable even if it does not work for github build. Can we integrate it with a build option, so it is possible to build a version with bridge support?

[cfergeau]

Yes we can add it so that people with the right entitlement can enable it. I'd prefer to get confirmation the code works though before merging it, but I don't have the required entitlement :(

[nirs]

I know that UTM (at least when installed from the app store) have such entitlement, so this is not impossible to do. I think this will the best way for long term.

We can document that bridge network is experimental.

[cfergeau]

I know that UTM (at least when installed from the app store) have such entitlement, so this is not impossible to do.

I'm trying to get this entitlement for vfkit, let's wait a few weeks to see how this works out.

[cfergeau]

I'm trying to get this entitlement for vfkit, let's wait a few weeks to see how this works out.

This did not work out so well, either I did not file my request in the right place, or they did not understand what I was asking for :-/ In short, I still do not have access to this entitlement.

[cfergeau]

See Code-Hex/vz#180 (comment)
lima is using a root vmnet helper to be able to use this without an entitlement.

[nirs]

See Code-Hex/vz#180 (comment) lima is using a root vmnet helper to be able to use this without an entitlement.

Yes, I mentioned socket_vmnet it in the description. There are 2 issues with this solution:

1. It does not work for vfkit
2. Poor performance

socket_vmnet supports now only the qemu socket device (-net socket,fd=). This uses unix stream socket to forwarding length prefixed packets (4 bytes length, packet payload) This is not compatible with VZ file handle, using unix datagram socket forwarding actual packets. To work with VZ, you need to convert the length prefix packets to standard packets when reading from socket_vmnet, and convert the packets from VZ to length prefixed packets when writing to socket_vmnet. This is done by Lima, creating this pipeline:

socket_vmnet <-unix-> lima <-unixgram-> vz

Instead of:

socket_vmnet <-unixgram-> vz

This leads to poor performance and high cpu usage since every packet is copied twice, once over the unix socket, and again over the unixgram socket.

Lima packet forwarding:
https://github.com/lima-vm/lima/blob/56405610b99ddca5c15f75e5aa0554c4c7ad312f/pkg/vz/network_darwin.go#L65

This is even worse because socket_vmnet is implementing a virtual hub instead of a virtual switch. It forwards packets to all vms connected to the daemon, so every packets is duplicated N times.

You can check the issue here:
lima-vm/socket_vmnet#58

Even if socket_vmnet fixes the issue with multiple vms, it is still 10 times slower than VZBridgedNetworkDeviceAttachment.

I tested this configurations:

configuration	throughput	tool
running iperf3 server and client locally in macos 14.7.1	59.4 Gbits/s	iperf3
VZBridgedNetworkDeviceAttachment using UTM bridged network	40.0 Gbits/s	iperf3
Copying packets between processes over unix datagram socket	21.0 Gbits/s	custom
Copying packets from vmnet to VZ datagram socket with vfkit	8.5 Gbits/s	iperf3
socket_vmnet 1 lima vm	3.5 Gbits/s	iperf3
socket_vmnet 3 lima vm	1.2 Gbits/s	iperf3

We use lima with socket_vmnet and 3 vms and it is good enough for testing DR flows locally, but we don't have much choices.

So the best solution is using the entitlement. This many work only if vfkit is installed via the app store, but I think this is good enough. I'm not sure Apple will be happy with this but we should try.

If this is not possible we can use a helper to connect the a unix datagram socket to vmnet network, or socket_vment if lima-vm/socket_vmnet#77 is implemented.

[njhsi]

try this: https://github.com/njhsi/macos-virtio-net.git
no entitlement needed..

[nirs]

Turns out that UTM installed from brew has "com.apple.vm.networking" entitlement:
lima-vm/lima#3033 (comment)