Shared network for vfkit driver using vmnet-helper

[nirs]

Add new network option for vfkit "vment-shared", connecting vfkit to the
vmnet shared network. Clusters using this network can access other
clusters in the same network, similar to socket_vment with QEMU driver.

If network is not specified, we default to the "nat" network, keeping
the previous behavior. If network is "vment-shared", the vfkit driver
manages 2 processes: vfkit and vmnet-helper.

Like vfkit, vmnet-helper is started in the background, in a new process
group, so it not terminated if the minikube process group is terminate.

Since vment-helper requires root to start the vmnet interface, we start
it with sudo, creating 2 child processes. vment-helper drops privileges
immediately after starting the vment interface, and run as the user and
group running minikube.

Stopping the cluster will stop sudo, which will stop the vmnet-helper
process. Deleting the cluster kill both sudo and vment-helper by killing
the process group.

This change is not complete, but it is good enough to play with the new
shared network.

Example usage:

1. Install vmnet-helper:
   https://github.com/nirs/vmnet-helper?tab=readme-ov-file#installation

2. Setup vment-helper sudoers rule:
   https://github.com/nirs/vmnet-helper?tab=readme-ov-file#granting-permission-to-run-vmnet-helper

3. Start 2 clusters with vmnet-shared network:

% minikube start -p c1 --driver vfkit --network vment-shared
...

% minikube start -p c2 --driver vfkit --network vmnet-shared
...

% minikube ip -p c1
192.168.105.18

% minikube ip -p c2
192.168.105.19

4. Both cluster can access the other cluster:

% minikube -p c1 ssh -- ping -c 3 192.168.105.19
PING 192.168.105.19 (192.168.105.19): 56 data bytes
64 bytes from 192.168.105.19: seq=0 ttl=64 time=0.621 ms
64 bytes from 192.168.105.19: seq=1 ttl=64 time=0.989 ms
64 bytes from 192.168.105.19: seq=2 ttl=64 time=0.490 ms

--- 192.168.105.19 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.490/0.700/0.989 ms

% minikube -p c2 ssh -- ping -c 3 192.168.105.18
PING 192.168.105.18 (192.168.105.18): 56 data bytes
64 bytes from 192.168.105.18: seq=0 ttl=64 time=0.289 ms
64 bytes from 192.168.105.18: seq=1 ttl=64 time=0.798 ms
64 bytes from 192.168.105.18: seq=2 ttl=64 time=0.993 ms

--- 192.168.105.18 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.289/0.693/0.993 ms

To complete this work we need:

• Update start flags to consider network options for vfkit
• Validate that vmnet is installed when using --network vment-shared
• Document vfkit driver network options
• Install vmnet-helper on the CI hosts
• Add test using --network vmnet-shared

Based on #20506

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nirs
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Hi @nirs. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[minikube-bot]

Can one of the admins verify this patch?

[nirs]

@afbjorklund can you review this?

[nirs]

Example machine config when vmnet-shared is used:

{
    "ConfigVersion": 3,
    "Driver": {
        "IPAddress": "192.168.105.21",
        "MachineName": "minikube",
        "SSHUser": "docker",
        "SSHPort": 22,
        "SSHKeyPath": "",
        "StorePath": "/Users/nir/.minikube",
        "SwarmMaster": false,
        "SwarmHost": "",
        "SwarmDiscovery": "",
        "Boot2DockerURL": "file:///Users/nir/.minikube/cache/iso/arm64/minikube-v1.35.0-arm64.iso",
        "DiskSize": 20000,
        "CPU": 2,
        "Memory": 6000,
        "Cmdline": "",
        "ExtraDisks": 0,
        "Network": "vmnet-shared",
        "MACAddress": "de:a2:9c:71:3c:f7",
        "VmnetHelper": {
            "MachineDir": "/Users/nir/.minikube/machines/minikube",
            "InterfaceID": "a0c43efb-2dcc-4abb-a310-568782c5dc7a"
        }
    },
    "DriverName": "vfkit",
    "HostOptions": {
        "Driver": "",
        "Memory": 0,
        "Disk": 0,
        "EngineOptions": {
            "ArbitraryFlags": null,
            "Dns": null,
            "GraphDir": "",
            "Env": null,
            "Ipv6": false,
            "InsecureRegistry": [
                "10.96.0.0/12"
            ],
            "Labels": null,
            "LogLevel": "",
            "StorageDriver": "",
            "SelinuxEnabled": false,
            "TlsVerify": false,
            "RegistryMirror": [],
            "InstallURL": "https://get.docker.com"
        },
        "SwarmOptions": {
            "IsSwarm": false,
            "Address": "",
            "Discovery": "",
            "Agent": false,
            "Master": false,
            "Host": "",
            "Image": "",
            "Strategy": "",
            "Heartbeat": 0,
            "Overcommit": 0,
            "ArbitraryFlags": null,
            "ArbitraryJoinFlags": null,
            "Env": null,
            "IsExperimental": false
        },
        "AuthOptions": {
            "CertDir": "/Users/nir/.minikube",
            "CaCertPath": "/Users/nir/.minikube/certs/ca.pem",
            "CaPrivateKeyPath": "/Users/nir/.minikube/certs/ca-key.pem",
            "CaCertRemotePath": "",
            "ServerCertPath": "/Users/nir/.minikube/machines/server.pem",
            "ServerKeyPath": "/Users/nir/.minikube/machines/server-key.pem",
            "ClientKeyPath": "/Users/nir/.minikube/certs/key.pem",
            "ServerCertRemotePath": "",
            "ServerKeyRemotePath": "",
            "ClientCertPath": "/Users/nir/.minikube/certs/cert.pem",
            "ServerCertSANs": null,
            "StorePath": "/Users/nir/.minikube"
        }
    },
    "Name": "minikube"
}

[nirs]

Example multi-node cluster

% minikube start --network vmnet-shared --nodes 2 --cni auto
😄  minikube v1.35.0 on Darwin 15.3.1 (arm64)
✨  Using the vfkit (experimental) driver based on user configuration
❗  --network flag is only valid with the docker/podman, KVM and Qemu drivers, it will be ignored
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=4050MB, Disk=20000MB) ...
📦  Preparing Kubernetes v1.32.2 on containerd 1.7.23 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner

👍  Starting "minikube-m02" worker node in "minikube" cluster
🔥  Creating vfkit VM (CPUs=2, Memory=4050MB, Disk=20000MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.105.22
📦  Preparing Kubernetes v1.32.2 on containerd 1.7.23 ...
    ▪ env NO_PROXY=192.168.105.22
    > kubelet.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubeadm.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl:  53.25 MiB / 53.25 MiB [------------] 100.00% 37.13 MiB p/s 1.6s
    > kubelet:  71.75 MiB / 71.75 MiB [------------] 100.00% 12.43 MiB p/s 6.0s
    > kubeadm:  66.81 MiB / 66.81 MiB [------------] 100.00% 10.11 MiB p/s 6.8s
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

% kubectl get node                                                           
NAME           STATUS   ROLES           AGE     VERSION
minikube       Ready    control-plane   5m1s    v1.32.2
minikube-m02   Ready    <none>          4m40s   v1.32.2

% kubectl get node -o jsonpath='{.items[*].status.addresses[0].address}{"\n"}'
192.168.105.22 192.168.105.23

Each node get its own vment-helper process:

% ps au | grep vmnet-helper | grep -v grep 
nir  60260   0.0  0.0 410743984   3792 s020  S     1:07AM   0:00.71 /opt/vmnet-helper/bin/vmnet-helper --fd 21 --interface-id c9ae713b-5937-42de-9018-bbb95a425506
root 60259   0.0  0.0 410737168   6448 s020  S     1:07AM   0:00.01 sudo --non-interactive --close-from 22 /opt/vmnet-helper/bin/vmnet-helper --fd 21 --interface-id c9ae713b-5937-42de-9018-bbb95a425506
nir  60254   0.0  0.0 410735792   3728 s020  S     1:07AM   0:00.73 /opt/vmnet-helper/bin/vmnet-helper --fd 13 --interface-id cf62fb28-2533-4c25-8d35-1eef1736f707
root 60253   0.0  0.0 410754576   6992 s020  S     1:07AM   0:00.01 sudo --non-interactive --close-from 14 /opt/vmnet-helper/bin/vmnet-helper --fd 13 --interface-id cf62fb28-2533-4c25-8d35-1eef1736f707

[afbjorklund]

@afbjorklund can you review this?

I could take a look at it later perhaps, but one of the minikube maintainers will still need to "take over" the vfkit driver.

Probably should have an issue, as well.

[cfergeau]

I also noticed several vment typos in commit logs

[medyagh]

maybe check it is not installed and Suggest ppl to install it

[nirs]

I plan to use NotFoundVmnetHelper in the same way we handle NotFoundSocketVmnet.

If you ask for --network vmnet-shared and vment-helper is not installed or not configured properly, you will get link to the docs on how to install and configure it.