enabling nested virtualization in gcp for minikube tests

[medyagh]

we added minikube tests that will run in a disposable GCP project (as required by sig-infra)
kubernetes/test-infra#35739

However we found out the tests are signifacntly slower than the Smaller instance in GCP with enabled nested virtualization (currently hosted in internal gcp project by google)

to enable that, we need disk images that support nested virtualization. (instead of vanila ubuntu 24).

in the current set up we spin up a new gcp project using boskos api and the project is destroyed after each test which means we can not have a disk image that we can use for nested-virtualization enabled instances

we already verified that we have added --enable-nested-virtualization

I would like to know what the test-infra suggests to solve this issue.

Example of test duration results are explained here
kubernetes/minikube#21826

Prow (32 cores - big instance - slower tests results):
https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/minikube/21556/integration-kvm-docker-linux-x86-64/1982572213521354752/artifacts/test.html

minikube internal gcp project (8cores much smaller instance):
https://storage.googleapis.com/minikube-builds/logs/master/42124/KVM_Linux.html

Benchmark Example (Prow) (32 cores n2) :
TestKubernetesUpgrade | 676.92 seconds

Benchmark Minikube Jenkins (8 cores n2):
TestKubernetesUpgrade | 208.78

[k8s-ci-robot]

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ameukam]

Why not use a different disk type to improve spin up ?

[medyagh]

@ameukam by disk type do you mean using SSD ?
I believe we already doing SSD @ComradeProgrammer can verify

but the currentl problem is we run KVM VMs inside a gcp VM that requires special work to make nested-virtualization enabled

as explained here
https://docs.cloud.google.com/compute/docs/instances/nested-virtualization/enabling

[ComradeProgrammer]

we did tried to use ssd(e.g using --boot-disk-type=pd-ssd, previously we use pd-standard)but it gives almost no performance improvement. The bottleneck is not the disk IO, but the kvm itself. Previously we build an special image and host it in a private gcp project to improve this. But now we don't know how to host this image.

[BenTheElder]

K8s infra folks: please make sure these will be garbage collected properly so as to not impact the stability of other CI jobs using the projects, and ideally strongly isolated from the projects hosting the core CI cluster. Otherwise this is a k8s-infra question and the config for that is in k8s.io

/transfer k8s.io