Minikube needs Mac Instances

[upodroid]

Minikube requires mac instances for testing. The mac runners from GitHub don't work as they don't support nested virtualisation.

I'm suggesting the following:

• 3x M2 Instances on AWS at the cost of 1500 USD per month. https://instances.vantage.sh/aws/ec2/mac2.metal?currency=USD&duration=monthly
• The VMs would be used with Prow and availability state managed in Boskos. The SSH credentials will be available there.
• The VMs will be deployed in a dedicated AWS account called k8s-infra-minikube, k8s-infra-macs.
• AWS requires the macs to be launched on a dedicated hosts with a mininum runtime of 24 hours so we'll have the macs running 24/7.
• The AWS infra would be managed via terraform and no clickops.

@kubernetes/sig-k8s-infra-leads Thoughts?

Part of kubernetes/minikube#20887 and kubernetes/minikube#21180

/sig k8s-infra

[BenTheElder]

The mac runners from GitHub don't work as they don't support nested virtualisation.

More specifically, the apple silicon runners don't currently (but could in the future if they move to M4), the intel runners do.

x-ref: lima-vm/lima#2390 (reply in thread) https://github.com/orgs/community/discussions/160591

The VMs will be deployed in a dedicated AWS account called k8s-infra-minikube.

We might consider something like "k8s-infra-macos", I could see this being useful for other subprojects later but given the 24/7 uptime we probably want to rent the mac instances from a single shared pool similar to the generic e2e accounts/projects.

Otherwise if another project asks for mac instances, we'll probably have wasted uptime.

---

EDIT: Suggestion / nit aside, SGTM.

[dims]

@upodroid can the minikube folks please document a bit more than "Minikube requires mac instances for testing." for the record please?

[upodroid]

@medyagh ^

There is some info at kubernetes/minikube#21180

[ameukam]

I think we need to define a threshold and some budget alerting if we want to consider a shared pool of machines for the entire community.
We can't provide infinite scaling while it's not guaranteed our funding will stay the same forever.

[BenTheElder]

I think we need to define a threshold and some budget alerting if we want to consider a shared pool of machines for the entire community.
We can't provide infinite scaling while it's not guaranteed our funding will stay the same forever.

There is no autoscaling, but also boskos is not a form of access control, other projects could already use it. I'm just saying framing it as minikube-only seems inefficient. We will surely have reasons to test other tools on macOS (perhaps even kubernetes/kubernetes kubectl ...), and we will not want to create seperate pools of 24/7 machines at odd sizes.

We don't do this for any of the e2e test accounts.

The budget will be the number of machines we turn up.

These jobs should probably be configured to use a prow job queue with a maximum concurrency set to match the number of machines, because unlike GCE test projects we're unlikely to aggressively scale this up since it has fixed 24/7 costs.

We have that for some of the scale testing and image promotion jobs.

[upodroid]

For now, let's deploy 3 instances and assess our status in a quarter.

[medyagh]

Thank you @upodroid and @dims we really need macos testing for drivrers and ISO on macbook,
according to minikube survey macos is most used OS for minikube develoeprs
the current intel only has x86 that can only test our depericated hyperkit driver
and also the current macos-13 are unusable too due to extremely high cpu loads. (we currently log the cpu load) we dont get any instance with less than 20 cpu loads (which is un-usable)

plus all our not-depreciated drivers are arm64 bsased (krunkit, vfkit,) and that needs testing, we currently dont have any testing on macos ( we used to have mac staduim subscription)

any testing on macos will be extremely valuable since we are blind at the moment relying on manual testing.

@upodroid is it possible to get this through github action large runners?

I suggest 4 would be better number to start with, to test all 4 drivers in parallel
vfkit, krunkit, qemu and docker

[medyagh]

Based on minikube surveys macos among our most used platforms
39% Linux
35% macOS
26% windows

[BenTheElder]

@upodroid is it possible to get this through github action large runners?

The CNCF has to pay for the hosted large runners now with actual currency, and doesn't have the budget, they've been moving other projects off of actions large runners to linux runners in Kubernetes on oracle cloud, there's more on that in #cncf-ci-infra in the CNCF slack.

EDIT: We also don't have a way to pay for these ourselves. And we know the CNCF won't since they're already actively trying to exit this. But we can get macOS instances on AWS and manage them ourselves.

We already run a large CI infrastructure with automation to ensure resource management, so it makes more sense to leverage that versus what the CNCF did, and their solution also doesn't provide for macOS.

[upodroid]

@upodroid is it possible to get this through github action large runners?

As Ben said we won't be able to offer them via Actions, but they will be available via Prow. Also, the minikube CI needs be reimplemented/rewritten and I have an open PR showcasing/prototyping this approach kubernetes/minikube#21607.

etcd went through the same exercise, of migrating their CI to Prow and leveraging our toolchain since 2024 with excellent results and new capabilities #6102

[medyagh]

thanks how can we invoke creation of the macos vms, any examples for that?

[medyagh]

@upodroid ping