How to start a amd64 based cluster on macOS running Apple Silicon?

[hcguersoy]

Hi Folks,

is there a way to start a Kubernetes cluster in RD for amd64 on an Apple Silicon Mac?

AFAIK Lima supports such scenarios (see Intel-on-ARM and ARM-on-Intel) but not found any hint in the RD documentation.

Unfortunately I've to work with some intel-only images 😬

[jandubois]

There is no mechanism for this yet. I find running the whole VM in virtualization to be really slow, so I'm hoping that we can eventually implement a multi-node cluster inside the single VM, so only a kubelet instance and its containers need to be emulated.

We already support this at the container runtime level:

$ nerdctl run --platform linux/arm64 busybox uname -a
Linux 642f631c0d94 5.10.82-0-virt #1-Alpine SMP Fri, 26 Nov 2021 13:19:46 +0000 aarch64 GNU/Linux
$ nerdctl run --platform linux/amd64 busybox uname -a
Linux bf243802ec6d 5.10.82-0-virt #1-Alpine SMP Fri, 26 Nov 2021 13:19:46 +0000 x86_64 GNU/Linux

Also works with dockerd runtime and the docker cli. But for k8s the platform is determined by the kubelet, so all pods/containers in a single node need to be for the same architecture.

We don't have a way to run a different architecture just for k3s either, so right now this is not supported.

[hcguersoy]

@jandubois Thank you for your detailed answer.

[corneliusroemer]

Any update here in the 21 months since? Would be a great feature.

[jandubois]

Unfortunately not, and it isn't on the immediate roadmap either. As you can see, you are only the 3rd person expressing an interest here, and that includes the single 👍 on my previous reply.

There is only so much we can do, and issues with more comments and reactions get prioritized.

[charlieok]

Seems worth mentioning here that I'm looking around for how to do a good setup for both arm64 and amd64 containers on an Apple Silicon macOS machine. Possibly using Rancher Desktop or possibly something else.

Sounds like that means a VM running under rosetta 2 acting as an x86 linux host for containerd/kubelet, and another VM running on arm64 to provide a linux host environment for an arm64 containerd/kubelet.

Maybe this is best done as two separate stacks of tools, one in rosetta 2 world and one not? Or maybe there is some nice Rancher Desktop like tool that would tie it all together and let those VMs be connected as a cluster?

edit: upon further reading maybe this doesn't require two VMs
https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta

[jose-rojas0]

In case it's useful to anyone else landing on this issue, I've been able to use amd64 images by doing the following:

• make sure the Virtual Machine - Emulation setting is set to VZ (Apple Virtualization Framework) and that Rosetta support is enabled
  • it may work with QEMU too, but I have not tried it
• manually pull the images before trying to run them on the cluster

export DOCKER_DEFAULT_PLATFORM=linux/amd64

docker pull public.ecr.aws/bitnami/rabbitmq:4.2.1

it seems the container engine won't verify that the image present on the machine is the same architecture and will just happily run it