iso: support building the arm64 ISO on an arm64 host#22862
iso: support building the arm64 ISO on an arm64 host#22862abedegno wants to merge 2 commits intokubernetes:masterfrom
Conversation
Add a buildroot patch applied during the 'buildroot' target after clone. The patch extends BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE1_ARCH_SUPPORTS's default-y list to include aarch64, matching the arches that host-go-bin actually supports. Without it, on an arm64 host BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS is unset and every Go-target package (cri-dockerd, cni-plugins, podman, crio-bin, ...) silently drops out of the arm64 minikube ISO, producing a 319 MB image that fails provisioning at the docker.service restart step. With the patch the ISO is back to the expected ~403 MB and boots cleanly under vfkit, qemu2, and virtualbox. The proper fix belongs upstream in buildroot; this is a pragmatic near-term workaround for local arm64-host builds (e.g. Colima on Apple Silicon). The new patches/buildroot/ directory is wired into the existing 'buildroot' target so any future per-host fixes can be dropped there without further Makefile changes.
k8s-ci-robot
added
cncf-cla: yes
|
Hi @abedegno. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Can one of the admins verify this patch? |
|
/ok-to-build-iso |
|
/cc @afbjorklund |
|
/cc @medyagh |
nirs
left a comment
nirs
left a comment
There was a problem hiding this comment.
Changes looks good to me, but the best way to validate the change is to post a patch to buildroot. The fix is like much simpler if we don't have to keep a patch file.
|
@abedegno maybe open an issue for building on macOS? the existing issue is about building on Fedora - but the issue fixed here is not related to Fedora. |
|
Hi @abedegno, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further. |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
kvm2 driver with docker runtime DetailsTimes for minikube (PR 22862) start: 37.3s 35.5s 36.5s 35.7s 35.3s Times for minikube (PR 22862) ingress: 16.2s 15.2s 15.7s 15.7s 15.2s docker driver with docker runtime DetailsTimes for minikube (PR 22862) start: 20.6s 18.1s 21.6s 18.4s 17.9s Times for minikube ingress: 10.6s 10.6s 40.6s 10.6s 10.6s docker driver with containerd runtime DetailsTimes for minikube start: 16.7s 16.2s 16.0s 18.8s 19.3s Times for minikube ingress: 23.1s 23.1s 23.1s 23.1s 23.1s |
|
Quick triage of the failed lanes - they all look environmental to me, and I don't think anything this PR changes can affect them. Flagging so we can kick a retest rather than chase them. |
|
/retest |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: abedegno, afbjorklund The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
It will be hard to get patches from upstream, if never updating the operating system... (this bug is on master too, though. buildroot doesn't really try to build on non-x86) |
|
Two failed lanes from the previous retest - appear to be environmental again. Retesting both lanes individually: /test pull-minikube-kvm-docker-linux-x86 |
Sure, this is another gap we need to close.
We also build on x86_64 in CI, but we want to build on aarch64 in GitHub actions for faster build. |
|
@abedegno: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
All required checks green:
|
6 participants
Fix for arm64-host buildroot builds of the minikube arm64 ISO.
Problem
On an arm64 Linux host (e.g. Colima on Apple Silicon),
make minikube-iso-aarch64completes with exit 0 but silently produces a ~319 MB ISO instead of the expected ~403 MB. The ISO boots fine, butminikube startfails during provisioning:Same failure mode on vfkit, qemu2, and virtualbox (so it isn't driver-specific).
Root cause
buildroot's
BR2_PACKAGE_HOST_GO_BOOTSTRAP_STAGE1_ARCH_SUPPORTSonly default-y's for x86, x86_64 and arm hosts. When the buildroot host is aarch64 the symbol stays unset, which leavesBR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTSunset, which silently drops every Go-target package:cri-dockerd-aarch64,cni-plugins-latest-aarch64,podman,crio-bin, plus transitive deps.docker-bin-aarch64has no host-go dependency and still installs, which is why the docker binary ends up in the ISO but none of the CRI service glue does.The gate is the right shape for
host-go-src(source bootstrap of Go has real host arch restrictions), but minikube resolves host-go viahost-go-binwhich has a prebuilt Go binary for aarch64 and works fine. The stage1 gate is overly strict for thehost-go-bincase.Fix
Adds a buildroot patch at
deploy/iso/minikube-iso/patches/buildroot/0001-host-go-aarch64-stage1.patch:Extends the
buildroot:target in the top-level Makefile to apply any*.patchfiles dropped into that directory right after the initialgit clone. Any future per-host fixes can be added there without further Makefile work.The proper long-term fix belongs upstream in buildroot; this is a pragmatic near-term workaround so local arm64-host builds produce the same artifact as CI.
Test
On darwin/arm64 (M-series Mac, Colima
--vm-type vz --arch aarch64):minikube-v1.38.0-arm64.iso),cri-docker.service,crio.service,podman.service,cri-dockerd/crio/podmanbinaries all present.minikube start --driver=vfkit|qemu2|virtualboxall produce aReadycontrol-plane node on the newly-built ISO.make minikube-iso-x86_64unaffected (patch only activates on aarch64 hosts).Upstream CI (x86_64 builders) is unaffected.
Related