bumping golang 1.25 failure when building using "kube-cross image"

[medyagh]

When trying to bump the golang from 1.24 to 1.25
#21942
the build fails in jenkins (it builds it inside a Docker using the Kube Cross image

# NOTE: "latest" as of 2021-02-06. kube-cross images aren't updated as often as Kubernetes
# https://github.com/kubernetes/kubernetes/blob/master/build/build-image/cross/VERSION

# BUILD_IMAGE 	?= registry.k8s.io/build-image/kube-cross:$(GO_K8S_VERSION_PREFIX)-go$(GO_VERSION)-bullseye.0

I noticed build fails in Jenkins (which runs in docker) "MINIKUBE_BUILD_IN_DOCKER" and also Functiona_Verified job also fails (builds minikube in docker) but "Functional" workflow passes (builds it on Ubuntu Baremetal

Functional_Verified Workflow Build (builds in Docker

minikube/.github/workflows/functional_verified.yml

Line 47 in bac7021

MINIKUBE_BUILD_IN_DOCKER=y make e2e-linux-arm64

MINIKUBE_BUILD_IN_DOCKER=y make e2e-linux-arm64

Functional Normal build passes (runs on baremetal)

minikube/.github/workflows/functional_test.yml

Line 46 in bac7021

make e2e-linux-amd64 e2e-darwin-amd64

We need to find out why new KubeCorss Image fails and how to fix it, because on Jenkins we should not rely on building on host

Logs From Jenkins build (trimmed)

12:35:43 Unable to find image 'registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0' locally
12:35:43 v1.35.0-go1.25.3-bullseye.0: Pulling from build-image/kube-cross
12:35:43 3adbbc83e99d: Waiting
12:37:12 a949bcef8c2a: Pull complete
12:37:12 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:12 Status: Downloaded newer image for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Status: Downloaded newer image for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:12 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:12 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:13 Digest: sha256:2d42755343ada0f56e9d381916edf5d2dbd00050643d4f558b9bcd891d3bdc81
12:37:13 Status: Image is up to date for registry.k8s.io/build-image/kube-cross:v1.35.0-go1.25.3-bullseye.0
12:37:13 warning: both GOPATH and GOROOT are the same directory (/usr/local/go); see https://go.dev/wiki/InstallTroubleshooting
12:37:13 warning: both GOPATH and GOROOT are the same directory (/usr/local/go); see https://go.dev/wiki/InstallTroubleshooting
12:37:13 warning: both GOPATH and GOROOT are the same directory (/usr/local/go); see https://go.dev/wiki/InstallTroubleshooting
# Get Repeated hunddreds of times...

12:37:13 GOOS="linux" GOARCH="s390x"  \
12:37:13 go build -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0-1763503576-21924 -X k8s.io/minikube/pkg/version.gitCommitID="2d19d546682a9aac70700f5e2d85cb8e7ae0d67a" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-linux-s390x k8s.io/minikube/cmd/minikube

12:37:13 GOOS="windows" GOARCH="amd64"  \
12:37:13 go build -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0-1763503576-21924 -X k8s.io/minikube/pkg/version.gitCommitID="2d19d546682a9aac70700f5e2d85cb8e7ae0d67a" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-windows-amd64 k8s.io/minikube/cmd/minikube
12:37:13 warning: both GOPATH and GOROOT are the same directory (/usr/local/go); see https://go.dev/wiki/InstallTroubleshooting
# ...

12:37:13 go: downloading gopkg.in/yaml.v2 v2.4.0
12:37:13 go: downloading golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c
12:37:13 warning: both GOPATH and GOROOT are the same directory (/usr/local/go); see https://go.dev/wiki/InstallTroubleshooting
12:37:13 GOOS="linux" GOARCH="386"  \
12:37:13 go build -tags "libvirt_dlopen" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.37.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.37.0-1763503576-21924 -X k8s.io/minikube/pkg/version.gitCommitID="2d19d546682a9aac70700f5e2d85cb8e7ae0d67a" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v5" -a -o out/minikube-linux-386 k8s.io/minikube/cmd/minikube
12:37:13 go: downloading github.com/spf13/pflag v1.0.9
12:37:13 go: downloading github.com/distribution/reference v0.6.0
12:37:13 go: downloading github.com/docker/docker v28.3.3+incompatible
12:37:13 go: downloading github.com/docker/go-units v0.5.0
#...


12:37:14 go: writing go.mod cache: mkdir /usr/local/go/pkg/mod/cache/download/cloud.google.com: permission denied
12:37:14 go: writing go.mod cache: mkdir /usr/local/go/pkg/mod/cache/download/cloud.google.com: permission denied
# .... (thousands of these line)

[medyagh]

COPilot suggested adding

-e GOPATH=/go -e GOMODCACHE=/app/.modcache

to our docker container that builds the image, I will give that a try

[BenTheElder]

In Kubernetes/Kubernetes we force go to store everything under tempdir within the repo (_output/...), which when building with docker will be on a volume writable by the container (today this means very roughly -v ${REPO_ROOT}:/go/src/k8s.io/kubernetes), GOPATH is one indirect way to influence this.

The most reliable solution will be to both set GOPATH (or GOMODCACHE/GOCACHE/...) and to make sure you mount a bind mount, volume, or tmpfs to that location.