Cannot build image natively from macOS

[akutz]

What steps did you take and what happened:
Run make on macOS fails because the getbins target fails when executing the Kubernetes get-kube-binaries.sh script due to how the K8s client bits are auto-detected and cannot be overridden. For several versions now Kubernetes no longer publishes the Darwin kubectl with builds. Even if it did, it would be the incorrect version to use when building the image.

$ make
mkdir -p _cache/v1.13.4
curl -SsL http://gcsweb.k8s.io/gcs/kubernetes-release/release/v1.13.4/kubernetes.tar.gz | tar -C _cache/v1.13.4 -xz
cd _cache/v1.13.4 && KUBE_VERSION="v1.13.4" \
                                  KUBERNETES_DOWNLOAD_TESTS=true \
				  KUBERNETES_SKIP_CONFIRM=true ./kubernetes/cluster/get-kube-binaries.sh
Kubernetes release: v1.13.4
Server: linux/amd64  (to override, set KUBERNETES_SERVER_ARCH)
Client: darwin/amd64  (autodetected)

Will download kubernetes-server-linux-amd64.tar.gz from https://dl.k8s.io/v1.13.4
Will download and extract kubernetes-client-darwin-amd64.tar.gz from https://dl.k8s.io/v1.13.4
Will download and extract kubernetes-test.tar.gz from https://dl.k8s.io/v1.13.4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    756      0 --:--:-- --:--:-- --:--:--   759
100  398M  100  398M    0     0  13.3M      0  0:00:29  0:00:29 --:--:-- 12.7M

md5sum(kubernetes-server-linux-amd64.tar.gz)=2ea25eea39b73fc8deb07cc6e86c13ed
sha1sum(kubernetes-server-linux-amd64.tar.gz)=3308ba88800a0c88f6cfcac8ca56c636ab278253

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    775      0 --:--:-- --:--:-- --:--:--   777
100 11.9M  100 11.9M    0     0  7571k      0  0:00:01  0:00:01 --:--:-- 10.8M

md5sum(kubernetes-client-darwin-amd64.tar.gz)=eeabafb8c33bae8374e1c12c1d70947b
sha1sum(kubernetes-client-darwin-amd64.tar.gz)=30e7637cf6c14950be161d4038eb45c3c4eebbdf

Extracting /Users/akutz/Projects/kube-conformance/_cache/v1.13.4/kubernetes/client/kubernetes-client-darwin-amd64.tar.gz into /Users/akutz/Projects/kube-conformance/_cache/v1.13.4/kubernetes/platforms/darwin/amd64
Add '/Users/akutz/Projects/kube-conformance/_cache/v1.13.4/kubernetes/client/bin' to your PATH to use newly-installed binaries.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    753      0 --:--:-- --:--:-- --:--:--   755
100 1176M  100 1176M    0     0  13.3M      0  0:01:27  0:01:27 --:--:-- 14.3M

md5sum(kubernetes-test.tar.gz)=04e3199f668e998b146ab0e428bd82f5
sha1sum(kubernetes-test.tar.gz)=f3c8f1160fff9455816bd3a3bbd9bdbe61e25252

Extracting kubernetes-test.tar.gz into /Users/akutz/Projects/kube-conformance/_cache/v1.13.4/kubernetes
mv _cache/v1.13.4/kubernetes/cluster ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/e2e.test ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/ginkgo ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/kubectl ./
mv: rename _cache/v1.13.4/kubernetes/platforms/linux/amd64/kubectl to ./kubectl: No such file or directory
make: *** [Makefile:50: _cache/.getbins.v1.13.4.timestamp] Error 1

What did you expect to happen:
A new kube-conformance image should be built.

Anything else you would like to add:
A workaround is to use Docker like so:

$ docker run -it --rm \
  -v "$(pwd)":/kube-conformance -w /kube-conformance golang \
  make getbins && make

The above command executes the getbins target inside of a Linux container and then executes the default target, containers, natively on the host system.

Please note, the golang image was used simply because it includes the GNU developer toolchain and the make command. Any image that has make could be used in place of golang in the above command.

The above command succeeds like so:

$ docker run -it --rm \
>   -v "$(pwd)":/kube-conformance -w /kube-conformance golang \
>   make getbins && make
mkdir -p _cache/v1.13.4
curl -SsL http://gcsweb.k8s.io/gcs/kubernetes-release/release/v1.13.4/kubernetes.tar.gz | tar -C _cache/v1.13.4 -xz
cd _cache/v1.13.4 && KUBE_VERSION="v1.13.4" \
                                  KUBERNETES_DOWNLOAD_TESTS=true \
				  KUBERNETES_SKIP_CONFIRM=true ./kubernetes/cluster/get-kube-binaries.sh
Kubernetes release: v1.13.4
Server: linux/amd64  (to override, set KUBERNETES_SERVER_ARCH)
Client: linux/amd64  (autodetected)

Will download kubernetes-server-linux-amd64.tar.gz from https://dl.k8s.io/v1.13.4
Will download and extract kubernetes-client-linux-amd64.tar.gz from https://dl.k8s.io/v1.13.4
Will download and extract kubernetes-test.tar.gz from https://dl.k8s.io/v1.13.4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    744      0 --:--:-- --:--:-- --:--:--   748
100  398M  100  398M    0     0  12.5M      0  0:00:31  0:00:31 --:--:-- 13.1M

md5sum(kubernetes-server-linux-amd64.tar.gz)=2ea25eea39b73fc8deb07cc6e86c13ed
sha1sum(kubernetes-server-linux-amd64.tar.gz)=3308ba88800a0c88f6cfcac8ca56c636ab278253

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    732      0 --:--:-- --:--:-- --:--:--   735
100 11.4M  100 11.4M    0     0  6899k      0  0:00:01  0:00:01 --:--:-- 8993k

md5sum(kubernetes-client-linux-amd64.tar.gz)=280505c2aff4b2cd4e2eeb97691c28a5
sha1sum(kubernetes-client-linux-amd64.tar.gz)=5a7fcbec907fda0ffad8f8501779f938d40eaf39

Extracting /kube-conformance/_cache/v1.13.4/kubernetes/client/kubernetes-client-linux-amd64.tar.gz into /kube-conformance/_cache/v1.13.4/kubernetes/platforms/linux/amd64
Add '/kube-conformance/_cache/v1.13.4/kubernetes/client/bin' to your PATH to use newly-installed binaries.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   161  100   161    0     0    757      0 --:--:-- --:--:-- --:--:--   755
100 1176M  100 1176M    0     0  12.7M      0  0:01:32  0:01:32 --:--:-- 11.7M

md5sum(kubernetes-test.tar.gz)=04e3199f668e998b146ab0e428bd82f5
sha1sum(kubernetes-test.tar.gz)=f3c8f1160fff9455816bd3a3bbd9bdbe61e25252

Extracting kubernetes-test.tar.gz into /kube-conformance/_cache/v1.13.4/kubernetes
mv _cache/v1.13.4/kubernetes/cluster ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/e2e.test ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/ginkgo ./
mv _cache/v1.13.4/kubernetes/platforms/linux/amd64/kubectl ./
touch _cache/.getbins.v1.13.4.timestamp
docker build -t gcr.io/heptio-images/kube-conformance:v1.13 \
                -t gcr.io/heptio-images/kube-conformance:v1.13.4 .
Sending build context to Docker daemon  172.1MB
Step 1/15 : FROM buildpack-deps:jessie-scm
jessie-scm: Pulling from library/buildpack-deps
ff4229790957: Pull complete 
de1f85cbc021: Pull complete 
e1b6a4984ab4: Pull complete 
Digest: sha256:851c8d0831af5440db4e97cd4d534fdb3d3e609f4d9ca9d04d84e7a98215da74
Status: Downloaded newer image for buildpack-deps:jessie-scm
 ---> 9627cf40087b
Step 2/15 : MAINTAINER Timothy St. Clair "tstclair@heptio.com"
 ---> Running in 471bf478a947
Removing intermediate container 471bf478a947
 ---> 54afaec64fe5
Step 3/15 : RUN apt-get update && apt-get -y --no-install-recommends install     ca-certificates     && rm -rf /var/cache/apt/*     && rm -rf /var/lib/apt/lists/*
 ---> Running in fbc832c25b3a
Ign http://deb.debian.org jessie InRelease
Get:1 http://deb.debian.org jessie-updates InRelease [145 kB]
Get:2 http://security.debian.org jessie/updates InRelease [44.9 kB]
Get:3 http://deb.debian.org jessie Release.gpg [2420 B]
Get:4 http://deb.debian.org jessie Release [148 kB]
Get:5 http://deb.debian.org jessie-updates/main amd64 Packages [23.0 kB]
Get:6 http://deb.debian.org jessie/main amd64 Packages [9098 kB]
Get:7 http://security.debian.org jessie/updates/main amd64 Packages [818 kB]
Fetched 10.3 MB in 7s (1319 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
ca-certificates is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.
Removing intermediate container fbc832c25b3a
 ---> 5b2c6709594c
Step 4/15 : COPY ginkgo /usr/local/bin/
 ---> 4974ce7bd87e
Step 5/15 : COPY e2e.test /usr/local/bin/
 ---> 0d5ea8d41602
Step 6/15 : COPY kubectl /usr/local/bin/
 ---> e6c4667c232e
Step 7/15 : COPY run_e2e.sh /run_e2e.sh
 ---> 0773b82920b7
Step 8/15 : COPY cluster /kubernetes/cluster
 ---> 738def5cdec5
Step 9/15 : WORKDIR /usr/local/bin
 ---> Running in de67378a31d9
Removing intermediate container de67378a31d9
 ---> 848e7f3da8a0
Step 10/15 : ENV E2E_FOCUS="\[Conformance\]"
 ---> Running in 74862fdd172a
Removing intermediate container 74862fdd172a
 ---> aa3ae9bc2d93
Step 11/15 : ENV E2E_SKIP="Alpha|\[(Disruptive|Feature:[^\]]+|Flaky)\]"
 ---> Running in d947b129c014
Removing intermediate container d947b129c014
 ---> f8c91dc16755
Step 12/15 : ENV E2E_PROVIDER="local"
 ---> Running in 821607cf4d8e
Removing intermediate container 821607cf4d8e
 ---> caac59a4a54c
Step 13/15 : ENV E2E_PARALLEL="1"
 ---> Running in d888514fefd1
Removing intermediate container d888514fefd1
 ---> 0d29f2b78cea
Step 14/15 : ENV RESULTS_DIR="/tmp/results"
 ---> Running in b2b51e577c74
Removing intermediate container b2b51e577c74
 ---> 35e85d9c6d26
Step 15/15 : CMD [ "/bin/bash", "-c", "/run_e2e.sh" ]
 ---> Running in f74f8fbca4a0
Removing intermediate container f74f8fbca4a0
 ---> 3f3fe64cd6a7
Successfully built 3f3fe64cd6a7
Successfully tagged gcr.io/heptio-images/kube-conformance:v1.13
Successfully tagged gcr.io/heptio-images/kube-conformance:v1.13.4
if [ "1.13" = "1.13" ]; then \
  docker tag gcr.io/heptio-images/kube-conformance:v1.13 gcr.io/heptio-images/kube-conformance:latest; \
fi

Environment:

• Sonobuoy version: (use sonobuoy version): NA
• Kubernetes version: (use kubectl version): NA
• Kubernetes installer & version: NA
• Cloud provider or hardware configuration: NA
• OS (e.g. from /etc/os-release): NA

[timothysc]

We "should" switch to using the upstream k8s container asap.

/assign @johnSchnake @stevesloka

[stevesloka]

I can look into this one.

[akutz]

Hi @stevesloka,

I'm also happy to look into things. One big happy family, right? :) FWIW, the problem is very straight-forward. It's one of the very first issues I encountered when joining the K8s community -- the Darwin client bits have not been published to the public GCS bucket in a very long time.

Aside from that, the Darwin client should never be used with this regardless. Since the K8s script doesn't allow for overriding the client OS/arch, you have to run it in a Linux container or on a Linux host.

[johnSchnake]

PR is up for switching to the upstream already. Phasing it in but if it merges I don’t anticipate needing to release this again and we can archive it: see vmware-tanzu/sonobuoy#617

[akutz]

Hi @stevesloka,

Please see kubernetes/kubernetes#74889 for an upstream change that will solve this PR (@johnSchnake's comment notwithstanding).