arm64/aarch64 isn't supported

[saulshanabrook]

Bug description

On my computer I am unable to use repo2docker to build a local directory, even if it's empty. I am not sure why.

Expected behaviour

I expected jupyter-repo2docker . to build locally.

Actual behaviour

It fails:

$ jupyter-repo2docker .
Picked Local content provider.
Using local repo ..
Using PythonBuildPack builder
Step 1/48 : FROM buildpack-deps:bionic
 ---> 98fc26e1509f
Step 2/48 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 1ddee175e87f
Step 3/48 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 4888afea1faf
Step 4/48 : RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
 ---> Using cache
 ---> 560885b9842a
Step 5/48 : ENV LC_ALL en_US.UTF-8
 ---> Using cache
 ---> b20252556587
Step 6/48 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> 555a9faba040
Step 7/48 : ENV LANGUAGE en_US.UTF-8
 ---> Using cache
 ---> bac9774badef
Step 8/48 : ENV SHELL /bin/bash
 ---> Using cache
 ---> c32aa3e88981
Step 9/48 : ARG NB_USER
 ---> Using cache
 ---> ebc588c6509a
Step 10/48 : ARG NB_UID
 ---> Using cache
 ---> 0d37f8730abc
Step 11/48 : ENV USER ${NB_USER}
 ---> Using cache
 ---> fea8c4d34e03
Step 12/48 : ENV HOME /home/${NB_USER}
 ---> Using cache
 ---> ff2316ba9655
Step 13/48 : RUN groupadd         --gid ${NB_UID}         ${NB_USER} &&     useradd         --comment "Default user"         --create-home         --gid ${NB_UID}         --no-log-init         --shell /bin/bash         --uid ${NB_UID}         ${NB_USER}
 ---> Using cache
 ---> 504d5ac77f2a
Step 14/48 : RUN wget --quiet -O - https://deb.nodesource.com/gpgkey/nodesource.gpg.key |  apt-key add - &&     DISTRO="bionic" &&     echo "deb https://deb.nodesource.com/node_14.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list &&     echo "deb-src https://deb.nodesource.com/node_14.x $DISTRO main" >> /etc/apt/sources.list.d/nodesource.list
 ---> Using cache
 ---> d994f49c133d
Step 15/48 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        less        nodejs        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 9eb32056c343
Step 16/48 : EXPOSE 8888
 ---> Using cache
 ---> ca77d6d4c24a
Step 17/48 : ENV APP_BASE /srv
 ---> Using cache
 ---> 20995168e0d1
Step 18/48 : ENV NPM_DIR ${APP_BASE}/npm
 ---> Using cache
 ---> 788da69757a7
Step 19/48 : ENV NPM_CONFIG_GLOBALCONFIG ${NPM_DIR}/npmrc
 ---> Using cache
 ---> a5a1601fee12
Step 20/48 : ENV CONDA_DIR ${APP_BASE}/conda
 ---> Using cache
 ---> 5f2c97f735a5
Step 21/48 : ENV NB_PYTHON_PREFIX ${CONDA_DIR}/envs/notebook
 ---> Using cache
 ---> 28b20d442c48
Step 22/48 : ENV NB_ENVIRONMENT_FILE /tmp/env/environment.lock
 ---> Using cache
 ---> e72759ea9337
Step 23/48 : ENV KERNEL_PYTHON_PREFIX ${NB_PYTHON_PREFIX}
 ---> Using cache
 ---> 3adaed635bb7
Step 24/48 : ENV PATH ${NB_PYTHON_PREFIX}/bin:${CONDA_DIR}/bin:${NPM_DIR}/bin:${PATH}
 ---> Using cache
 ---> 1e8f85a6301f
Step 25/48 : COPY --chown=501:501 build_script_files/-2fopt-2fhomebrew-2fcaskroom-2fminiconda-2fbase-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-91d161 /etc/profile.d/activate-conda.sh
 ---> Using cache
 ---> ab3cb148fe1e
Step 26/48 : COPY --chown=501:501 build_script_files/-2fopt-2fhomebrew-2fcaskroom-2fminiconda-2fbase-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2fenvironment-2epy-2d3-2e7-2elock-d00ccd /tmp/env/environment.lock
 ---> Using cache
 ---> 6ab76d080211
Step 27/48 : COPY --chown=501:501 build_script_files/-2fopt-2fhomebrew-2fcaskroom-2fminiconda-2fbase-2flib-2fpython3-2e8-2fsite-2dpackages-2frepo2docker-2fbuildpacks-2fconda-2finstall-2dminiforge-2ebash-ec0cb5 /tmp/install-miniforge.bash
 ---> Using cache
 ---> 047aef216f89
Step 28/48 : RUN mkdir -p ${NPM_DIR} && chown -R ${NB_USER}:${NB_USER} ${NPM_DIR}
 ---> Using cache
 ---> 4ce6e6ed3739
Step 29/48 : USER ${NB_USER}
 ---> Using cache
 ---> f7b89ca9c16b
Step 30/48 : RUN npm config --global set prefix ${NPM_DIR}
 ---> Using cache
 ---> 5f91a5ac3cff
Step 31/48 : USER root
 ---> Using cache
 ---> 88e5735d912c
Step 32/48 : RUN TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-miniforge.bash' && rm -rf /tmp/install-miniforge.bash /tmp/env
 ---> Running in a9e4741c585a
++ dirname /tmp/install-miniforge.bash
+ cd /tmp
+ MINIFORGE_VERSION=4.9.2-2
+ MAMBA_VERSION=0.7.4
+ SHA256SUM=7a7bfaff87680298304a97ba69bcf92f66c810995a7155a2918b99fafb8ca1dc
+ URL=https://github.com/conda-forge/miniforge/releases/download/4.9.2-2/Mambaforge-4.9.2-2-Linux-x86_64.sh
+ INSTALLER_PATH=/tmp/miniforge-installer.sh
+ unset HOME
+ wget --quiet https://github.com/conda-forge/miniforge/releases/download/4.9.2-2/Mambaforge-4.9.2-2-Linux-x86_64.sh -O /tmp/miniforge-installer.sh
time: 12.992
+ chmod +x /tmp/miniforge-installer.sh
+ echo '7a7bfaff87680298304a97ba69bcf92f66c810995a7155a2918b99fafb8ca1dc  /tmp/miniforge-installer.sh'
+ sha256sum --quiet -c -
+ bash /tmp/miniforge-installer.sh -b -p /srv/conda
PREFIX=/srv/conda
Unpacking payload ...
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
time: 0.257
time: 13.550
Removing intermediate container a9e4741c585a
The command '/bin/sh -c TIMEFORMAT='time: %3R' bash -c 'time /tmp/install-miniforge.bash' && rm -rf /tmp/install-miniforge.bash /tmp/env' returned a non-zero code: 1

How to reproduce

Create a new directory and run repo2docker on it.

Your personal set up

• OS: macOS 12.1 (21C52) Apple M1 Pro
• Docker version: docker version 20.10.12
• repo2docker version repo2docker --version 2021.08.0

$ docker version
Client:
 Cloud integration: v1.0.22
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:46:56 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:07 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[manics]

Unfortunately arm64 isn't currently supported. I've retitled this issue to make it clear to anyone skimming through the issues.

[saulshanabrook]

@manics Thank you! I thought it was some apple silicon thing...

Are there any workarounds for running it locally on a new mac? Or would the recommendation be to provision some remote linux machine and try running on that?

[yuvipanda]

@saulshanabrook #1112 (comment) is the current workaround.

[saulshanabrook]

@yuvipanda Sweet, thank you!

[manics]

I looked a bit more into what supporting arm64 involves. The freeze.py script works for linux-aarch64 Python 3.* but not Python 2.7.

See https://github.com/jupyterhub/repo2docker/compare/main...manics:freeze-aarch64?expand=1

In general we need to decide what "supporting" arm64 means. Is a "best effort" enough, which implies only supporting buildpacks and versions that work with minimal changes? This should be easier, but may cause confusion for people who expect repo2docker to work the same. On the other hand we have no control over whether someone's repo requires a package that's not built for arm64, or is pinned to an arch specific version, so there'll be inconsistencies anyway.

Or do we invest time (assuming someone has time!) in getting repo2docker to work as well as possible for the base dependencies installed directly by repo2docker, so any failures are purely down to the repo being built?

[mattwigway]

The linked workaround doesn't work for me, on macOS 12.2 on an M1. On version 2022.02 installed from pip, I get the error mentioned above. On repo2docker main, I get this error:

Picked Local content provider.
Using local repo ./.
Using JuliaProjectTomlBuildPack builder
Step 1/60 : FROM buildpack-deps:bionic
 ---> 3256ddf0c93f
Step 2/60 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> 58ae9404d4a7
Step 3/60 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends locales > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 2cbca6ec34ae
Step 4/60 : RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
 ---> Using cache
 ---> 10360eecacc2
Step 5/60 : ENV LC_ALL en_US.UTF-8
 ---> Using cache
 ---> 97c415331f24
Step 6/60 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> 75fd2a768176
Step 7/60 : ENV LANGUAGE en_US.UTF-8
 ---> Using cache
 ---> 2b986fba9e92
Step 8/60 : ENV SHELL /bin/bash
 ---> Using cache
 ---> 60cbaa77acfb
Step 9/60 : ARG NB_USER
 ---> Using cache
 ---> 77ed50ab07ae
Step 10/60 : ARG NB_UID
 ---> Using cache
 ---> a8f859145263
Step 11/60 : ENV USER ${NB_USER}
 ---> Using cache
 ---> 64976df1bdcf
Step 12/60 : ENV HOME /home/${NB_USER}
 ---> Using cache
 ---> d9e95195884f
Step 13/60 : RUN groupadd         --gid ${NB_UID}         ${NB_USER} &&     useradd         --comment "Default user"         --create-home         --gid ${NB_UID}         --no-log-init         --shell /bin/bash         --uid ${NB_UID}         ${NB_USER}
 ---> Using cache
 ---> 9c089b6a7b7a
Step 14/60 : RUN apt-get -qq update &&     apt-get -qq install --yes --no-install-recommends        less        unzip        > /dev/null &&     apt-get -qq purge &&     apt-get -qq clean &&     rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> 6362e2ff99f7
Step 15/60 : EXPOSE 8888
 ---> Using cache
 ---> 3bfa3030a172
Step 16/60 : ENV APP_BASE /srv
 ---> Using cache
 ---> df691ce4a8ab
Step 17/60 : ENV CONDA_DIR ${APP_BASE}/conda
 ---> Using cache
 ---> 6b8be1509aeb
Step 18/60 : ENV NB_PYTHON_PREFIX ${CONDA_DIR}/envs/notebook
 ---> Using cache
 ---> 1bcd688bc14d
Step 19/60 : ENV NPM_DIR ${APP_BASE}/npm
 ---> Using cache
 ---> 9806a2b121d8
Step 20/60 : ENV NPM_CONFIG_GLOBALCONFIG ${NPM_DIR}/npmrc
 ---> Using cache
 ---> 517bc2b74553
Step 21/60 : ENV NB_ENVIRONMENT_FILE /tmp/env/environment.lock
 ---> Using cache
 ---> edd823fd6b53
Step 22/60 : ENV MAMBA_ROOT_PREFIX ${CONDA_DIR}
 ---> Using cache
 ---> c892fd95481d
Step 23/60 : ENV MAMBA_EXE ${CONDA_DIR}/bin/mamba
 ---> Using cache
 ---> 2569bc59a1be
Step 24/60 : ENV KERNEL_PYTHON_PREFIX ${NB_PYTHON_PREFIX}
 ---> Using cache
 ---> 3c4ffb6185c4
Step 25/60 : ENV JULIA_PATH ${APP_BASE}/julia
 ---> Using cache
 ---> 2fb085c45677
Step 26/60 : ENV JULIA_DEPOT_PATH ${JULIA_PATH}/pkg
 ---> Using cache
 ---> 5580c82c0cf3
Step 27/60 : ENV JULIA_VERSION 1.7.1
 ---> Using cache
 ---> 7df42bf8d935
Step 28/60 : ENV JUPYTER ${NB_PYTHON_PREFIX}/bin/jupyter
 ---> Using cache
 ---> f86590af1935
Step 29/60 : ENV JUPYTER_DATA_DIR ${NB_PYTHON_PREFIX}/share/jupyter
 ---> Using cache
 ---> 5d6ddad6a743
Step 30/60 : ENV PATH ${NB_PYTHON_PREFIX}/bin:${CONDA_DIR}/bin:${NPM_DIR}/bin:${JULIA_PATH}/bin:${PATH}
 ---> Using cache
 ---> ef5decb6bebd
Step 31/60 : COPY --chown=501:501 build_script_files/-2fusers-2fmwbc-2fgit-2frepo2docker-2frepo2docker-2fbuildpacks-2fconda-2factivate-2dconda-2esh-d53b3e /etc/profile.d/activate-conda.sh
failed to get destination image "sha256:ef5decb6bebd0ffde318b34923270f1215e07d3f6f86f8aed046aeaf6d0f8017": image with reference sha256:ef5decb6bebd0ffde318b34923270f1215e07d3f6f86f8aed046aeaf6d0f8017 was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64/v8

I have tried using traitlets.config.json as well as adding --Repo2Docker.extra_build_kwargs=platform=linux/amd64 on the command line.

Your personal setup

OS: macOS 12.2 on Apple M1
docker version 20.10.2 (Docker Desktop)
repo2docker version: see above

docker version output:

Client:
 Cloud integration: v1.0.22
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:46:56 2021
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:07 2021
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[mattwigway]

Oh, and I should add I am able to successfully build other Docker images using docker build --platform linux/amd64

[mattwigway]

Deleting all of the cached images from the Images tab in Docker Desktop solved this problem for me, for both 2022.02 and main (be careful if you have any images you want to keep). I'm guessing somehow there was a cached build step for the wrong platform messing things up.

[yuvipanda]

So to actually support building ARM images, we need to parameterize our mamba installs - is that all we need to do?

[manics]

@yuvipanda you'll also need to pass the platform to the freeze script
https://github.com/jupyterhub/repo2docker/blob/main/repo2docker/buildpacks/conda/freeze.py#L129
and generate two versions of the lock files, one for each supported architecture.

[yuvipanda]

@manics #1180 :D yeah, have that running locally.