crud-web-apps can't be built for ppc64le

[kimwnasptd]

/kind bug

What steps did you take and what happened:
Right now none of the crud-web-apps can be build for the ppc64le arch.

To reproduce you can try the following:

cd components/crud-web-apps/jupyter
ARCH=linux/ppc64le make docker-build-multi-arch

cd ../ && docker buildx build --load --platform linux/ppc64le --tag jupyter-web-app:v1.5.0-rc.0-336-g1e02d1ec -f jupyter/Dockerfile .
[+] Building 1.4s (16/37)
 => [internal] load build definition from Dockerfile                                                                                                                                  0.0s
 => => transferring dockerfile: 32B                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                     0.0s
 => => transferring context: 34B                                                                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/python:3.10-slim                                                                                                                   1.1s
 => [internal] load metadata for docker.io/library/node:12-buster-slim                                                                                                                1.1s
 => [frontend  1/12] FROM docker.io/library/node:12-buster-slim@sha256:680bdb99eb94b7bf4329247557da66db03cf584bb6f273a8de823ff1b588c3b6                                               0.0s
 => [internal] load build context                                                                                                                                                     0.1s
 => => transferring context: 61.13kB                                                                                                                                                  0.0s
 => CANCELED [backend-kubeflow-wheel 1/4] FROM docker.io/library/python:3.10-slim@sha256:84ecab4ecf38604b04bfb8623d9159b863afe154f74c41350c619fa2470e0ef0                             0.6s
 => => resolve docker.io/library/python:3.10-slim@sha256:84ecab4ecf38604b04bfb8623d9159b863afe154f74c41350c619fa2470e0ef0                                                             0.1s
 => => sha256:84ecab4ecf38604b04bfb8623d9159b863afe154f74c41350c619fa2470e0ef0 1.65kB / 1.65kB                                                                                        0.0s
 => => sha256:16e9fa3ffd17ad6ddcd27ad7a71c556741040c593f65d17b5557a0d348db62fa 1.37kB / 1.37kB                                                                                        0.0s
 => => sha256:9ef7effe6fd9753f354cada13a58739d7e32e730c31236ee4d1ff2157d2b3536 6.92kB / 6.92kB                                                                                        0.0s
 => => sha256:ad974e0ada84177d519dec70d0d7d9ff6a4f468bc7306425feb8429bc276bd8a 0B / 33.12MB                                                                                           0.5s
 => => sha256:51b72fcffead54227e1098f60bdf819dc263e2477e583ea4ad0f95a0c3cbeec1 0B / 3.70MB                                                                                            0.5s
 => => sha256:8a329106867dcc1041ed5e044be782a72a9c8c3d75ab5e67852c78cc9243e37d 0B / 17.81MB                                                                                           0.5s
 => CACHED [frontend  2/12] WORKDIR /src                                                                                                                                              0.0s
 => CACHED [frontend-kubeflow-lib 3/9] COPY ./common/frontend/kubeflow-common-lib/package.json ./                                                                                     0.0s
 => CACHED [frontend-kubeflow-lib 4/9] COPY ./common/frontend/kubeflow-common-lib/package-lock.json ./                                                                                0.0s
 => CACHED [frontend  3/12] COPY ./jupyter/frontend/package.json ./                                                                                                                   0.0s
 => CACHED [frontend  4/12] COPY ./jupyter/frontend/package-lock.json ./                                                                                                              0.0s
 => CACHED [frontend  5/12] COPY ./jupyter/frontend/tsconfig.json ./                                                                                                                  0.0s
 => CACHED [frontend  6/12] COPY ./jupyter/frontend/tsconfig.app.json ./                                                                                                              0.0s
 => ERROR [frontend-kubeflow-lib 5/9] RUN npm ci                                                                                                                                      0.3s
 => [frontend  7/12] COPY ./jupyter/frontend/tsconfig.spec.json ./                                                                                                                    0.2s
 => [frontend  8/12] COPY ./jupyter/frontend/angular.json ./                                                                                                                          0.1s
 => CANCELED [frontend  9/12] COPY ./jupyter/frontend/src ./src                                                                                                                       0.2s
------
 > [frontend-kubeflow-lib 5/9] RUN npm ci:
#0 0.261 exec /bin/sh: exec format error
------
ERROR: failed to solve: executor failed running [/bin/sh -c npm ci]: exit code: 1
make: *** [Makefile:15: docker-build-multi-arch] Error 1

the same holds for both the volumes and the tensorboards PRs

What did you expect to happen:
The web apps should be build as expected.

Anything else you would like to add:
We didn't catch this earlier since

1. the CI was broken after Disabling multi-arch builds for PRs and enable them on Merge for JWA, NB Controller, AWH & TB Controller kubeflow#7171 and Disabling multi-arch builds for PRs and enable them on Merge for Profile Controller, Profile+KFAM, VWA &TWA kubeflow#7183, which should now be resolved with fix: remove superfluous space in docker builds kubeflow#7224
2. we test the multi arch builds only on merge

@lehrig do you have any more insights on why this error is only happening in ppc64le architectures?

We'll also need to unblock the CI asap, since the targeted date for the RC0 was for tomorrow.

[kimwnasptd]

/assign @lehrig

[kimwnasptd]

Small clarification, the above error log I showed was because I was in a different development branch. Updating the description. Trying to rebuild the jupyter app locally with latest master branch I'm getting the following:

 => ERROR [backend-kubeflow-wheel 4/4] RUN python3 setup.py bdist_wheel                 0.3s
 => CANCELED [frontend-kubeflow-lib 5/9] RUN npm ci                                     0.5s
 => [frontend  6/12] COPY ./jupyter/frontend/tsconfig.app.json ./                       0.1s
 => CANCELED [frontend  7/12] COPY ./jupyter/frontend/tsconfig.spec.json ./             0.2s
------
 > [backend-kubeflow-wheel 4/4] RUN python3 setup.py bdist_wheel:
#0 0.258 exec /bin/sh: exec format error
------
ERROR: failed to solve: executor failed running [/bin/sh -c python3 setup.py bdist_wheel]: exit code: 1
make: *** [Makefile:15: docker-build-multi-arch] Error 1

Trying to build the volumes app though results in the error in the first message:

 => ERROR [frontend-kubeflow-lib 5/9] RUN npm ci                                        0.4s
 => [frontend  6/12] COPY ./volumes/frontend/tsconfig.app.json ./                       0.1s
 => [frontend  7/12] COPY ./volumes/frontend/tsconfig.spec.json ./                      0.1s
 => CANCELED [frontend  8/12] COPY ./volumes/frontend/angular.json ./                   0.2s
------
 > [frontend-kubeflow-lib 5/9] RUN npm ci:
#0 0.353 exec /bin/sh: exec format error
------
ERROR: failed to solve: executor failed running [/bin/sh -c npm ci]: exit code: 1
make: *** [Makefile:14: docker-build-multi-arch] Error 1

[pranavpandit1]

@kimwnasptd : seems to be an issue with the latest gevent release .
I tried locally building jupyter with a change in setup.py -> "gevent <= 22.10.2". , it builds successfully.

as currently no version is specified against gevent in setup.py, it tries to pull the latest one. (Not sure as of now about the changes in the latest gevent release causing the exceptions)

[kimwnasptd]

@pranavpandit1 this indeed solves the issue reported with bdist_wheel but I'm still getting an issue with the /bin/sh -c npm ci step above.

Although the above error might have something to do with my setup (Docker installed from snap and ubuntu on ARM). But let's tackle the issues one by one. Can you send a PR to update the gevent with the above version limitation you described?

[pranavpandit1]

@kimwnasptd : PR is ready to get merged with all successful checks.

[andreyvelich]

/transfer notebooks

[Sunidhi-Gaonkar1]

Hi @kimwnasptd , I followed the same steps mentioned above and was able to build the image successfully on Power. Attaching the logs below. Please confirm from your side.Thank you.
Jupyter_logs.txt