ci instability: image build may fail with segfault during bash execution

[jgehrcke]

Example: https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17234543930/job/48896504715#step:7:4828

 > [linux/amd64->arm64 build  7/14] RUN cd /bashbuild/bash-static/releases && ./bash*-static --version:
0.168 Segmentation fault (core dumped)

This appears to be non-deterministic in the sense that a simple re-run was observed to resolve this. Let's observe frequency of occurrence and debug&fix as needed.

[jgehrcke]

another occurrence: https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17235430869/job/48899375015?pr=487#step:7:3954

[jgehrcke]

Talked to Kevin about this. The current bash build first builds musl-gcc and then uses that to continue the build (code). In case of the amd64->arm64 scenario that means that we create a musl-gcc binary built for arm64, and then we keep running that on amd64. In that case, the compiler executes in an emulated environment. Instead, we probably want proper cross-compilation.

Related:
https://wiki.musl-libc.org/getting-started.html#Building-a-cross-compiler-targeting-musl-libc
https://stackoverflow.com/questions/64230710/how-does-musls-gcc-wrapper-differ-from-musls-cross-compiler

On my machine, out of curiosity, I just built a cross-compilation toolchain with the help of musl-cross-make; resulting in those binaries which should be all we need to make a static bash build against musl for aarch64, built on amd64:

aarch64-linux-musl-addr2line   aarch64-linux-musl-elfedit     aarch64-linux-musl-gcov        aarch64-linux-musl-objcopy
aarch64-linux-musl-ar          aarch64-linux-musl-g++         aarch64-linux-musl-gcov-dump   aarch64-linux-musl-objdump
aarch64-linux-musl-as          aarch64-linux-musl-gcc         aarch64-linux-musl-gcov-tool   aarch64-linux-musl-ranlib
aarch64-linux-musl-c++         aarch64-linux-musl-gcc-9.4.0   aarch64-linux-musl-gprof       aarch64-linux-musl-readelf
aarch64-linux-musl-cc          aarch64-linux-musl-gcc-ar      aarch64-linux-musl-ld          aarch64-linux-musl-size
aarch64-linux-musl-c++filt     aarch64-linux-musl-gcc-nm      aarch64-linux-musl-ld.bfd      aarch64-linux-musl-strings
aarch64-linux-musl-cpp         aarch64-linux-musl-gcc-ranlib  aarch64-linux-musl-nm          aarch64-linux-musl-strip

[jgehrcke]

failed again here: https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17296694142/job/49096911731?pr=505#step:7:3882

[jgehrcke]

another one, but I think this is a slightly different symptom:

#27 104.8 /bashbuild/bash-static/build/musl-install-1.2.5/bin/musl-gcc -c   -DHAVE_CONFIG_H -DSHELL  -I. -I..  -I.. -I../include -I../lib -I.    -Wno-error=implicit-function-declaration -Os -static -Wno-error=implicit-function-declaration -Os -static times.c || ( rm -f times.c ; exit 1 )
#27 104.9 make[1]: *** [Makefile:120: test.o] Segmentation fault (core dumped)
#27 104.9 make[1]: *** Waiting for unfinished jobs....

This time, this failed during build.
https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17297457001/job/49099535095?pr=506#step:7:4629

[jgehrcke]

Again, different occasion (this time during building trap.c):

209.2 /bashbuild/bash-static/build/musl-install-1.2.5/bin/musl-gcc -c   -DHAVE_CONFIG_H -DSHELL  -I. -I..  -I.. -I../include -I../lib -I.    -Wno-error=implicit-function-declaration -Os -static -Wno-error=implicit-function-declaration -Os -static trap.c || ( rm -f trap.c ; exit 1 )
209.3 make[1]: *** [Makefile:120: suspend.o] Segmentation fault (core dumped)
209.3 make[1]: *** Waiting for unfinished jobs....

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17447212172/job/49544785891?pr=433#step:8:4699

[jgehrcke]

All build logs above have in common: amd64->arm64 build.

The segfaults however may seemingly happen any moment in time, not tied to invoking a specific binary. Sometiems happened during execution of musl-gcc, other times during execution of bash.

We can conclude that the execution in the emulated environment is not stable. Maybe not even because the emulation isn't working, but because there isn't enough memory or so.

[jgehrcke]

another symptom variant:

#26 80.55 checking for suffix of executables... 
#26 80.57 checking whether we are cross compiling... configure: error: in `/bashbuild/bash-static/build/bash-5.2':
#26 80.68 configure: error: cannot run C compiled programs.
#26 80.68 If you meant to cross compile, use `--host'.
#26 80.68 See `config.log' for more details

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17647694593/job/50150385142?pr=558#step:8:3305

[elezar]

In other C-heavy projects I have had to use different tonistiigi/binfmt:qemu image versions.

In the CUDA samples for example I landed on the following version:

https://github.com/NVIDIA/k8s-samples/blob/6dc12f17159fc101dd5603dbf6a8dc978a2614b3/.github/workflows/image.yaml#L51

It could be that master is a moving target which adds to the instability.

[jgehrcke]

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/17944387630/job/51027578581#step:8:4657

115.5 ./mkbuiltins -D . declare.def
115.5 ./mkbuiltins -D . echo.def
115.6 make[1]: *** [Makefile:119: echo.o] Segmentation fault (core dumped)

[jgehrcke]

130.0 ./mkbuiltins -D . command.def
130.0 make[1]: *** [Makefile:119: caller.o] Segmentation fault (core dumped)

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/18281749344/job/52046720878?pr=638#step:8:4240

[jgehrcke]

129.4 ./mkbuiltins -D . caller.def
129.5 rm -f alias.c
129.6 make[1]: *** [Makefile:121: caller.o] Segmentation fault (core dumped)
129.6 make[1]: *** Waiting for unfinished jobs....

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/18369601736/job/52329798613

[jgehrcke]

296.0 ./mkbuiltins -D . shift.def
296.2 make[1]: *** [Makefile:119: shift.o] Segmentation fault (core dumped)
296.2 make[1]: *** Waiting for unfinished jobs....
296.4 rm -f set.c
296.5 rm -f setattr.c
296.5 rm -f read.c

https://github.com/NVIDIA/k8s-dra-driver-gpu/actions/runs/18556709003/job/52896166568#step:8:4729

Happens too often. We need to after all prioritize a fix or a workaround.

The binary built here rarely changes -- maybe we want to store it somewhere and then not rebuild every time.