document cross compile arm on linux

[gengjiawen]

I didn't find it in BUILDING.md, but it looks like this in our Jenkins CI.

cc @bnoordhuis @targos @nodejs/platform-arm

[rvagg]

This is for armv6 (ancient, tho used by unofficial-builds still) and armv7, primarily aimed at Raspberry Pi as a baseline:

https://github.com/nodejs/build/blob/364b8a9d0efe473b22cc04e6b4fa63a4b5e1d67a/ansible/roles/docker/files/cc-selector.sh is the script that figures out what it needs to compile and sets up the environment for compiling using environment variables which configure & Makefile use.

Cross compile tools used are here https://github.com/rvagg/rpi-newer-crosstools

[gengjiawen]

@rvagg Just to make sure, can this be done to in a docker image ?

[gengjiawen]

apt install lib32z1-dev gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf -y
CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ CC_host="gcc -m32" CXX_host="g++ -m32" ./configure --dest-cpu=arm --cross-compiling --dest-os=linux
make -j8

Now failed with

In file included from /usr/include/c++/9/memory:62,
                 from ../deps/icu-small/source/common/unicode/localpointer.h:45,
                 from ../deps/icu-small/source/common/cmemory.h:33,
                 from ../deps/icu-small/source/common/cstring.h:31,
                 from ../deps/icu-small/source/tools/toolutil/ucm.cpp:29:
/usr/include/c++/9/bits/stl_algobase.h:59:10: fatal error: bits/c++config.h: No such file or directory
   59 | #include <bits/c++config.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[1]: *** [tools/icu/icutools.host.mk:565: /root/node/out/Release/obj.host/icutools/deps/icu-small/source/tools/toolutil/ucm.o] Error 1
In file included from /usr/include/c++/9/stdlib.h:36,
                 from ../deps/icu-small/source/tools/toolutil/pkg_gencmn.cpp:11:
/usr/include/c++/9/cstdlib:41:10: fatal error: bits/c++config.h: No such file or directory
   41 | #include <bits/c++config.h>

[gengjiawen]

Below failed with new error.

apt install gcc-multilib g++-multilib -y
apt install lib32z1-dev gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf -y
ln -s /usr/include/asm-generic /usr/include/asm
CC=arm-linux-gnueabihf-gcc CXX=arm-linux-gnueabihf-g++ CC_host="gcc -m32" CXX_host="g++ -m32" ./configure --dest-cpu=arm --cross-compiling --dest-os=linux
make -j8

Failed with:

  g++ -m32 -o /root/node/out/Release/mksnapshot -pthread -rdynamic -m32  -Wl,--start-group /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/embedded-empty.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/embedded-file-writer.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-aix.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-base.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-generic.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-mac.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/embedded/platform-embedded-file-writer-win.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/mksnapshot.o /root/node/out/Release/obj.host/mksnapshot/deps/v8/src/snapshot/snapshot-empty.o /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_base_without_compiler.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_init.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_libbase.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_libplatform.a /root/node/out/Release/obj.host/tools/icu/libicutools.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_libsampler.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_zlib.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_compiler.a /root/node/out/Release/obj.host/tools/v8_gypfiles/libv8_initializers.a -latomic -ldl -lrt -Wl,--end-group
  touch 754ac2b9be6dad0d787f347675b94fc76885c25e.intermediate
  LD_LIBRARY_PATH=/root/node/out/Release/lib.host:/root/node/out/Release/lib.target:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; cd ../tools/v8_gypfiles; mkdir -p /root/node/out/Release/obj.target/v8_snapshot/geni; "/root/node/out/Release/mksnapshot" --turbo_instruction_scheduling "--target_os=linux" "--target_arch=arm" --startup_src "/root/node/out/Release/obj.target/v8_snapshot/geni/snapshot.cc" --embedded_variant Default --embedded_src "/root/node/out/Release/obj.target/v8_snapshot/geni/embedded.S" --no-native-code-counters


#
# Fatal error in , line 0
# unreachable code
#
#
#
#FailureMessage Object: 0xf7243fd4Aborted (core dumped)
make[1]: *** [tools/v8_gypfiles/v8_snapshot.target.mk:17: 754ac2b9be6dad0d787f347675b94fc76885c25e.intermediate] Error 134
make[1]: *** Waiting for unfinished jobs....
rm c6fc3b144c2c2c043e31af246edb4dbfb14c7156.intermediate b526cacdb009d9043833bcd06f98ac686fca6a10.intermediate 754ac2b9be6dad0d787f347675b94fc76885c25e.intermediate c96bbeb3f85793f3eac5a4ed576b0ecaa045b4fe.intermediate 90a906610746ab46c26081c72bb287fcde287c13.intermediate ecff0af5c31084ad57edfbfe7a9fc8e888d67a52.intermediate
make: *** [Makefile:104: node] Error 2

[rvagg]

Yep, here's the Dockerfile we use for all the cases https://github.com/nodejs/build/blob/master/ansible/roles/docker/templates/ubuntu1604_arm_cross.Dockerfile.j2

I don't think we touch the generic arm-linux-gnueabihf toolchain anywhere in our CI lineup because it doesn't quite cover the cases we want; or at least it hasn't in the past. It could be you're hitting a genuine bug because it's missing in our coverage, my guess would be something else about the configuration steps involved here though.

[gengjiawen]

@rvagg Thanks. That's great help. With the help of nodejs/TSC#931 (comment) by @mmarchini , I successfully build arm version. Though not running due to kernel.

My previous build failure reason stays unknown though. Maybe related to gcc (I am on ubuntu 20.04 with gcc 9.3) ?

[gengjiawen]

Update: it's actually in ./out/tools/v8_gypfiles/v8_base_without_compiler.host.mk. The reported url in console is not accurate.

Looks build failure will cause mk file auto delete. like I can't `tools/v8_gypfiles/v8_base_without_compiler.host.mk` in following build failure:

/tmp/cc2FUjAY.s: Assembler messages:
/tmp/cc2FUjAY.s:7: Error: invalid char '{' beginning operand 1 `{r3-r11'
/tmp/cc2FUjAY.s:8: Error: too many memory references for `mov'
/tmp/cc2FUjAY.s:9: Error: too many memory references for `mov'
/tmp/cc2FUjAY.s:10: Error: no such instruction: `blx r3'
/tmp/cc2FUjAY.s:11: Error: too many memory references for `add'
/tmp/cc2FUjAY.s:12: Error: invalid char '{' beginning operand 1 `{pc}'
tools/v8_gypfiles/v8_base_without_compiler.host.mk:710: recipe for target '/root/node/out/Release/obj.host/v8_base_without_compiler/deps/v8/src/heap/base/asm/arm/push_registers_asm.o' failed

How do I prevent this ?

[gengjiawen]

@rvagg Is there a plan for cc-selector.sh support ubuntu 20.04 and gcc-9 ?

[rvagg]

@gengjiawen we haven't directly discussed potential updates to this yet. It's designed to match https://github.com/nodejs/node/blob/master/BUILDING.md#official-binary-platforms-and-toolchains and needs to ensure maximum binary compatibility.

We've just locked down 15.x @ nodejs/build#2423 and it's not changing for that.

16.x planning has opened up @ nodejs/build#2445 but the only thing that's been agreed is that we need to get to at least GCC 7.

The problem with the cross compiler is that we don't have the nice backward compatibility afforded by devtoolsets on CentOS like on our other Linux release builders. Vanilla GCC is pretty sad for binary compatibility in that it's tied directly to libc and libstdc++ versions. So your binaries become tied to the computer you build on; so the GCC version and base operating system version matters. We'll have to look at the 18.04 support window and see if it aligns OK with the 16.x window (I haven't checked yet). But there's even risks with that and pulling in a libstdc++ (or just a libc) that is too new and then our binaries won't work on older platforms.