Cannot find crt1.o and friends when cross compiling against a custom sysroot

[kvasilye]

Hello,

First thank you for packaging GCC for MacOS, it's awesome that you decided to that!

I have an issue though.

Trying to cross compile for Linux x86_64 on an Apple Silicon Mac system with a custom sysroot.

The custom sysroot contains glibc and libstdc++ headers as well as libraries, and a few "standard" things like OpenSSL and libCurl.

The sysroot comes from Ubuntu Jammy and we are able to successfully use it to build with Clang. One reason we do it is that our target executable needs to run on Ubuntu Jammy and this is how we make sure that we're linking against the right Glibc and not some newer version. Having OpenSSL and libCurl and a few others right in the sysroot also makes things easier.

With your tooling, however, we're running into a link error -

1 - On the one hand, the compiler still tries to use paths from the sysroot which is included with it
2 - On the other hand, it passes crt1.o crti.o and crtn.o to the linker without paths, so the linker can't find them

Clang is able to find those files inside our custom sysroot. They are in <sysroot>/usr/lib/x86_64-linux-gnu/crti.o.

It feels to me that (1) above is wrong - if we are using our own sysroot, then GCC should not use the bundled sysroot. And then if (2) worked, it would find those files inside the sysroot.

Note crtbegin.o / crtend.o are referenced using absolute paths in the sysroot provided by your GCC package and are found.

The crt1.o / crti.o / crtn.o should probably be in sync with crtbegin.o / crtend.o and so the same logic should apply to them all. Either they should all be referenced from the sysroot provided by your package or they should be referenced in the sysroot we provide.

More details:

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2 -plugin /opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/liblto_plugin.so -plugin-opt=/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/lto-wrapper -plugin-opt=-fresolution=/var/folders/x7/kj402x853h31slgpsnfp62m00000gr/T//cc0T9Lcj.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc

--sysroot=/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64

--build-id --eh-frame-hdr --hash-style=both -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o test

crt1.o
crti.o

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/crtbegin.o -L/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/usr/lib/x86_64-linux-gnu -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/lib/../lib64 -L/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/lib/../lib64 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/lib -L/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/lib -L/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/usr/lib /var/folders/x7/kj402x853h31slgpsnfp62m00000gr/T//ccCbyGGB.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/crtend.o

crtn.o

This is the error:

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find crt1.o: No such file or directory
/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status

Analysis

Files like crt?.o are referenced without paths and cannot be found.

They are in the sysroot provided by GCC and yet not referenced with a path:

find /opt/homebrew/Cellar -name "crt?.o"

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/x86_64-unknown-linux-gnu/sysroot/usr/lib/crt1.o
/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/x86_64-unknown-linux-gnu/sysroot/usr/lib/crti.o
/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/x86_64-unknown-linux-gnu/sysroot/usr/lib/crtn.o

But they are in the sysroot and GCC should have been able to find them:

/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/usr/lib/x86_64-linux-gnu/crt1.o
/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/usr/lib/x86_64-linux-gnu/crti.o
/Users/kvasilye/AudioAdGenerator/env/StarfruitWhiskey-1.0/runtime/bin/sysroot-jammy-amd64/usr/lib/x86_64-linux-gnu/crtn.o

On a real Ubuntu Jammy which is where we derive our sysroot from:

find / -name "crt?.o"

/usr/lib/x86_64-linux-gnu/crti.o
/usr/lib/x86_64-linux-gnu/crt1.o
/usr/lib/x86_64-linux-gnu/crtn.o

If I run GCC without --sysroot then those files are referenced using absolute paths:

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/collect2 -plugin /opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/liblto_plugin.so -plugin-opt=/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../libexec/gcc/x86_64-unknown-linux-gnu/13.3.0/lto-wrapper -plugin-opt=-fresolution=/var/folders/x7/kj402x853h31slgpsnfp62m00000gr/T//ccxz7aV8.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot --build-id --eh-frame-hdr --hash-style=both -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o test.out

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crt1.o

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crti.o

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/crtbegin.o -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/lib/../lib64 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/lib/../lib64 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64 -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/../../../../x86_64-unknown-linux-gnu/lib -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/lib -L/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib /var/folders/x7/kj402x853h31slgpsnfp62m00000gr/T//ccExOt0f.o -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/crtend.o

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../x86_64-unknown-linux-gnu/sysroot/usr/lib/../lib64/crtn.o

[messense]

On first look it seems related to the multi-arch layout of Ubuntu/Debian sysroot? https://wiki.debian.org/Multiarch

[messense]

Note that crosstool-ng support for multilib is experimental and has a few caveats, so we may not be able to resolve this.

https://crosstool-ng.github.io/docs/caveats-features/

[kvasilye]

Re: multiarch

All of our libraries are of one system architecture - amd64 in this case

Maybe there is impact from multiarch in that the names of include and link directories have the arch name in them?

But still - Clang is able to find everything under the sysroot.

It's weird that GCC in this package:

1 - Still uses some CRT files from its own sysroot even though I've passed a sysroot option

/opt/homebrew/Cellar/x86_64-unknown-linux-gnu/13.3.0/toolchain/bin/../lib/gcc/x86_64-unknown-linux-gnu/13.3.0/crtbegin.o

2 - Doesn't use any directory names for the other CRT files, causing the linker to not be able to find them (see above)

PS - do you have any plans to support GCC 14?

[kvasilye]

Ok trying to build without --sysroot - allowing the GCC toolchain to use the sysroot that's already bundled.

Running into a different issue, would you be able to help?

Linking an executable which uses OpenSSL:

cross-cc.sh -g  CMakeFiles/rtmpdump.dir/rtmpdump.c.o -o rtmpdump  librtmp/librtmp.a  -lssl  -lcrypto  -lz  -lpthread  -ldl

Produces a bunch of unresolved symbols like this:

.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'fstat', version 'GLIBC_2.33'
.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'pthread_rwlock_destroy', version 'GLIBC_2.34'
.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'pthread_setspecific', version 'GLIBC_2.34'
.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'pthread_key_delete', version 'GLIBC_2.34'
.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'dlerror', version 'GLIBC_2.34'
.../aux/usr/lib/x86_64-linux-gnu/libcrypto.so: error: undefined reference to 'dladdr', version 'GLIBC_2.34'

Where ".../aux/usr/lib/.... libcrypto.so" comes from Ubuntu Jammy, which uses an earlier GLIBC version that the one in the GCC's sysroot.

I did check that GCC's sysroot contains libpthread and libdl (and I'm not sure where fstat is supposed to come from???).

Any ideas why the link errors then?

Thank you!

[messense]

PS - do you have any plans to support GCC 14?

Will upgrade with #51, just need to find the time to do the work.

Produces a bunch of unresolved symbols

These toolchains are built for GLIBC 2.17 currently.

homebrew-macos-cross-toolchains/x86_64-unknown-linux-gnu/.config

Lines 449 to 450 in 2a0a403

	CT_GLIBC_V_2_17=y
	CT_GLIBC_VERSION="2.17"