Spurious glibc Version Error When Using Build Scripts

[Alexhuszagh]

Issue Description

When using build scripts, even if rustc changes (path or target), cargo won't always do a full re-compile, and this can lead to spurious glibc errors. This is linked to rust-lang/cargo#10367 (comment). Since we change targets all the time, we're much more likely to cause this "feature" to manifest than standard builds.

There's 2 solutions:

1. Use a custom target directory for each cross-compilation target (--target-dir path/to/x).
2. Use cargo clean when the error appears.

More detailed information and a robust workaround are found on the wiki.

Update

This seems to happen sporadically, with any mixing of toolchains or targets. The issue occurs sporadically when the toolchain changes, and seems to be related to #858 (comment). The solution seems to be running cargo clean (with the appropriate target dir), but there needs to be a better solution.

Version Info

$ cross --version
cross 0.2.1 (459e03e 2022-05-26)
Warning: Falling back to `cargo` on the host.
cargo 1.63.0-nightly (39ad1039d 2022-05-25)

Sample code to reproduce:

$ git clone https://github.com/cross-rs/rust-cpp-hello-word
$ cd rust-cpp-hello-word
$ cross run --target powerpc64-unknown-linux-gnu --verbose
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "/usr/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "1000:1000" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=ahuszagh" "-e" "CROSS_RUNNER=" "-v" "/home/ahuszagh/.xargo:/xargo:Z" "-v" "/home/ahuszagh/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word:/project:Z" "-v" "/home/ahuszagh/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/home/ahuszagh/Desktop/cross/rust-cpp-hello-word/target:/target:Z" "-w" "/project" "-i" "-t" "ghcr.io/cross-rs/powerpc64-unknown-linux-gnu:main" "sh" "-c" "PATH=$PATH:/rust/bin cargo run --target powerpc64-unknown-linux-gnu --verbose"
       Fresh cc v1.0.73
   Compiling hellopp v0.1.0 (/project)
     Running `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build`
error: failed to run custom build command for `hellopp v0.1.0 (/project)`

Caused by:
  process didn't exit successfully: `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)

This should compile normally, however, we are unable to find the appropriate glibc version. This was detected while debugging #441.

[Emilgardis]

I can't replicate this with nightly 2022-05-26 on windows with the latest powerpc64 image (sha256:d52b1c1e2f0666cf8e244e890f8e6dfe1907e427e394de10d11c385ae728912c)

❯ cross run --target powerpc64-unknown-linux-gnu --verbose
+ "cargo" "metadata" "--format-version=1" "--filter-platform" "powerpc64-unknown-linux-gnu" "--features" ""
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
Warning: using newer rustc `1.63.0-nightly (490324f7b 2022-05-26)` for the target. Current active rustc on the host is `rustc 1.63.0-nightly (76761db59 2022-05-24)`.
 > Update with `rustup update`
+ "rustup" "target" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "nightly-x86_64-unknown-linux-gnu"
+ "C:\\WINDOWS\\system32\\wsl.exe" "-e" "wslpath" "-a" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word"
+ "C:\\WINDOWS\\system32\\wsl.exe" "-e" "wslpath" "-a" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word"
+ "C:\\Program Files\\Docker\\Docker\\resources\\bin\\docker.EXE" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=Emil" "-e" "CROSS_RUNNER=" "-v" "C:\\Users\\Emil\\.xargo:/xargo:Z" "-v" "C:\\Users\\Emil\\.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word:/project:Z" "-v" "C:\\Users\\Emil\\.rustup\\toolchains\\nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "G:\\workspace\\small dev space\\cross-issue724\\rust-cpp-hello-word\\target:/target:Z" "-w" "/project" "-i" "-t" "ghcr.io/cross-rs/powerpc64-unknown-linux-gnu:main" "sh" "-c" "PATH=$PATH:/rust/bin cargo run --target powerpc64-unknown-linux-gnu --verbose"
   Compiling cc v1.0.73
     Running `rustc --crate-name cc --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.73/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=8451de8edbca495a -C extra-filename=-8451de8edbca495a --out-dir /target/debug/deps -L dependency=/target/debug/deps --cap-lints allow`
   Compiling hellopp v0.1.0 (/project)
     Running `rustc --crate-name build_script_build build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=0a565c6e09d5b0ce -C extra-filename=-0a565c6e09d5b0ce --out-dir /target/debug/build/hellopp-0a565c6e09d5b0ce -C incremental=/target/debug/incremental -L dependency=/target/debug/deps --extern cc=/target/debug/deps/libcc-8451de8edbca495a.rlib`
     Running `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build`
     Running `rustc --crate-name hellopp src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=fc5c3be60a3d8fee -C extra-filename=-fc5c3be60a3d8fee --out-dir /target/powerpc64-unknown-linux-gnu/debug/deps --target powerpc64-unknown-linux-gnu -C linker=powerpc64-linux-gnu-gcc -C incremental=/target/powerpc64-unknown-linux-gnu/debug/incremental -L dependency=/target/powerpc64-unknown-linux-gnu/debug/deps -L dependency=/target/debug/deps -L native=/target/powerpc64-unknown-linux-gnu/debug/build/hellopp-618833d9398f1fb8/out -l static=hellopp -l stdc++`
    Finished dev [unoptimized + debuginfo] target(s) in 30.29s
     Running `/linux-runner powerpc64 /target/powerpc64-unknown-linux-gnu/debug/hellopp`
Hello, world!

[Alexhuszagh]

Running cargo clean fixed this, so it might be a build fingerprint issue or incremental build issue. Either way, not replicating for me anymore.

[Alexhuszagh]

I'm getting this when I mix the nightly and stable toolchains for a given target.

$ cross run --target armv7-unknown-linux-gnueabihf
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `/linux-runner armv7 /target/armv7-unknown-linux-gnueabihf/debug/hellopp`
Hello, world!
$ cross run +nightly --target armv7-unknown-linux-gnueabihf
Warning: using newer rustc `1.63.0-nightly (490324f7b 2022-05-26)` for the target. Current active rustc on the host is `rustc 1.61.0 (fe5b13d68 2022-05-18)`.
 > Update with `rustup update`
   Compiling hellopp v0.1.0 (/project)
error: failed to run custom build command for `hellopp v0.1.0 (/project)`

Caused by:
  process didn't exit successfully: `/target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build` (exit status: 1)
  --- stderr
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)
  /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.27' not found (required by /target/debug/build/hellopp-0a565c6e09d5b0ce/build-script-build)

We might have to document this as a known issue. Thoughts @Emilgardis?

[Emilgardis]

I want to know why this happens. Is the build script not invalidated with different toolchains?

[Alexhuszagh]

Still not sure, it doesn't reproduce very frequently. I think it's the above, but it should be invalidated and most of the time it is.