Duplicate symbol `pthread_atfork` when compiling Android release

[TheCataliasTNT2k]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

Building https://github.com/MarcelCoding/zia for target aarch64-linux-android fails to link. The linker fails because of "duplicate symbol pthread_atfork".

What target(s) are you cross-compiling for?

aarch64-linux-android

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.5 99b8069 2023-02-12

Example

# git clone https://github.com/MarcelCoding/zia; cd zia;
# cross +nightly build --target aarch64-linux-android 
 > Update with `rustup update`
Unable to find image 'ghcr.io/cross-rs/aarch64-linux-android:main' locally
main: Pulling from cross-rs/aarch64-linux-android
06d39c85623a: Pull complete 
d9c78f021986: Pull complete 
0fa3c2ce84ad: Pull complete 
3eea81fec61a: Pull complete 
7bf1365fd495: Pull complete 
dca916d29372: Pull complete 
3c5b70f3a37d: Pull complete 
c66e998c8ff4: Pull complete 
fc42a38b3c93: Pull complete 
d2a1a78ebf7d: Pull complete 
4f4fb700ef54: Pull complete 
b73c2c51a779: Pull complete 
39135fd47d8c: Pull complete 
2d1b4a229f04: Pull complete 
bf04508a63bc: Pull complete 
d0c929fc0ccd: Pull complete 
5d8dd7fd34a4: Pull complete 
1fc8abb5cd0f: Pull complete 
f2138848a78b: Pull complete 
7071b05dabc5: Pull complete 
fd6382120484: Pull complete 
Digest: sha256:61b7313a96033176e683e3577c10bbaaf01cf8bcecbcf58a9fa1fadb33236a40
Status: Downloaded newer image for ghcr.io/cross-rs/aarch64-linux-android:main
   Compiling serde_derive v1.0.147
   Compiling tokio v1.21.2
   Compiling futures-util v0.3.25
   Compiling rustls v0.20.7
   Compiling tikv-jemalloc-sys v0.5.2+5.3.0-patched
   Compiling serde v1.0.147
   Compiling pin-project-internal v1.0.12
   Compiling url v2.3.1
   Compiling tracing-attributes v0.1.23
   Compiling tungstenite v0.17.3
   Compiling minimal-lexical v0.2.1
   Compiling overload v0.1.1
   Compiling nu-ansi-term v0.46.0
   Compiling nom v7.1.1
   Compiling tracing v0.1.37
   Compiling tokio-tungstenite v0.17.2
   Compiling pin-project v1.0.12
   Compiling tokio-rustls v0.23.4
   Compiling async-http-proxy v1.2.5
   Compiling async-trait v0.1.58
   Compiling anyhow v1.0.66
   Compiling sharded-slab v0.1.4
   Compiling tracing-log v0.1.3
   Compiling webpki-roots v0.22.5
   Compiling thread_local v1.1.4
   Compiling smallvec v1.10.0
   Compiling pathdiff v0.2.1
   Compiling config v0.13.2
   Compiling tracing-subscriber v0.3.16
   Compiling zia-common v0.0.0-git (/root/zia/zia-common)
   Compiling tikv-jemallocator v0.5.0
   Compiling zia-client v0.0.0-git (/root/zia/zia-client)
   Compiling zia-server v0.0.0-git (/root/zia/zia-server)
error: linking with `aarch64-linux-android-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/android-ndk/bin:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "aarch64-linux-android-gcc" "/tmp/rustcr3D9Hx/symbols.o" "/target/aarch64-linux-android/release/deps/zia_server-f52298ac6ead9b2b.zia_server.97089f19-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-linux-android/release/deps" "-L" "/target/release/deps" "-L" "/target/aarch64-linux-android/release/build/tikv-jemalloc-sys-f0ee31bc29cef112/out/build/lib" "-L" "/target/aarch64-linux-android/release/build/ring-687200c902e83cf5/out" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "-Wl,-Bstatic" "/tmp/rustcr3D9Hx/libring-b491d8f9a053feb4.rlib" "/tmp/rustcr3D9Hx/libtikv_jemalloc_sys-d8a6a51c69091c9d.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libcompiler_builtins-85fd0e385adc9fc1.rlib" "-Wl,-Bdynamic" "-lgcc" "-ldl" "-llog" "-lunwind" "-ldl" "-lm" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "-o" "/target/aarch64-linux-android/release/deps/zia_server-f52298ac6ead9b2b" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-all" "-nodefaultlibs"
  = note: ld: error: duplicate symbol: pthread_atfork
          >>> defined at crtbegin.c
          >>>            /android-ndk/bin/../sysroot/usr/lib/aarch64-linux-android/28/crtbegin_dynamic.o:(pthread_atfork)
          >>> defined at zia_server.97089f19-cgu.0
          >>>            /target/aarch64-linux-android/release/deps/zia_server-f52298ac6ead9b2b.zia_server.97089f19-cgu.0.rcgu.o:(.text.pthread_atfork+0x0)
          clang140: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: could not compile `zia-server` (bin "zia-server") due to previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `aarch64-linux-android-gcc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/android-ndk/bin:/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin" VSLANG="1033" "aarch64-linux-android-gcc" "/tmp/rustcashmom/symbols.o" "/target/aarch64-linux-android/release/deps/zia_client-8271c8824ac6a2eb.zia_client.4e48e67e-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-linux-android/release/deps" "-L" "/target/release/deps" "-L" "/target/aarch64-linux-android/release/build/tikv-jemalloc-sys-f0ee31bc29cef112/out/build/lib" "-L" "/target/aarch64-linux-android/release/build/ring-687200c902e83cf5/out" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "-Wl,-Bstatic" "/tmp/rustcashmom/libring-b491d8f9a053feb4.rlib" "/tmp/rustcashmom/libtikv_jemalloc_sys-d8a6a51c69091c9d.rlib" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib/libcompiler_builtins-85fd0e385adc9fc1.rlib" "-Wl,-Bdynamic" "-lgcc" "-ldl" "-llog" "-lunwind" "-ldl" "-lm" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/aarch64-linux-android/lib" "-o" "/target/aarch64-linux-android/release/deps/zia_client-8271c8824ac6a2eb" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-all" "-nodefaultlibs"
  = note: ld: error: duplicate symbol: pthread_atfork
          >>> defined at crtbegin.c
          >>>            /android-ndk/bin/../sysroot/usr/lib/aarch64-linux-android/28/crtbegin_dynamic.o:(pthread_atfork)
          >>> defined at zia_client.4e48e67e-cgu.0
          >>>            /target/aarch64-linux-android/release/deps/zia_client-8271c8824ac6a2eb.zia_client.4e48e67e-cgu.0.rcgu.o:(.text.pthread_atfork+0x0)
          clang140: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: could not compile `zia-client` (bin "zia-client") due to previous error

Additional information / notes

Same build is fine for e.g aarch64-unknown-linux-gnu. And it builds fine for aarch64-linux-android if you do the debug build, and not the release build.

Do you know any solution for this?
It is very strange, that only the release version is affected....
It does not matter, if I do it in GH actions, or locally, using the same commands. Same error.

[Emilgardis]

Going to ask the same question here as I asked in #1222, what version of rust is +nightly? cargo +nightly -V

this might be a wrong assumption done for the new ndk

also, did this recently break (e.g has it worked before)? We did a fresh build of the images on :main here, it might've broken something we don't check for

[TheCataliasTNT2k]

cargo +nightly -V: cargo 1.70.0-nightly (4a3c588b1 2023-03-14)
has it worked before: No, never. we tried this Oct 1 last year for the first time, did not work. I waited and hoped, that new cross version fixed it, did not.
Will try to build it with stable, if the "features" are now stable: Ignoring the feature thing, does not build the "server version", which does not have the feature thing, "server" build fails with the same error on stable as on nightly.

So no chance an stable either.

[TheCataliasTNT2k]

It is not a very complex program, compiling does not take longer than 3 minutes for me. Docker pull takes way longer.
Easy way for replication:

git clone https://github.com/MarcelCoding/zia.git
cd zia/
git checkout v0.2.5
cross +nightly build --target aarch64-linux-android --release

docker image inspect ran on the pulled image:

[
    {
        "Id": "sha256:13628d7b91a1db2fce2e9cf8354e230fdec285070a5a3a9acf440a13aaee6249",
        "RepoTags": [
            "ghcr.io/cross-rs/aarch64-linux-android:main"
        ],
        "RepoDigests": [
            "ghcr.io/cross-rs/aarch64-linux-android@sha256:61b7313a96033176e683e3577c10bbaaf01cf8bcecbcf58a9fa1fadb33236a40"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2023-03-17T00:46:47.363047059Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/android-ndk/bin",
                "CROSS_TOOLCHAIN_PREFIX=aarch64-linux-android-",
                "CROSS_SYSROOT=/android-ndk/sysroot",
                "CROSS_ANDROID_SDK=28",
                "CROSS_TARGET_RUNNER=/android-runner aarch64",
                "CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc",
                "CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER=/android-runner aarch64",
                "AR_aarch64_linux_android=aarch64-linux-android-ar",
                "AS_aarch64_linux_android=aarch64-linux-android-as",
                "CC_aarch64_linux_android=aarch64-linux-android-gcc",
                "CXX_aarch64_linux_android=aarch64-linux-android-g++",
                "LD_aarch64_linux_android=aarch64-linux-android-ld",
                "NM_aarch64_linux_android=aarch64-linux-android-nm",
                "OBJCOPY_aarch64_linux_android=aarch64-linux-android-objcopy",
                "OBJDUMP_aarch64_linux_android=aarch64-linux-android-objdump",
                "RANLIB_aarch64_linux_android=aarch64-linux-android-ranlib",
                "READELF_aarch64_linux_android=aarch64-linux-android-readelf",
                "SIZE_aarch64_linux_android=aarch64-linux-android-size",
                "STRINGS_aarch64_linux_android=aarch64-linux-android-strings",
                "STRIP_aarch64_linux_android=aarch64-linux-android-strip",
                "CMAKE_TOOLCHAIN_FILE_aarch64_linux_android=/opt/toolchain.cmake",
                "BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android=--sysroot=/android-ndk/sysroot",
                "DEP_Z_INCLUDE=/android-ndk/sysroot/usr/include/",
                "RUST_TEST_THREADS=1",
                "HOME=/tmp/",
                "TMPDIR=/tmp/",
                "ANDROID_DATA=/",
                "ANDROID_DNS_MODE=local",
                "ANDROID_ROOT=/system",
                "CROSS_CMAKE_SYSTEM_NAME=Android",
                "CROSS_CMAKE_SYSTEM_PROCESSOR=aarch64",
                "CROSS_CMAKE_CRT=android",
                "CROSS_CMAKE_OBJECT_FLAGS=-DANDROID -ffunction-sections -fdata-sections -fPIC"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "org.cross-rs.for-cross-target": "aarch64-linux-android",
                "org.cross-rs.runs-with": "x86_64-unknown-linux-gnu",
                "org.opencontainers.image.created": "2023-03-17T00:14:51.373Z",
                "org.opencontainers.image.description": "“Zero setup” cross compilation and “cross testing” of Rust crates",
                "org.opencontainers.image.licenses": "MIT OR Apache-2.0",
                "org.opencontainers.image.ref.name": "ubuntu",
                "org.opencontainers.image.revision": "99b8069c0d977a14cd421ad8a3ef3255dc5802be",
                "org.opencontainers.image.source": "https://github.com/cross-rs/cross",
                "org.opencontainers.image.title": "cross (for aarch64-linux-android)",
                "org.opencontainers.image.url": "https://github.com/cross-rs/cross",
                "org.opencontainers.image.version": "nightly"
            }
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 3851018308,
        "VirtualSize": 3851018308,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/5cce033ab430ef60a5a571f68d741a7c5684dee1fdc3abcfbb3a11592358272c/diff:/var/lib/docker/overlay2/22cec9ed2ec1bc9ed31cf3f040d01846a0d54ec5e514338226d504ffd67c8eeb/diff:/var/lib/docker/overlay2/e754c482a8989bf7bff9cebf315f73c9d6a47d43a79261505f3852b554a43a16/diff:/var/lib/docker/overlay2/c0fc851c3f53ed25b90a2c02b1ab6f8f2a56f87a692f3e1a0f9d7054a9de61c9/diff:/var/lib/docker/overlay2/7bd161e00d0ed8615a641c343efd83cddb8211a8d7c08244fa6bd1d4693d088c/diff:/var/lib/docker/overlay2/54808dc41fa77037c3baae0f23a6a3799f40ea36f5e2490e961ec393281d364c/diff:/var/lib/docker/overlay2/e2177db68169ecd23fa205bdc1afbee3fb4933ddc6eb1a7df77a4ce2a0892348/diff:/var/lib/docker/overlay2/ff5b099ad28c3eca76a15038a1bf8b945746fa30569d6723c7732241468f7a70/diff:/var/lib/docker/overlay2/e46c4fabcfcd15eff56bcec2a703f34832dd1917dfdeec1f9b184e22871ef369/diff:/var/lib/docker/overlay2/c38994d5933e325aa08d6dfaecc5a94a8fc00acd9fe01270c3af1bfccb877a21/diff:/var/lib/docker/overlay2/3d49e05a2933b5d1ae6eacd61d1345b45b0d5e4485b90cef41d7fa05197e808c/diff:/var/lib/docker/overlay2/39fb8c77b3a8bcbbc989f8915e60bc25d8a289b7a73fa4e7494aaf66f43bb663/diff:/var/lib/docker/overlay2/96fcc38861fc6ff89022863e72a127d424983f370f8acf3fdba81d1f82ddb293/diff:/var/lib/docker/overlay2/3268ec6b132085ea44d3c5364dae8f7509149ea1d9876ea6205defd4d1f74603/diff:/var/lib/docker/overlay2/669d682e63115184a5797d8e0b920d76506464ce4d5a8d8e8f95f72c87b5ad58/diff:/var/lib/docker/overlay2/fac6f4b9a72382800fb4d14bddf4cd87667de3b870265b6c2f04e39862313a86/diff:/var/lib/docker/overlay2/b1ac4efb2007db1e7211d63eeee475bd34304828796dc6931ae6f38ee3e116d2/diff:/var/lib/docker/overlay2/76cac661a7edf6c1a301d29cecea42a24b423f8956d194fb76524717f6ce4cb4/diff:/var/lib/docker/overlay2/f658082081574bc52cc437f1835b7b6fe933ff75399ed7fc34304e75342ddfd7/diff:/var/lib/docker/overlay2/fd332ea3eef4a2488a6d5caec7884cfd9f48d7fd3c7d7c999b275a6d6f7b4d43/diff",
                "MergedDir": "/var/lib/docker/overlay2/58e839fe67460c78cc005fb2cb955f05408d43fed3e1954a142e54f9ea6e83dd/merged",
                "UpperDir": "/var/lib/docker/overlay2/58e839fe67460c78cc005fb2cb955f05408d43fed3e1954a142e54f9ea6e83dd/diff",
                "WorkDir": "/var/lib/docker/overlay2/58e839fe67460c78cc005fb2cb955f05408d43fed3e1954a142e54f9ea6e83dd/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:62789ffcd78ce0eae545b2743541d5c063270fe9f45477acacf2031b93cea70f",
                "sha256:f288490fc571e66d9fef0e511831e6def5b2751f33cdbd87417e159ae26579a4",
                "sha256:4d6ef59942e1a2dbc11024836368423026efa87c9470d4d3c8cc2d3b9530007d",
                "sha256:5e8d3c8453d547185add9e6778b260e5df0b410f0603a7c4c27193ff1590f23a",
                "sha256:955c572af9a6eddd642262b4158d9a17da2710b98f4607ba68d745fd6ec1a39a",
                "sha256:25be8d9c0371f4794c44081950fd0453fd39589144066e03bdc20060b4eb8831",
                "sha256:94e2989ddcb4404b01384ede7fc671b430dcac0cf05d5796cec56fbcc6339ae6",
                "sha256:d63fdc401776f5c9f25488ccc0aea04f26c2ed8d6f433cea6f588c5b8ad67e65",
                "sha256:831f3d1f9c147c0341e5fe35f83ccb2ae337ed79423ec6413ebbc158b985db51",
                "sha256:3ff876f8e7c4d45dc959f95ac97c25a07e8b06e36e0d1ee61b1679146060a176",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:6e78fba945abb41f7dbcf1b3e983c7046125b0036c6dfb98a401111b744c06bf",
                "sha256:6021a06baed6fe77a8f0c6c73c4678c6e17b4275794859e6ec6de7cf722c6bd7",
                "sha256:dd06d201ebfc163771a085e660bad744ee1446c21874af28d7f8c7a43605455f",
                "sha256:f9eaa5a3bf11da708224c8051561c60eb54554a59c2b037556dc8d3b0ca05fa3",
                "sha256:ec9d399525ae1bc5c488e05a9bea45d4c90e7d8ed2fcd28365474b98da1f4752",
                "sha256:9ae0e7a2244d4f23be1106d40310b1b444ba617d2d9440843ec3612fec4c70c0",
                "sha256:27203ac3a85904cab21c4966fa6dc79189e61dc02abaa752658460f91927c666",
                "sha256:955f4e4ed8f37bc9cc7cbe33887b276b3a343e1f18f5b38df6887a0dec2226a1",
                "sha256:4917728baa0cec2dc2f35ccd649691d8ea5ff8457e02f641141a9d1acd843992",
                "sha256:f5a8432fc7e5aa22c83a383082dbe6ddfe097eb721821eb49c32cd5eedba9d09"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[Emilgardis]

sorry, the information-required label is for why pthread_atfork is defined in the zia-client cgu

I'll remove the label :)

[TheCataliasTNT2k]

Ok, no worries, at least now you have a little more info 😉
But yes, that was a little misleading :)
If you need more info, just ask.
Thank you for this great project!

[MarcelCoding]

Removing these lines solves the problem: https://github.com/MarcelCoding/zia/blob/main/Cargo.toml#L8-L17 I'll further investigate which property exact is the problem.

[MarcelCoding]

Removing jmalloc MarcelCoding/zia@9583a8d (#42) is also solving the problem. I don't know why I didn't consider that a problem, sorry for that.

[TheCataliasTNT2k]

Seems to work with the changes.
@Emilgardis If you want to, you can close this, the workaround is enough (for us). If you want to fix it, to prevent future issues (for anyone): My replicate script above is version pinned, so you can still use that script.