Add linkage attributes to extern "C" blocks

[saethlin]

This crate is now used in the compiler since rust-lang/rust#126930.
The problem I'm fixing is just like rust-lang/rust#118084; if you try to run

RUSTFLAGS=-Zcross-crate-inline-threshold=always ./x.py build library

After around 30 minutes (-Zcross-crate-inline-threshold=always makes compilation much slower) you get

  = note: rust-lld: error: undefined symbol: blake3_hash_many_avx512
          >>> referenced by ffi_avx512.rs:49 (/home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.5.2/src/ffi_avx512.rs:49)
          >>>               /home/ben/rust-master/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-1766fdd0f895a9f6.rustc_main.5f5b835cb13221b6-cgu.0.rcgu.o:(blake3::compress_subtree_wide::<blake3::join::SerialJoin>)
          >>> referenced by ffi_avx512.rs:49 (/home/ben/.cargo/registry/src/index.crates.io-6f17d22bba15001f/blake3-1.5.2/src/ffi_avx512.rs:49)
          >>>               /home/ben/rust-master/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_main-1766fdd0f895a9f6.rustc_main.5f5b835cb13221b6-cgu.0.rcgu.o:(blake3::compress_parents_parallel)
... any many more!

I think this is exactly the same problem as the above-linked issue: the blake3 crate does not have any link(name = attributes, so it only links correctly if all calls to the extern "C" functions are monomorphized by blake3 crate. But with -Zcross-crate-inline-threshold=always, nearly all monomorphization is done by the last crate to be compiled, and without these attributes we lose track of these symbols by the time we need them.

I'm not filing an issue because I'm pretty sure nobody else cares about my silly flag continuing to work when bootstrapping the compiler, but I occasionally find legitimate codegen bugs with it, so I'm sending you the patch to keep it working.

I've confirmed in my local dev setup that this PR makes it again possible to build the compiler with -Zcross-crate-inline-threshold=always.

[oconnor663]

Interesting, yes, let's fix this. CI is showing some build failures in cargo test --features=prefer_intrinsics. As you'll see, we get these symbols from different places in different modes. One confusing point is that SSE and AVX2 are available in stable Rust, so we have pure Rust intrinsics implementations for those, but not for AVX-512, which still links against C code (but different C sources). Can you get that working on your box?

[saethlin]

Oh I see what you mean, the avx512 situation adds another dimension to the matrix. I'll need a bit to untangle this.

[saethlin]

Oh dear it's the workflow approval dance.

[saethlin]

A bit of scope creep later, now I've:

• gated all of the linkage attributes on their matching cfg that's set by the build script
• renamed blake3_neon to blake3_neon_ffi so that it matches
• Added two new cfgs so that the build script can indicate which library is providing avx512 implementations

[saethlin]

Works on my machine too!

[saethlin]

👋 Just generating a notification to make sure this PR isn't lost.

[oconnor663]

Apologies for letting this sit for so long. I'll get back to it this weekend.

renamed blake3_neon to blake3_neon_ffi so that it matches

I think I added blake3_neon in a few more places since your last rebase, so that's probably causing some warnings that become errors in CI.

[oconnor663]

All of these extensions have assembly implementations and intrinsics implementations, which we switch between with --feature prefer_intrinsics. Do they all need the same treatment as AVX-512?

More broadly, could you help me understand what the link(...) attribute actually means :)

[saethlin]

I started digging into this and actually now I have questions about why the compiler seems to correctly forward native libraries dependencies through the crate graph if they come from #[link( but not if they come from command line arguments. I've poked some other compiler people, and I'll try to get back to you soon.

[saethlin]

I was randomly reminded that this PR is still open, and it's open because I don't actually understand the linkage situation here. @bjorn3 does though, can you help me out? (I could dig up what Björn has told me in the past but I would be blindly pasting it).

[bjorn3]

Basically the issue is that when building a rust dylib, rustc needs to know exactly which symbols to export. For rust functions this is easy. Rustc knows all functions and statics exported from rlibs and the local crate. For C code however there are two cases: The easy cases is where the C code is included through a dylib. In this case the rust dylib doesn't need to export the symbol and in fact trying to do so will result in a linker error. The hard case is where the C code is included through a static library. In this case rustc can only guess which symbols exist. The way rustc does this guessing is as follows: It exports all symbols that are declared in an extern "C" {} block for which it knows for a fact that it comes from a static library and in addition for which a call can be generated by downstream rust code. For example because the extern "C" {} declaration is exported from a crate or because it is called by a cross-crate inlineable function. What matters here is the "for which it knows for a fact that it comes from a static library" part. Rustc only knows this for sure if you annotate the extern "C" {} block with #[link(name = "...", kind = "static")]. Otherwise it assumes that it might come from a dylib and thus can't export it. The logic for this is at: https://github.com/rust-lang/rust/blob/e65b983161d52688ff8e05245ed5dc70ef01a904/compiler/rustc_codegen_ssa/src/back/symbol_export.rs#L60-L78 with is_statically_included being a literal check for #[link(name = "...", kind = "static")] and as such #[link(name = "...")] without specified kind or no #[link] at all being considered as possibly dynamically linked by this check.

[oconnor663]

Is this something that the cc crate might want to document in their examples? Like "FYI, if you use this to build and link C code, you need a #[link(... kind="static")] directive around the bindings or else no one will be able to build a dylib that includes your crate"? (I clearly dont understand the problem quite right though.) I guess the best possible outcome here could be that cc emits something to fix this automatically, but maybe that's not possible since it can't know what symbols you actually use?

[bjorn3]

I guess the best possible outcome here could be that cc emits something to fix this automatically, but maybe that's not possible since it can't know what symbols you actually use?

Yeah, I don't think the cc crate can help with this other than providing documentation for the user.