Document how to do cross-compilation

[arlyon]

The readme mentions how to cross compile for wasm, but doesn't explain cross compilation for any other platforms.

I am using rules_rust in conjunction with rules_docker and would like to compile with macOS as the host, targeting linux. Currently, without modifications, the docker image is build for macOS and fails on launch:

standard_init_linux.go:211: exec user process caused "exec format error"

My current attempt adds an entry to the .bazelrc file which specifies the platform for the image target.

build:image --platforms=//server:docker_platform

The platform is defined as such:

platform(
    name = "docker_platform",
    constraint_values = [
        "@platforms//os:linux",
    ]
)

The error on build is

ERROR: While resolving toolchains for target //server:image_binary: no matching toolchains found for types @io_bazel_rules_rust//rust:toolchain
ERROR: Analysis of target '//server:image' failed; build aborted: no matching toolchains found for types @io_bazel_rules_rust//rust:toolchain
INFO: Elapsed time: 0.152s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 146 targets configured)

Edit

I have loaded a custom toolchain in the WORKSPACE which seems like a step in the right direction but I am still hitting the same error.

rust_repository_set(
    name = "rust_darwin_x86_64",
    exec_triple = "x86_64-apple-darwin",
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    version = "1.39.0",
)

And updated .bazelrc, having tried both @io_bazel_rules_rust//rust/platform:linux and //server:docker_platform for the platforms argument

build:image --extra_toolchains=@rust_darwin_x86_64_toolchains//:toolchain_for_x86_64-unknown-linux-gnu --platforms=...

Hey @arlyon , I am trying this myself. Did you manage to compile rust to be used in dockers in the end?

[arlyon]

Unfortunately, no. My main motivation is to use this with skaffold, and I have resorted to maintaining my own docker files instead. I may take another crack at it when o get the time so if you make any headway, feel free to include it here.

[UebelAndre]

#770 also seems related to this.

[hlopko]

It seems https://github.com/GoogleContainerTools/distroless/pull/462/files is a nice example for doing what seems like what you ask about (even though it uses quite an old version of rules Rust). Is there anything more to do here?

One thing I think we could do is to document how to cross compile Rust locally (without docker, just locally, so the docs are not distracted by docker).

[hlopko]

Volunteers welcomed :)

[duarten]

I have an example here on doing cross-compilation on a macOS host to a Linux musl target.

It leverages the platforms infrastructure and --incompatible_enable_cc_toolchain_resolution. It also uses transitions to unconditionally compile the binary to the musl toolchain.

Not sure if this is the correct approach, but it seems to be working.

The example omits any dependencies, which complicate things a bit. In particular, if we want to compile the target for multiple platforms (e.g., to run unit tests on the host), it may be necessary to sprinkle selects on some raze-generated BUILD files (e.g., ring requires setting some platform-dependent env variables).

The biggest missing piece I think is lack of support for build_settings, especially when we want to add rustc flags like target-cpu to the target compilation across all deps.

[djmarcin]

I'm trying to cross-compile from macOS intel -> macOS arm64. I had to set up this rust_repository_set in WORKSPACE:

rust_repository_set(
    name = "rust_darwin_aarch64_cross",
    exec_triple = "x86_64-apple-darwin",
    extra_target_triples = ["aarch64-apple-darwin"],
    iso_date = "2021-06-09",
    version = "nightly",
)

And then set this up in the root BUILD

platform(
    name = "apple_m1",
    constraint_values = [
        "@platforms//os:macos",
        "@platforms//cpu:aarch64",
    ],
)

This appeared to work when called with bazel build --platforms //:apple_m1 //some:target -- at least until the third party packages finished building. Once that was done, the build failed because it couldn't find any third party dep -- maybe a missing crate 'envconfig' -- even though there is a line like --extern 'envconfig=bazel-out/darwin-fastbuild/bin/external/raze__envconfig__0_10_0/libenvconfig-1665319476.rlib' in the rustc command that is printed with bazel build -s. Similar errors were seen for other third party deps as well.

Is there something about deps that still needs to be solved before cross compilation will work?

[djmarcin]

🤦 This error is caused because I needed to specify edition = "2018" on the rust_repository_set. Otherwise this seems to work. I am still curious why rules_rust does not declare all the main supported platforms as valid cross-compilation targets by default, though.

[nickdecooman]

I found the example from @duarten very useful and got it working for basic crates when cross-compiling from MacOS to Linux.

However, when a crate contains custom Rust macros, it throws an error like this:

can't find crate for futures_macro
--> external/raze__futures_util__0_3_19/src/async_await/stream_select_mod.rs:6:9
|
6 | pub use futures_macro::stream_select_internal;
| ^^^^^^^^^^^^^ can't find crate
|
= note: extern location for futures_macro is of an unknown type: bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/raze__futures_macro__0_3_19/libfutures_macro-152066848.dylib
= help: file name should be lib*.rlib or lib*..so

More info about the issue here: duarten/rust-bazel-cross#2

[DeCarabas]

FWIW I started down this path, learning about transitions and whatnot, and then realized, oh, I don't think I need that. I ended up writing the following two entries in my WORKSPACE:

rust_repository_set(
    name = "rust_macos_arm64_linux_tuple",
    edition = "2021",
    exec_triple = "aarch64-apple-darwin",
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    version = "1.61.0",
)

rust_repository_set(
    name = "rust_macos_x86_64_linux_tuple",
    edition = "2021",
    exec_triple = "x86_64-apple-darwin",
    extra_target_triples = ["x86_64-unknown-linux-gnu"],
    version = "1.61.0",
)

...and everything just started working. (I put both in because we have folks on M1 macs and folks on Intel macs.)

I suspect that this was simple for us because we had already done the work to configure a clang-based toolchain that could target x86_64-unknown-linux-gnu from MacOS, which seems to be what most of @duarten's example is doing (albeit with *-linux-musl instead of *-linux-gnu).

But the problems seem to be separable, and this was not obvious to me from the discussion above.

Step 1: Configure a cc toolchain that runs on MacOS and targets Linux.
Step 2: Call rust_repository_set to declare a rust that does the same.