Add support for cargo:rustc-env outputs from build scripts

[yxdai-nju]

Summary

This PR adds support for collecting cargo:rustc-env outputs from build scripts.

Relates to facebookincubator/reindeer#74.

Changes

• Allow the build script runner to collect cargo:rustc-env outputs as a sequence of --env/--path-env flags
• Add an [env_flags] sub-target for retrieving the flags output from build-script-run
• Add an env_flags attribute to rust_library/rust_binary rules that accepts such flags

Example with env_flags attribute and [env_flags] sub-target:

cargo.rust_library(
    name = "mime_guess-2.0.5",
    env_flags = ["@$(location :mime_guess-2.0.5-build-script-run[env_flags])"],
    # ...
)

After the PR to Reindeer, the env_flags = ["@$(... [env_flags])"], line will be automatically added, when buildscript.run = true.

Details

For each cargo:rustc-env={key}={value} output from a build script:

If value is an absolute path to an artifact, e.g.,

• /nativelink_work_dir/[hash]/work/buck-out/.../outfile in remote execution,
• /buck_root_dir/buck-out/.../outfile in local execution,

generate a --path-env={key}={relative_path} flag where relative_path is the path with "workdir-prefix" trimmed, e.g.:

--path-env=OUTFILE_PATH=buck-out/.../outfile

Otherwise,

generate a standard --env={key}={value} flag, e.g.:

--env=RING_CORE_PREFIX=ring_core_0_17_8_

This aligns with the behavior of process_env in prelude/rust/build.bzl when dealing with paths in environment settings.

Testing

A demonstration repository verifies this works for mime_guess and ring crates, both have cargo:rustc-env outputs from build scripts.

Closes #918.

[facebook-github-bot]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. (Because this pull request was imported automatically, there will not be any future comments.)

[cormacrelf]

LGTM but can you confirm this works for buckifying the mime-guess crate via reindeer? IIRC it did include a path in its generated environment variable and is likely to surface any issues with that.

[cormacrelf]

Suggested change

	for flag in ctx.attrs.env_flags:
	compile_cmd.add(cmd_args(flag))
	compile_cmd.add(ctx.attrs.env_flags)

Same effect I think.

[yxdai-nju]

Thanks! I applied it in this commit.

[yxdai-nju]

LGTM but can you confirm this works for buckifying the mime-guess crate via reindeer? IIRC it did include a path in its generated environment variable and is likely to surface any issues with that.

Sure! I just created a demonstrating repo: https://github.com/yxdai-nju/buck2-buildscript-envflags-demo.

[cormacrelf]

@yxdai-nju The desired outcome is:

--env=MIME_TYPES_GENERATED_PATH=/workspace/buck-out/v2/gen/root/[hash]/project/__mime_guess-2.0.5-build-script-run__/OUT_DIR/mime_types_generated.rs

Doesn't look like it will work in Remote Execution. That's an absolute path (/workspace/ prefix) being passed to rustc_action.py, which will probably fail because the workdir changes between actions on RE. For example:

1. the buildscript_run step may run at /tmp/work/000000000000 and write an env var MIME_TYPES_GENERATED=/tmp/work/000000000000/buck-out/.../OUT_DIR/mime_types_generated.rs.
2. The library compile step may run at /tmp/work/111111111111. The directory /tmp/work/000000000000 is 99.9% sure to be absent, having run on a completely different build machine, or simply having already been deleted as the action completed. The environment variable referring to it is passed to rustc, which complains "no such file or directory".

It might be necessary to special-case any paths that are prefixed with $OUT_DIR. I think it would be sufficient to de-absolute-ify the paths, and transform

/workspace/buck-out/v2/gen/root/[hash]/project/__mime_guess-2.0.5-build-script-run__/OUT_DIR/XXXXXXXXXXXXX

to

buck-out/v2/gen/root/[hash]/project/__mime_guess-2.0.5-build-script-run__/OUT_DIR/XXXXXXXXXXXXX

which will be guaranteed to exist at the time we execute the compile step, at least as long as OUT_DIR is passed in as environment, which it is, if you use [buildscript.gen_srcs]. The de-absolute path transform should be done immediately as you read the cargo:rustc-env=VAR=VALUE and written to the env vars file as a buck-out/... relative path.

[cormacrelf]

Also thanks for making that demo repo. I was going to just take your word for it, but this has been very helpful.

[yxdai-nju]

2. The library compile step may run at /tmp/work/111111111111. The directory /tmp/work/000000000000 is 99.9% sure to be absent, having run on a completely different build machine, or simply having already been deleted as the action completed. The environment variable referring to it is passed to rustc, which complains "no such file or directory".

After you pointed this out, I tried mime_guess with remote execution, and that's exactly what happened 😅

The de-absolute path transform should be done immediately as you read the cargo:rustc-env=VAR=VALUE and written to the env vars file as a buck-out/... relative path.

Your suggestion is accurate! I added this transformation to this PR, may you have a look? @cormacrelf

It relies on the fact that the build script runner (prelude/rust/tools/buildscript_run.py) runs in the "work directory root" (e.g., /tmp/work/111111111111/ if the buck-out directory is /tmp/work/111111111111/buck-out), so it removes that prefix if an environment variable value starts with it. I'm not certain whether the build script runner always runs in this location.

I have simultaneously updated the demonstration repo, adding NativeLink remote execution. You can use it directly with VS Code Dev Container.

[cormacrelf]

Lgtm. This should cover 99% of build scripts using env vars. Someone from meta should have a look.

Buck will always execute build actions from the root, containing buck-out. The only thing that can go wrong here is windows paths having more than one valid prefix, and the windows people can deal with that if it ever arises, which may never happen as long as we live.