Add support for cargo:rustc-env outputs from build scripts (draft)

[yxdai-nju]

Summary

This PR adds support for collecting cargo:rustc-env outputs from build scripts.

Relates to facebookincubator/reindeer#67.

Changes

• Modified the build script runner to collect cargo:rustc-env outputs, in the format of "--env=NAME=val" flags
• Added a new sub-target [env_flags] on buildscript_run, for getting the "--env" flags output
• Added a new attribute env_flags to rust_library and rust_binary rule, accepting "--env" flags

Example usage:

cargo.rust_library(
    name = "mime_guess-2.0.5",
    env_flags = ["@$(location :mime_guess-2.0.5-build-script-run[env_flags])"],
    # ...
)

After facebookincubator/reindeer#67, the env_flags = ["@$(location ...)"], setting will be automatically added, when any buildscript fixup of rustc_flags or gen_srcs type exists.

Closes #918.

[facebook-github-bot]

@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. (Because this pull request was imported automatically, there will not be any future comments.)

[cormacrelf]

All of this would be dramatically simpler, I'm talking +3 lines of code instead of +300, with the unstable --env-set flag (rust-lang/rust#118372).

Could this PR encode the flags the same way (as --env-set=KEY=VALUE once per line) so that when the flag is stabilised and rustc can do this natively, we simply remove special handling of the environment argfile and pass it directly to rustc?

[cormacrelf]

This is the +3 for posterity

facebookincubator/reindeer#48 (comment)

[yxdai-nju]

Could this PR encode the flags the same way (as --env-set=KEY=VALUE once per line) so that when the flag is stabilised and rustc can do this natively, we simply remove special handling of the environment argfile and pass it directly to rustc?

Thank you for your suggestions, this would be a really helpful feature!

However, things are a bit more complex with remote execution. If an environment variable contains a path, it should get converted to an absolute path before passed to rustc, to ensure correct behavior.

Currently, any environment variable values you set in the env attribute will automatically undergo this conversion.

The workflow of my solution is, for example:

1. Collect build script outputs:

# filtered build script outputs
cargo:rustc-env=NAME=val
cargo:rustc-env=MYFILE=path/to/myfile

--->

# flags for internal tool
--env=NAME=val
--env=MYFILE=path/to/myfile

This produces "--env" flags to be passed to write_env_lines_action internal tool.

2. Determine whether the values contain paths:
A critical step in prelude/rust/build.bzl in the PR is to determine whether each environment variable value is a path:

# Step 3: Utilize 'process_env' to determine whether each environment variable
#         is "plain" or "with path", then produce an *-env-flags.txt file
#         containing environment variable flags suitable to pass to the
#         `_long_command` macro. For instance
#         ```
#         --env=USER=john
#         --path-env=HOME=/home/john
#         ```
plain_env, path_env = process_env(compile_ctx, env_map, exec_is_windows, escape_for_rustc_action)

So the conversion done here is:

# flags for internal tool
--env=NAME=val
--env=MYFILE=path/to/myfile

--->

# flags for `_long_command`
--env=NAME=val
--path-env=MYFILE=path/to/myfile

This produces "--env" and "--path-env" flags to be passed to the _long_command macro, which is responsible for preparing the environment variables for rustc invocations; it converts a path to an absolute path, if an environment variable is passed to it with "--path-env" flag.

[yxdai-nju]

I misunderstood the purpose of process_env function and did lots of unnecessary conversions in the code. I'm closing this PR and filing a new one.

the process_env function is for distinguishing normal values and path values which are expanded from macros. For example:

env = {
    "NAME": "john",
    "RESULT_PATH": "$(location :my_crate-1.0.0-build-script-run[out_dir])/result.txt",
}

The mapping goes through the process_env function, and will be turned into:

--env=NAME=john
--path-env=buck-out/v2/gen/.../__my_crate-1.0.0-build-script-run__/OUT_DIR/result.txt

And with --path-env instead of --env, the _long_command macro can turn the buck-out/... path into absolute path before passing it to rustc.

Environment variable values emitted by build scripts don't contain macros and don't need this conversion. Using just --env=NAME=val flags would suffice.

[yxdai-nju]

Superseded by #929.