Add Hermetic Swift Toolchain Support with Bzlmod Extension #1630
+5,284
−30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ma-oli
requested review from
aaronsky,
adincebic,
brentleyjones,
keith,
luispadron and
mattrobmattrob
as code owners
ma-oli
force-pushed
the
add-hermetic-toolchains-support
branch
from
f3857bb to
8c8acca
Compare
ma-oli
force-pushed
the
add-hermetic-toolchains-support
branch
3 times, most recently
from
1dec45d to
74acdeb
Compare
When using `swift_executable` option to swift_toolchain(), individual swift tools will be called using --driver-mode. This works when switching between swift and swiftc, but tools such as swift-autolink-extract and swift-symbolgraph-extract are separate and aren't recognized by --driver-mode. Therefore, these actions consistently fail in that scenario. As we're trying to add standalone hermetic swift toolchain support, we must have a way to include these tools in actions' input roots, so we're replacing the `swift_executable` toolchain parameter with a `swift_tools` one, which can be used to specify what binary to use for each particular action, as bazel labels. `swift_executable` is still available for backward compatibility, but it will only used if `swift_tools` is not set.
The file_prefix_map feature only works when DEVELOPER_DIR is set, which is only available in an Xcode toolchain. As we're removing the Apple macros in the bazel substitution code, we need to make sure unexpected variables don't show-up in configuration where they're not supported. We're doing so through the introduction of a new private feature called `swift._supports_developer_dir`, that will be enabled only on Xcode toolchains. Individual ActionConfigInfo can then check that feature if they use the __BAZEL_XCODE_DEVELOPER_DIR__ string.
The substitution logic is currently different on MacOS and on Linux. This is a problem as we're trying to add support for a standalone downloadable toolchain, so we're generalizing the logic here through the following changes: * __BAZEL_XCODE_DEVELOPER_DIR__ and __BAZEL_XCODE_SDKROOT__ only make sense in an Xcode context. The rules should make sure they are only used when, respectively, DEVELOPER_DIR and SDKROOT are required, and therefore if these are requested on a standalone toolchain, we would probably want to error out. So we're removing the __APPLE__ macro check, and instead error out if they are called and we are not able to perform the resolution. This is, arguable, a better user experience than silently returning the empty string, as it gives the user the opportunity to adjust toolchain features accordingly. * __BAZEL_SWIFT_TOOLCHAIN_PATH__ should be usable both in Xcode and in a standalone toolchain context. Therefore, we're introducing a new TOOLCHAIN_PATH variable, that will take precedence over the xcrun resolution if it is set. And if it is not, and that xcrun is not available on the system (Linux case), we will also let the user know something went wrong by erroring out. This variable will be set automatically if the user chooses to use swiftc_executable in its swift_toolchain() definition.
ma-oli
force-pushed
the
add-hermetic-toolchains-support
branch
from
74acdeb to
38ccdfc
Compare
This extension can be used to download and declare a precompiled
hermetic toolchain (same toolchain as what swiftly would download).
It has been tested on both Linux and MacOS.
Hermetic toolchains are provided through the `toolchain` extension tag
(swift.toolchain()), and allow for selecting the toolchain version.
Currently, only 6.2.1 is supported, but more can be added as we go.
Example usage:
```
swift = use_extension("@rules_swift//swift:extensions.bzl", "swift")
swift.toolchain(
name = "swift_toolchain",
swift_version = "6.2.1",
)
use_repo(
swift,
"swift_toolchain",
"swift_toolchain_ubuntu24.04-aarch64",
"swift_toolchain_xcode",
)
register_toolchains(
"@swift_toolchain//:cc_toolchain_embedded_ubuntu24.04",
"@swift_toolchain//:cc_toolchain_embedded_ubuntu24.04-aarch64",
"@swift_toolchain//:cc_toolchain_embedded_xcode",
"@swift_toolchain//:swift_toolchain_embedded_ubuntu24.04",
"@swift_toolchain//:swift_toolchain_embedded_ubuntu24.04-aarch64",
"@swift_toolchain//:swift_toolchain_embedded_xcode",
"@swift_toolchain//:swift_toolchain_exec_ubuntu24.04",
"@swift_toolchain//:swift_toolchain_exec_ubuntu24.04-aarch64",
)
```
This will also serve as a test to make sure swift-embedded works properly, along with the standalone toolchain extension. The example is derived from the swift embedded example project here: https://github.com/swiftlang/swift-embedded-examples To try it, do the following: $ cd examples/toolchain/embedded $ bazelisk build --platforms=//:aarch64 //rpi-4b-blink:Application.bin Note that at this stage, because this embedded code is making use of SPM, and because SPM bazel rules currently tries to run `swift package update` in its repository rules, having a swift executable in the PATH is still a requirement.
ma-oli
force-pushed
the
add-hermetic-toolchains-support
branch
2 times, most recently
from
e699500 to
b725865
Compare
Generating the toolchain dictionary currently requires downloading toolchains one by one and calculating their checksum, which is tedious and time consuming. We're adding a utility here that will do that automatically. The tool supports caching downloaded archives and can fetch release information from swift.org's API to generate the checksums dictionary format used by the build system. While we're at it, we're running that utility for the 2 latest releases (6.2.2 and 6.2.3) and adding them to the dict. Because that dictionary is probably gonna keep growing over time, we're also moving it in its own standalone file (swift_releases.bzl).
ma-oli
force-pushed
the
add-hermetic-toolchains-support
branch
from
b725865 to
e8cb1cc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new Bzlmod extension that downloads and configures standalone Swift toolchains, eliminating the need for local Swift installations. Users can now declaratively specify a Swift version in their MODULE.bazel, and the toolchain will be automatically downloaded from swift.org.
The implementation includes support for Swift 6.2.1 on macOS and all the Linux distributions and platforms currently supported by this version. It also refactors the toolchain configuration to properly support individual Swift tools (swiftc, swift-autolink-extract, swift-symbolgraph-extract) and generalizes the worker substitution logic to work across platforms.
An embedded Swift example (Raspberry Pi 4B blink) is included to demonstrate usage and serve as an integration test.
Usage
Testing
cd examples/toolchain/embedded bazelisk build --platforms=//:aarch64 //rpi-4b-blink:Application.bin