Testing macros in the OSS toolchain snapshots for macOS don't seem to work with SDK bundles

[finagolfin]

Description

I recently added the Testing library to my Android SDK bundles for Swift 6.1 and 6.2, finagolfin/swift-android-sdk@eb1567b1c, and it works well to cross-compile Testing tests using the latest linux snapshot toolchain and my CI-generated 6.1 SDK bundle, which I checked by cross-compiling @tayloraswift's swift-png and its tests. 😄

However, @marcprux tells me the same didn't work with the OSS mac toolchain snapshot when he tried it with another Testing repo, as he got errors saying:

error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found

suggesting there is some issue with how these Testing macros are installed on macOS.

Reproduction

Follow the instructions to use my Android SDK bundle on macOS, except using the 6.1 bundle linked above, not 6.0.3 as I have not backported the Android patches in this repo to that tag. An alternative would be to try the linux SDK bundle generated by swift-sdk-generator, if it supports this Testing library.

Expected behavior

Testing tests cross-compile fine using the mac toolchain's prebuilt macros, as they do on linux.

Environment

MacOS

Additional information

I have not reproduced this myself. Marc said he'd file this himself, but I'm doing it now so we don't forget and so somebody behind this repo can take a look in the meantime.

[grynspan]

jon@JGStudio swift-PR-76034-1812.xctoolchain % find . -name 'libTestingMacros.dylib'
./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib

So the library is being built, but our CMake script installs it to a different location than it does on Linux. I'm not sure why that is or if it might be the root cause. @stmontgomery @rintaro does either of you have any insight?

[finagolfin]

Oh, Marc also told me there's a ~/Library/Developer/Toolchains/swift-6.0.3-RELEASE.xctoolchain/usr/lib/swift/host/plugins/testing/libTestingMacros.dylib in the OSS toolchain for macOS, but maybe it is deficient or misconfigured in some way.

[grynspan]

Also I don't see a .swiftmodule for TestingMacros. Dunno if that is needed in this context.

[finagolfin]

Also I don't see a .swiftmodule for TestingMacros. Dunno if that is needed in this context.

I don't think it is, as the upcoming 6.1 toolchain for linux only has swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a-fedora39/usr/lib/swift/linux/Testing.swiftmodule/x86_64-unknown-linux-gnu.swiftmodule and the aforementioned plugin usr/lib/swift/host/plugins/libTestingMacros.so, works fine with my 6.1 Android SDK bundle.

[stmontgomery]

Looks like this is, or at least was, a bug in Swift Package Manager: in 6.1, it was checking whether the target triple represented macOS, and only then included the necessary macro plugin search paths to locate libTestingMacros.dylib in macOS toolchains. I think that was happening here (link from the release/6.1 branch):

https://github.com/swiftlang/swift-package-manager/blob/release/6.1/Sources/PackageModel/UserToolchain.swift#L676-L684

However, this code all changed somewhat recently in #8295 and it might have affected this, or perhaps even fixed the issue. I'm not certain, but it would be worth trying using a main development snapshot toolchain which has that change to see.

Regardless, I'm going to transfer this to SwiftPM since it's something controlled at that layer. It may be a dupe, though.

[finagolfin]

Thanks for likely finding the culprit, but my read of the code is that it is still broken in trunk for cross-compilation to non-mac platforms, because it checks the target triple instead of the host triple before applying the macOS plugin paths:

        self.targetTriple = triple

        var swiftCompilerFlags: [String] = []
        var extraLinkerFlags: [String] = []

        let swiftTestingPath: AbsolutePath? = try Self.deriveSwiftTestingPath(
            derivedSwiftCompiler: swiftCompilers.compile,
            swiftSDK: self.swiftSDK,
            triple: triple,
            environment: environment,
            fileSystem: fileSystem
        )

        if triple.isMacOSX, let swiftTestingPath {
            // swift-testing in CommandLineTools, needs extra frameworks search path
            if swiftTestingPath.extension == "framework" {
                swiftCompilerFlags += ["-F", swiftTestingPath.pathString]
            }

            // Otherwise we must have a custom toolchain, add overrides to find its swift-testing ahead of any in the
            // SDK. We expect the library to be in `lib/swift/macosx/testing` and the plugin in
            // `lib/swift/host/plugins/testing`
            if let pluginsPath = try? AbsolutePath(
                validating: "../../host/plugins/testing",
                relativeTo: swiftTestingPath
            ) {
                swiftCompilerFlags += [
                    "-I", swiftTestingPath.pathString,
                    "-L", swiftTestingPath.pathString,
                    "-plugin-path", pluginsPath.pathString,
                ]
            }
        }

I think this was broken in #7920 last fall by @xedin, ironically when he was trying to fix cross-compilation to wasm. The issue appears to be that deriveSwiftTestingPath is used to determine both host and target paths, so it will need to be split up to determine where these host plugins are separately from the target compilation flags, ie checking both host and target triples separately.

[marcprux]

Just to add some data, I have a bare-bones repository https://github.com/marcprux/swift-testing-cross-compile-demo with CI that uses the same Android SDK on both macOS and Ubuntu. You can see the test build output for both hosts at: https://github.com/marcprux/swift-testing-cross-compile-demo/actions/runs/13816030659. Ubuntu passes, but macOS fails:

Building for debugging...
[0/13] Write sources
[2/13] Write swift-version--451CF4548FE66429.txt
[4/15] Compiling TestingCrossCompile TestingCrossCompile.swift
[5/15] Emitting module TestingCrossCompile
[6/16] Wrapping AST for TestingCrossCompile for debugging
error: emit-module command failed with exit code 1 (use -v to see invocation)
[8/18] Emitting module TestingCrossCompileTests
/Users/runner/work/swift-testing-cross-compile-demo/swift-testing-cross-compile-demo/Tests/TestingCrossCompileTests/TestingCrossCompileTests.swift:4:12: error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
2 | @testable import TestingCrossCompile
3 | 
4 | @Test func example() async throws {
  |            `- error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
5 |     // Write your test here and use APIs like `#expect(...)` to check expected conditions.
6 | }

Testing.Test:1:30: note: 'Test' declared here
1 | @attached(peer) public macro Test(_ traits: any TestTrait...) = #externalMacro(module: "TestingMacros", type: "TestDeclarationMacro")
  |                              `- note: 'Test' declared here
[9/18] Compiling TestingCrossCompileTests TestingCrossCompileTests.swift
/Users/runner/work/swift-testing-cross-compile-demo/swift-testing-cross-compile-demo/Tests/TestingCrossCompileTests/TestingCrossCompileTests.swift:4:12: error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
2 | @testable import TestingCrossCompile
3 | 
4 | @Test func example() async throws {
  |            `- error: external macro implementation type 'TestingMacros.TestDeclarationMacro' could not be found for macro 'Test'; plugin for module 'TestingMacros' not found
5 |     // Write your test here and use APIs like `#expect(...)` to check expected conditions.
6 | }

Testing.Test:1:30: note: 'Test' declared here
1 | @attached(peer) public macro Test(_ traits: any TestTrait...) = #externalMacro(module: "TestingMacros", type: "TestDeclarationMacro")
  |                              `- note: 'Test' declared here

The locally-installed testing libraries on macOS are:

zap Toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a.xctoolchain % find . -name '*Test*dylib'
./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib
./usr/lib/swift/macosx/testing/libTesting.dylib

And on Ubuntu are:

marcprux@skipbox:~/.swiftpm/toolchains/swift-6.1-DEVELOPMENT-SNAPSHOT-2025-03-07-a-ubuntu24.04$ find . -name '*Test*.so'
./usr/lib/swift/linux/libTesting.so
./usr/lib/swift/linux/libXCTest.so
./usr/lib/swift/host/plugins/libTestingMacros.so

Let me know if there is any jiggery-pokery I can do with file paths or toolchain config to experiment with getting it working.

[stmontgomery]

Just as a temporary experiment, you could try moving or copying ./usr/lib/swift/host/plugins/testing/libTestingMacros.dylib to ./usr/lib/swift/host/plugins (up one level from the testing/ directory) to confirm my theory that it's a search path issue.

[finagolfin]

My take is that it is incorrectly not passing in the -plugin-path when cross-compiling from macOS to other platforms. So first build it natively for macOS with -v, find the -plugin-path flag that ends with /testing, then pass it in to the cross-compile manually with -Xswiftc -plugin-path -Xswiftc <found-path>/testing. That should get it to find the Testing macros at least when cross-compiling, though it may still fail later if missing other flags too.