Decide on final semantics for Android NDK lookup

[jakepetroules]

(continued from discussion with @finagolfin in swiftlang/swift#88282)

...which will cause SwifftPM+SwiftBuild to use that NDK version, since it respects ANDROID_NDK_ROOT.

I don't think we should rely on env vars like that above what the user specified.

I'm guessing by "what the user specified" you're referring to the sdkRootPath property in the Swift SDK, but environment variables are also a mechanism by which users specify the NDK location.

Swift Build does not look at the symlinked NDK in the Swift SDK bundle, as that is a native-build-system-specific workaround.

You could say both are workarounds for broken SwiftPM, but ours is actually tying into the documented way to find the platform C/C++ SDK for Swift SDK bundles, ie by setting sdkRootPath to the symlinked NDK sysroot.

It's by design that Swift Build handles the NDK location for you. Users should not have to take on the burden of manually setting up symlinks.

The reason that Swift Build has to ignore the path specified in the Android Swift SDK is becomes it comes with a path entry pointing to a non-existent location at the moment. So the user experience is that you install the Swift SDK and it's just broken out of the box (with native build). With Swift Build, it "just works" immediately if you have already installed the NDK, without having to run any scripts to mutate the SDK.

That said, I'm happy to discuss making some tweaks to the logic that will satisfy a broader set of folks' requirements.

I think something like the following might work:

• sdkRootPath property is omitted from the Swift SDK by default, rather than containing a value pointing to a non-existing relative path like today
• The setup script ONLY adds sdkRootPath to the JSON, pointing to an explicit NDK path, without mutating ANY files in the Swift SDK -- no symlinks to the clang resource directory, no copyright swiftrt.o around
• (optional) add an error to the native build system if sdkRootPath is not set up. Currently it just silently proceeds to build the project with a macOS sysroot (or whatever the host sysroot is, or perhaps none) and an Android target triple 😬
• We document that users only need to run the setup script if using the SwiftPM native build system, otherwise it is optional.
• We document that setting a specific NDK can be done by setting ANDROID_NDK_ROOT in the environment OR using the script to set sdkRootPath in the Swift SDK
• If Finally wire up the sdk configure settings to compilation swift-package-manager#9229 is finished soon, we can eliminate the script entirely, and direct users to swift sdk configure instead.
• If sdkRootPath is not present and ANDROID_NDK_ROOT is not specified, Swift Build continues to use its existing lookup mechanism to find the newest NDK installed on the system

I think in that model the only potentially controversial piece is what happens when both ANDROID_NDK_ROOT is set and sdkRootPath is set in the Swift SDK (to potentially different values). Is that an error? Is that a warning, with one or the other chosen to win? Does one or the other win without emitting any diagnostic?

You seem to be advocating that the sdkRootPath win over the ANDROID_NDK_ROOT, I think I don't have a strong opinion.

One thing I want to be VERY clear about though: Swift Build is the final arbiter of the arguments and environment passed to the underlying build tools. In particular, it tends NOT to forward the calling environment down to tools, so we generally shouldn't have to worry about problems like with the native build system where the Swift SDK specified one Android NDK, but the ANDROID_NDK_ROOT "leaked" down to the compiler and pointed to a different NDK path. I suspect that's related to some of the issues you've seen in the past.

No we are not.

How are you picking up NDK 29 otherwise?

If ANDROID_NDK_ROOT is not specified, Swift Build chooses the latest installed NDK version if the NDK is installed in one of the known default locations.

[jakepetroules]

@finagolfin Please take a read through that and let us know what your specific proposals are. Thanks!

[finagolfin]

I'm guessing by "what the user specified" you're referring to the sdkRootPath property in the Swift SDK, but environment variables are also a mechanism by which users specify the NDK location.

Sure, the difference is that the Swift Android SDK bundle config is specific to Swift compilation, whereas that env var is mostly for C/C++ code built by the NDK.

It's by design that Swift Build handles the NDK location for you.

Android is a cross-compilation sysroot like any other, the less magic used to configure it the better, ie better not to use env vars.

Users should not have to take on the burden of manually setting up symlinks.

They wouldn't, the symlinks are a temporary workaround because the swift sdk configure that shipped when Swift SDK bundles were announced is broken, which I've fixed in the under-review swiftlang/swift-package-manager#9229. Once that feature is fixed, all SDKs would link to external proprietary platform C/C++ sysroots using that command.

I favor such general commands that users specifically configure over ad-hoc magic variables like ANDROID_NDK_ROOT for particular platforms.

The reason that Swift Build has to ignore the path specified in the Android Swift SDK is becomes it comes with a path entry pointing to a non-existent location at the moment. So the user experience is that you install the Swift SDK and it's just broken out of the box (with native build). With Swift Build, it "just works" immediately if you have already installed the NDK, without having to run any scripts to mutate the SDK.

Understood, but that "just works" has the above costs and more, such as potentially overriding any sdkRootPath the user explicitly chose for Swift cross-compilation, so I don't think it is worth it.

I think something like the following might work:

Much of this is predicated on still using the existing setup script or your env var if my pull isn't "finished soon," but I see no reason we can't get my pull in quickly and "we can eliminate the script entirely, and direct users to swift sdk configure instead."

Regarding "no symlinks to the clang resource directory, no copyright swiftrt.o around," I don't see how you plan to solve those, as those are the results of separate long-standing problems with Swift cross-compilation, completely unrelated to swift-build or SwiftPM, that we've had to work around in this way, see swiftlang/swift#63645 and swiftlang/swift-driver#1822.

You seem to be advocating that the sdkRootPath win over the ANDROID_NDK_ROOT, I think I don't have a strong opinion.

I am advocating that we only check sdkRootPath for Swift cross-compilation, and ignore the traditional C/C++ env vars used with the NDK altogether.

In particular, it tends NOT to forward the calling environment down to tools, so we generally shouldn't have to worry about problems like with the native build system where the Swift SDK specified one Android NDK, but the ANDROID_NDK_ROOT "leaked" down to the compiler and pointed to a different NDK path. I suspect that's related to some of the issues you've seen in the past.

I see, the calling environment variables are not passed into the Swift compiler by swift-build? I find that surprising.

If it does, that particular case, ie swiftlang/swift-driver#1879, would have broken swift-build too, as it was using the new -sysroot flag, which Swift SDK bundles do not support at all.

If ANDROID_NDK_ROOT is not specified, Swift Build chooses the latest installed NDK version if the NDK is installed in one of the known default locations.

I don't much like this either, but assuming it is explicitly warned about on the command-line, it can be a good backup to configure the NDK location.

Pinging the rest of the @swiftlang/android-workgroup for your input.

[jakepetroules]

I'm guessing by "what the user specified" you're referring to the sdkRootPath property in the Swift SDK, but environment variables are also a mechanism by which users specify the NDK location.

Sure, the difference is that the Swift Android SDK bundle config is specific to Swift compilation, whereas that env var is mostly for C/C++ code built by the NDK.

Remember that SwiftPM supports both C/C++ and Swift code. Swift code needs the Android NDK by proxy, but there are also Swift packages which don't include any Swift code, and therefore don't even need the Swift SDK for Android at all, to compile for Android.

In that case there's not even any swift sdk configure to speak of -- just swift build --triple aarch64-unknown-linux-android and you're done. This already works today (with Swift Build).

The reason that Swift Build has to ignore the path specified in the Android Swift SDK is becomes it comes with a path entry pointing to a non-existent location at the moment. So the user experience is that you install the Swift SDK and it's just broken out of the box (with native build). With Swift Build, it "just works" immediately if you have already installed the NDK, without having to run any scripts to mutate the SDK.

Understood, but that "just works" has the above costs and more, such as potentially overriding any sdkRootPath the user explicitly chose for Swift cross-compilation, so I don't think it is worth it.

There's no "potentially" here, it will behave deterministically based on the order of precedence we decide...

I think your proposal of having sdkRootPath take precedence over the environment variables is perfectly reasonable, but that doesn't have to be mutually exclusive with a reasonable fallback mechanism for when that property is not specified.

I am advocating that we only check sdkRootPath for Swift cross-compilation, and ignore the traditional C/C++ env vars used with the NDK altogether.

I strongly disagree with this; those env vars are defined by the platform vendor as the canonical way for tools to find the Android NDK. As a user, I do not want to run any step after swift sdk install to find the NDK, including swift sdk configure (and see above, swift sdk configure doesn't always apply since you might not be using a Swift SDK if you're only building C/C++ code or if you're on a Windows host, where you can use Swift SDK for Android or the Android SDK from the Windows toolchain). As I said before, I'm happy with emitting warnings and/or errors if both are specified and in conflict.

This is very similar to how Apple platforms work; the DEVELOPER_DIR environment variable is used to look up the native sysroots for macOS and iOS within Xcode or the Developer Command Line Tools when compiling for a macOS or iOS triple.

Windows, too, for the sysroot-equivalent bits.

Regarding "no symlinks to the clang resource directory, no copyright swiftrt.o around," I don't see how you plan to solve those, as those are the results of separate long-standing problems with Swift cross-compilation, completely unrelated to swift-build or SwiftPM, that we've had to work around in this way, see swiftlang/swift#63645 and swiftlang/swift-driver#1822.

As far as I'm aware, it's all working with top-of-tree main of SwiftPM+SwiftBuild.

That said, can you clarify how you're invoking the compiler in all of your relevant use cases? I can think of three potential ones:

• You're using SwiftPM + Swift Android SDK
• You're using CMake
• You're using direct compiler invocations

All of these are mutable.

If there are issues in the underlying compiler that cause problems for SwiftPM/SwiftBuild, then we just change the business logic in Swift Build to work around them until the compiler can be fixed. I don't see why this would be a significant issue at all. CMake, I have no idea.

[finagolfin]

In that case there's not even any swift sdk configure to speak of -- just swift build --triple aarch64-unknown-linux-android and you're done. This already works today (with Swift Build).

Not a good idea, better to stick to an explicit tools config rather than magic auto-config that breaks in weird ways. You think you are helping matters, but you often end up making installation more complex and error-prone.

I think your proposal of having sdkRootPath take precedence over the environment variables is perfectly reasonable, but that doesn't have to be mutually exclusive with a reasonable fallback mechanism for when that property is not specified.

It's not my proposal, that's how the SDK bundle evolution proposal and years of precedent have it working already. It would not be good to override that in ad-hoc ways for specific platforms.

I strongly disagree with this; those env vars are defined by the platform vendor as the canonical way for tools to find the Android NDK. As a user, I do not want to run any step after swift sdk install to find the NDK, including swift sdk configure

I highly doubt almost anyone is using those env vars in practice for local dev, they are relics for old-fashioned workflows, especially since almost all devs have their IDE handle everything now. 😛 The problem with your one-off Android env var approach is that now you have to document it and make sure Android users know that is an alternate way to set the NDK path for the Swift SDK for Android. That is all extra overhead that is usually simply not worth it just for specific platforms.

and see above, swift sdk configure doesn't always apply since you might not be using a Swift SDK if you're only building C/C++ code or if you're on a Windows host, where you can use Swift SDK for Android or the Android SDK from the Windows toolchain

Since the Windows toolchain does not support the swift sdk command or these SDK bundles at all yet, swiftlang/swift-package-manager#9148, building for Android on Windows will require a completely separate workflow, so no point in bringing that up. Similarly, almost nobody is going to build C/C++ code alone.

As I said before, I'm happy with emitting warnings and/or errors if both are specified and in conflict.

Better to avoid the possibility altogether.

This is very similar to how Apple platforms work; the DEVELOPER_DIR environment variable is used to look up the native sysroots for macOS and iOS within Xcode or the Developer Command Line Tools when compiling for a macOS or iOS triple.

I grepped through the source for the OSS toolchain and it appears to invoke that variable in only one place, just to find an xctest helper for SwiftPM. If you mean you check that env var in the newly used swift-build library, that has never been used in the OSS toolchain until recently.

Regarding "no symlinks to the clang resource directory, no copyright swiftrt.o around," I don't see how you plan to solve those, as those are the results of separate long-standing problems with Swift cross-compilation, completely unrelated to swift-build or SwiftPM, that we've had to work around in this way, see swiftlang/swift#63645 and swiftlang/swift-driver#1822.

As far as I'm aware, it's all working with top-of-tree main of SwiftPM+SwiftBuild.

Sure, as long as you put in extra config like this for the former, which we couldn't use with our SDK bundle because you can't pass sdkRootPath into toolsets, and workarounds like this for the latter. We have all had to work around these long-standing Swift compiler problems: you with those Android-specific config workarounds in swift-build, Marc and I in the layout and setup script for the SDK bundle.

That said, can you clarify how you're invoking the compiler in all of your relevant use cases?

Only SwiftPM with the installed Swift SDK for Android is currently supported.

If there are issues in the underlying compiler that cause problems for SwiftPM/SwiftBuild, then we just change the business logic in Swift Build to work around them until the compiler can be fixed. I don't see why this would be a significant issue at all.

Yep, that's what we have been doing with our non-standard SDK bundle format and setup script, but then one or two knowledgeable people have complained about that non-standard setup. 😉

We all want to get this fixed, but my preference is for standard mechanisms like swift sdk configure until the underlying compiler issues are worked out, not a bunch of Android-specific workarounds in the SwiftPM or swift-build source.

All that said, I'm not completely against checking the Android env vars or some other automatic NDK lookup as backup mechanisms. But such magic has to follow certain rules to make it usable for Swift on Android devs:

1. sdkRootPath is always given precedence, ie the Android env vars are completely ignored once it's set. sdkRootPath will be unset by default in the Android bundle once swift sdk configure works.
2. The use of the Android env vars by SwiftPM must be publicly documented and every time SwiftPM uses them or any other non-standard lookup, it must dump out some output saying so, ie the magic must be logged.
3. No setting new and largely untested flags like the Swift compiler's -sysroot flag with the env vars, since that was actually silently breaking builds before the linked swift-driver pull was merged, as we just had a user complaint about with the GitHub runners.

While I'm skeptical of such Android-specific magic config, those will at least help users figure out what is going on.

[marcprux]

I highly doubt almost anyone is using those env vars in practice for local dev, they are relics for old-fashioned workflows, especially since almost all devs have their IDE handle everything now.

I think these env vars are widely used. For example, GitHub actions sets them in their environment. Also, when configuring your NDK home via an IDE (like Studio or IntelliJ), the NDK preference will be communicated through to the underlying build system via the ANDROID_NDK_ROOT¹ environment. For example, once we sort out how we want to recommend Swift builds should be invoked from Gradle (see swiftlang/swift-android-examples#39), many Android-centric projects will likely have Gradle as the primary build environment, with the Swift toolchain being invoked by Gradle. When this is all happening from an IDE, I think the developer would want to have whichever NDK they have selected in the IDE to be the one that the Swift toolchain eventually uses.

All told, I am in favor of magic. I don't think the user should be required to do anything after running swift sdk install (neither swift sdk configure, and especially not our stopgap post-install swift-android/scripts/setup-android-sdk.sh script), and their ANDROID_NDK_ROOT should be consulted.

I even think that if ANDROID_NDK_ROOT is not set then we should try to automatically pick one from $ANDROID_HOME/sdk/ndk/, choosing the highest installed version (e.g., 27.0.12077973 instead of 26.3.11579264) when there is more than one in place. That seems to align with what is currently envisioned, if I understand @jakepetroules's afterword (#1279 (comment)) correctly.

If ANDROID_NDK_ROOT is not specified, Swift Build chooses the latest installed NDK version if the NDK is installed in one of the known default locations:

/cc @swiftlang/android-workgroup for their opinion

Footnotes

1. we should not conflate this issue with the previous issue of ANDROID_NDK_ROOT vs ANDROID_NDK_HOME that caused lengthy discussion in https://github.com/swiftlang/swift-driver/pull/1879 (see also https://github.com/swiftlang/swift/issues/43974) ↩

[finagolfin]

I think these env vars are widely used. For example, GitHub actions sets them in their environment.

Perhaps they are- I don't know for sure- but your examples show precisely the problems with using env vars. For one, github actions are precisely not the "local dev" like IDEs, that I said I doubt people are using them for. Second, that variable is automatically set by the CI, not by the user, so it can and will cause problems when the wrong NDK is chosen.

many Android-centric projects will likely have Gradle as the primary build environment, with the Swift toolchain being invoked by Gradle. When this is all happening from an IDE, I think the developer would want to have whichever NDK they have selected in the IDE to be the one that the Swift toolchain eventually uses.

I don't use Gradle, and my understanding is that a Gradle plugin is still being worked on for swift-java. But again, since we support certain NDK features in a particular NDK version and not others, plus there are concerns about NDK libc++ incompatibility, we don't want the NDK version choice to be made automatically by the tools, not by the user.

All told, I am in favor of magic. I don't think the user should be required to do anything after running swift sdk install... I even think that if ANDROID_NDK_ROOT is not set then we should try to automatically pick one from $ANDROID_HOME/sdk/ndk/

That's fine to have that preference, we all do to some extent. 😉 But since both your examples are more likely to break users' setups, because they are not made intentionally by the user, I don't think such magic is a good idea.

cc android-workgroup for their opinion

I just pinged the workgroup three days ago, I'm not sure what repeated pings are supposed to accomplish.

[jakepetroules]

Since the Windows toolchain does not support the swift sdk command or these SDK bundles at all yet, swiftlang/swift-package-manager#9148, building for Android on Windows will require a completely separate workflow, so no point in bringing that up. Similarly, almost nobody is going to build C/C++ code alone.

FYI, swift sdk, the Swift SDK bundles for Android, Static Linux, and Wasm, and the Android SDK in the Windows toolchain, are all fully supported and working with SwiftPM+SwiftBuild on Windows in the main branch and are covered by tests.

That issue can probably be closed at this point as the core issues have been resolved, though other folks may want to do a little more validation first.

We all want to get this fixed, but my preference is for standard mechanisms like swift sdk configure until the underlying compiler issues are worked out, not a bunch of Android-specific workarounds in the SwiftPM or swift-build source.

One thing I'd like to be clear about: swift-build contains Android-specific business logic because it supports Android as a first-class platform. While there may be some "Android-specific workarounds" in certain edge cases, most of it is simply "support". That's also not mutually exclusive with fixing any compiler-specific issues, these things take time to resolve.

All that said, I'm not completely against checking the Android env vars or some other automatic NDK lookup as backup mechanisms. But such magic has to follow certain rules to make it usable for Swift on Android devs:

1. sdkRootPath is always given precedence, ie the Android env vars are completely ignored once it's set. sdkRootPath will be unset by default in the Android bundle once swift sdk configure works.

Happy to make that change, but first I need you to remove sdkRootPath from the initial swift-sdk.json (https://github.com/swift-android-sdk/swift-docker/blob/67c14ef6c4d58fdfc7bb3377b26b0287aa88c435/swift-ci/sdks/android/scripts/build.sh#L789) and change that to being set by the setup-android.sh script. Maybe you can call the variable something like __sdkRootPath and then use sed to remove the leading __ in the setup script, since it's probably tricky to add a property to the JSON after-the-fact.

This will avoid breaking the zero-config path, and remove the need for the Android-specific workarounds in swift-build to ignore sdkRootPath. Then the logic path can be: if sdkRootPath is set, use that, otherwise fall back to standard ANDROID_NDK_* env vars + default search paths.

2. The use of the Android env vars by SwiftPM must be publicly documented and every time SwiftPM uses them or any other non-standard lookup, it must dump out some output saying so, ie the magic must be logged.
3. No setting new and largely untested flags like the Swift compiler's -sysroot flag with the env vars, since that was actually silently breaking builds before the linked swift-driver pull was merged, as we just had a user complaint about with the GitHub runners.>
   While I'm skeptical of such Android-specific magic config, those will at least help users figure out what is going on.

[finagolfin]

FYI, swift sdk, the Swift SDK bundles for Android, Static Linux, and Wasm, and the Android SDK in the Windows toolchain, are all fully supported and working with SwiftPM+SwiftBuild on Windows in the main branch and are covered by tests.

That's great news, but it does raise a couple questions:

• Will this SDK bundle support be extended to the old native llbuild build system for Windows SwiftPM also?
• What happens to "the Android SDK in the Windows toolchain" that the Windows devs have been distributing so far, do we remove that or provide two ways to install and use Android SDKs on Windows?

I'm all for using the same SDK bundles on Windows that we've been using on linux and macOS- in fact, I asked about that possibility on a couple of our Android workgroup calls months ago- but if you finally got it working with trunk SwiftPM on Windows, we'll need to address these potential issues now.

One thing I'd like to be clear about: swift-build contains Android-specific business logic because it supports Android as a first-class platform. While there may be some "Android-specific workarounds" in certain edge cases, most of it is simply "support". That's also not mutually exclusive with fixing any compiler-specific issues, these things take time to resolve.

I made no comment on the larger issue of swift-build support for Android generally. You simply raised the prospect of removing the clang symlinks and stopping moving the object file, which works around the swift-driver bug I submitted a one-line fix for late last year. We had put in those workarounds for those Swift compiler and SwiftPM bugs when creating the Swift SDK bundle on the CI, but you seemed to imply you hadn't hit those underlying toolchain issues with swift-build, so I pointed out where you had to work around those problems in swift-build also.

Happy to make that change, but first I need you to remove sdkRootPath from the initial swift-sdk.json... and change that to being set by the setup-android.sh script.

I don't think that's worth doing: I'd rather just fix swift sdk configure, then unset sdkRootPath like you say, and get rid of that script in the trunk SDK bundle altogether! 😃 I was told "we should review this change with the community:" is the couple weeks it's been up on GitHub with reviewers pinged enough for that, or do we need to do anything else for review of my SwiftPM pull? Otherwise, I'll just add some tests and get it through CI, so we can get that SwiftPM fix in this week.

This will avoid breaking the zero-config path, and remove the need for the Android-specific workarounds in swift-build to ignore sdkRootPath. Then the logic path can be: if sdkRootPath is set, use that, otherwise fall back to standard ANDROID_NDK_* env vars + default search paths.

I'm not worried about breaking the "zero-config path" that nobody is using in trunk yet, 😉 but once sdkRootPath is enabled in swift-build, will the swiftrt.o workaround in swift-build that I linked above still apply properly? We really should get that swift-driver fix in, since it represents zero risk, ie that fix still works fine for those few who want to use -sysroot instead, and will allow us to finally remove these SwiftPM SDK workarounds.

I don't mind the logic path you lay out, but you have not addressed the other two rules I laid out for applying such "magic" config for Android alone. Specifically, any time magic like using env vars is invoked because the user didn't run swift sdk configure, I think it's important to notify the user.

[jakepetroules]

FYI, swift sdk, the Swift SDK bundles for Android, Static Linux, and Wasm, and the Android SDK in the Windows toolchain, are all fully supported and working with SwiftPM+SwiftBuild on Windows in the main branch and are covered by tests.

That's great news, but it does raise a couple questions:

• Will this SDK bundle support be extended to the old native llbuild build system for Windows SwiftPM also?

No - the native build system is deprecated starting in Swift 6.4, so it will no longer receive significant changes (maybe a few targeted bug fixes), and shortly thereafter be removed completely.

• What happens to "the Android SDK in the Windows toolchain" that the Windows devs have been distributing so far, do we remove that or provide two ways to install and use Android SDKs on Windows?

Don't know. I think we need to have discussions involving @compnerd and the wider community on what the strategy for platforms and SDKs should be moving forward. He had some interesting ideas there.

For now, Swift Build will simply support both.

I'm all for using the same SDK bundles on Windows that we've been using on linux and macOS- in fact, I asked about that possibility on a couple of our Android workgroup calls months ago- but if you finally got it working with trunk SwiftPM on Windows, we'll need to address these potential issues now.

One thing I'd like to be clear about: swift-build contains Android-specific business logic because it supports Android as a first-class platform. While there may be some "Android-specific workarounds" in certain edge cases, most of it is simply "support". That's also not mutually exclusive with fixing any compiler-specific issues, these things take time to resolve.

I made no comment on the larger issue of swift-build support for Android generally. You simply raised the prospect of removing the clang symlinks and stopping moving the object file, which works around the swift-driver bug I submitted a one-line fix for late last year. We had put in those workarounds for those Swift compiler and SwiftPM bugs when creating the Swift SDK bundle on the CI, but you seemed to imply you hadn't hit those underlying toolchain issues with swift-build, so I pointed out where you had to work around those problems in swift-build also.

Happy to make that change, but first I need you to remove sdkRootPath from the initial swift-sdk.json... and change that to being set by the setup-android.sh script.

I don't think that's worth doing: I'd rather just fix swift sdk configure, then unset sdkRootPath like you say, and get rid of that script in the trunk SDK bundle altogether! 😃 I was told "we should review this change with the community:" is the couple weeks it's been up on GitHub with reviewers pinged enough for that, or do we need to do anything else for review of my SwiftPM pull? Otherwise, I'll just add some tests and get it through CI, so we can get that SwiftPM fix in this week.

Are you referring to #9229? I see it's still in draft, but keep in mind we also need to get it working with the Swift Build backend, which I don't think that PR does. I think as a first step you can break it up into separate PRs as suggested, and we can work through them.

If we can get swift sdk configure working with both native and swiftbuild, and consequently remove the script and the sdkRootPath property from the initial swift-sdk.json in time for 6.4, I think we'll be in really good shape for Android across the board.

This will avoid breaking the zero-config path, and remove the need for the Android-specific workarounds in swift-build to ignore sdkRootPath. Then the logic path can be: if sdkRootPath is set, use that, otherwise fall back to standard ANDROID_NDK_* env vars + default search paths.

I'm not worried about breaking the "zero-config path" that nobody is using in trunk yet, 😉

I do, I want that functioning properly.

but once sdkRootPath is enabled in swift-build, will the swiftrt.o workaround in swift-build that I linked above still apply properly?

It should, I think, but I have no problem making whatever additional tweaks are necessary in Swift Build to enable the Android SDK to work script-free.

We really should get that swift-driver fix in, since it represents zero risk, ie that fix still works fine for those few who want to use -sysroot instead, and will allow us to finally remove these SwiftPM SDK workarounds.

Sure, but I'm not quite sure what you mean by "who want to use" -- this isn't really an end user decision, it's something the build system will pass on your behalf, and whether Swift Build passes -sysroot or not may change over time, potentially on a per-platform basis, as a pure implementation detail.

I don't mind the logic path you lay out, but you have not addressed the other two rules I laid out for applying such "magic" config for Android alone. Specifically, any time magic like using env vars is invoked because the user didn't run swift sdk configure, I think it's important to notify the user.

I wouldn't mind adding that under --verbose mode.

[finagolfin]

Don't know. I think we need to have discussions involving @compnerd and the wider community on what the strategy for platforms and SDKs should be moving forward. He had some interesting ideas there.
For now, Swift Build will simply support both.

OK, I brought this up at our Android workgroup call this week, and he said he'd like SDK bundles to integrate with the existing MSI installation process they're using for their current non-bundle Android SDKs on Windows before SDK bundles can be considered a viable alternative.

Are you referring to swiftlang/swift-package-manager#9229?

Yes, that's the SwiftPM pull that gets swift sdk configure working.

I see it's still in draft, but keep in mind we also need to get it working with the Swift Build backend, which I don't think that PR does.

Sure, my pull works for the native backend and we can get it working with the swift-build backend, alongside the change for swift-build to actually use sdkRootPath.

I think as a first step you can break it up into separate PRs as suggested, and we can work through them.

Since done, and the others are almost in.

If we can get swift sdk configure working with both native and swiftbuild, and consequently remove the script and the sdkRootPath property from the initial swift-sdk.json in time for 6.4, I think we'll be in really good shape for Android across the board.

That would get it working, but we'd still be working around those underlying compiler bugs.

Sure, but I'm not quite sure what you mean by "who want to use" -- this isn't really an end user decision, it's something the build system will pass on your behalf, and whether Swift Build passes -sysroot or not may change over time, potentially on a per-platform basis, as a pure implementation detail.

The problem is that a few Swift devs are using that new -sysroot flag in their unique environs, but the SDK bundle format does not support that flag, using -resource-dir instead, but those two compiler flags are mutually exclusive. As a result, I implemented a compromise that if someone supplies the -sysroot flag to swift-driver, it will be used, otherwise default to the much more common use of -resource-dir.

swiftlang/swift-driver#1822 would keep builds working for all of us, but other than this one comment by Saleem, nobody has really addressed my final November version of that pull, which would allow us to remove workarounds that you and Marc and I had to put in this Android Swift SDK for that broken swift-driver behavior.

I don't mind the logic path you lay out, but you have not addressed the other two rules I laid out for applying such "magic" config for Android alone. Specifically, any time magic like using env vars is invoked because the user didn't run swift sdk configure, I think it's important to notify the user.

I wouldn't mind adding that under --verbose mode.

I think it should be logged by default, to avoid people getting confused about this magic swift-build behavior and simply unsetting all env vars instead. That's why we always log what our current Android SDK install script does, so people know exactly what it has done.

I'll take a look at how swift-build deals with the swift-sdk.json config for SDK bundles differently than the current native build, and see if I can get swift sdk configure working with swift-build too.