Release 6.3 Toolchains build broken since 2026-01-12

[sundresh]

The Release 6.3 Toolchains build has been broken since January 12, 2026.

Last successful build: 552d8c1

First failing build: 5ea85ae

Diff: 552d8c1...5ea85ae

Problem reported here: https://thebrowsercompany.slack.com/archives/C07CTN5TWDT/p1769180227002779

Example error message (see the logs for more context and variants of this for different build targets):

[5/13] Linking Swift static library clang\libswift_Builtin_float.lib
FAILED: clang/libswift_Builtin_float.lib 
C:\Windows\system32\cmd.exe /C "cd . && D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swiftc.exe -target x86_64-unknown-windows-msvc -emit-library -static -o clang\libswift_Builtin_float.lib clang/CMakeFiles/swift_Builtin_float.dir/__/linker-support/magic-symbols-for-install-name.c.obj clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj  && cd ."
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swift-driver-new.exe'

<unknown>:0: error: no such file or directory: 'clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj'

[sundresh]

I've confirmed that in my local Windows build of the Swift toolchain on main from yesterday (not 6.3), I have clang\CMakeFiles\swift_Builtin_float.dir\float.swift.obj in these two places:

PS S:\> Get-ChildItem -Path .\b\ -Recurse -File -Filter "float.swift.obj"


    Directory: S:\b\x86_64-unknown-windows-msvc\ExperimentalDynamicOverlay\clang\CMakeFiles\swift_Builtin_float.dir


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         1/22/2026   7:04 PM          11345 float.swift.obj


    Directory: S:\b\x86_64-unknown-windows-msvc\ExperimentalStaticOverlay\clang\CMakeFiles\swift_Builtin_float.dir


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----         1/22/2026   7:48 PM           9815 float.swift.obj

clang\CMakeFiles\swift_Builtin_float.dir\__\linker-support\magic-symbols-for-install-name.c.obj is also found in these directories.

[sundresh]

Differences in source code repo revisions between the last good and first bad swift-build revisions:

< llvm_project_revision=efff856192b6d05be7258ac9a0942595f4b58e90
---
> llvm_project_revision=f771343fba375f1c5eed1875551c0d8e3746f76e

< swift_revision=150fc514efd35258818fc03ffac3b82dbe919f60
---
> swift_revision=b45812020ad9866ed2ca7c01830a1d92f31d215d

Derived from last good context and first bad context.

swiftlang/swift differences

llvm/llvm-project differences

These don't seem relevant...

[sundresh]

I'm going to try branching off of the release/6.3 branch, removing the Experimental SDK jobs' dependencies on earlier jobs, and adding in a run-id argument pointing to the most recent completed workflow, and then just run one of the Experimental SDK jobs. It's possible I may need to create a special GitHub token to access the previous run's artifacts. If that is able to run, it will give me a way to relatively quickly iterate on the Experimental SDK jobs.

[sundresh]

Work in progress:

I used Claude Code to initially set up the above concept. It created a separate workflow that has a copy of the experimental-sdk job with inputs from the context of an earlier run ID I provided hardcoded + with the run-id parameter added to actions/download-artifact@v4 calls. Since thebrowsercompany/gha-download-tar-artifact doesn't accept a run-id parameter, it had to expand the three invocations of that action, and it made some mistakes in the process. It would be useful to add an optional run-id parameter to the action to make this process easier in the future.

I also disabled the "Setup sccache" step because it was having problems accessing the S3 bucket. Hopefully that doesn't matter much for the build failure in this job.

Edit: created PR thebrowsercompany/gha-download-tar-artifact#2 to add run-id parameter to thebrowsercompany/gha-download-tar-artifact.

[sundresh]

"Build Dispatch (C parts only)" is now failing, which occurs before the failing step that motivated this issue. Maybe that's something that effectively would have been skipped over if we had sccache enabled?

I see the build is called with sccache, so maybe it's breaking because sccache isn't configured. I'm trying re-enabling "Setup sccache," with just the optional AWS S3 parts commented out.

[sundresh]

Uncommenting "Setup sccache" except for the AWS S3 parts made "Build Dispatch (C parts only)" succeed. Currently 9.5 minutes into the build and towards the end of "Build Experimental Runtime," close to "Build Experimental Overlay," which is what has been failing on release/6.3.

[sundresh]

"Build Experimental Runtime" took a bit longer without the sccache S3 bucket: 9:24 vs. 7:39 in the most recent failing scheduled build. But! "Build Experimental Overlay" passed. I'm now 90% certain this is a caching issue: I'm guessing in the failing builds, the step that builds float.swift.obj is not actually building it, rather, sccache is pulling an empty (or otherwise corrupted/invalid) result for that build command from S3.

Next steps:

1. Can we get the S3 cache working with my branch that directly runs the experimental-sdk job and then turn on maximal debugging with cmake and sccache there to confirm my theory above?
2. Can we invalidate this sccache entry in the S3 bucket? Or does this make us question what else is invalid/out of date as well?

Test workflow runs: https://github.com/thebrowsercompany/swift-build/actions/workflows/test-experimental-sdk.yml
Branch: sundresh/debug_release/6.3_failure_2026-01-23

[sundresh]

Turns out I just had to add this to the top level of my workflow YAML file to enable the S3 bucket with sccache:

permissions:
  id-token: write

[sundresh]

Attempt in progress to build with S3 bucket for sccache: https://github.com/thebrowsercompany/swift-build/actions/runs/21307030116/job/61336776854

If that fails in the same way that "Release 6.3 Toolchains" does, the next step is to make "Build Experimental Overlay" really verbose to fully confirm that the the cached results on S3 are the problem:

- name: Build Experimental Overlay
  run: |
    cmake --build ${{ github.workspace }}/BinaryCache/ExperimentalOverlay --verbose
    sccache --show-stats
  env:
    SCCACHE_LOG: trace

Finally, if we confirm that's the problem, we can re-run the build with SCCACHE_RECACHE=1 set for this step to overwrite what's on S3. We'll need to do that for all job matrix values.

[sundresh]

Well... the run of my workflow with the S3 bucket enabled for sccache didn't fail in "Build Experimental Overlay." But maybe that's because sccache is (correctly) using different cache keys? I've confirmed that a second attempt of my workflow with S3 enabled runs faster:

• First attempt
  • Build Experimental Runtime: 9:36
  • Build Experimental Overaly: 22 seconds
• Second attempt
  • Build Experimental Runtime: 6:52 (significantly faster)
  • Build Experimental Overlay: 18 seconds

Now I'll run with verbose cmake and sccache tracing.

[sundresh]

I noticed that the CMake steps also aren't identical between my branch and Release 6.3 Toolchains. This is probably because they're running on different runners that have different pre-installed versions of some things. Trying to switch my branch over to use the same runners.

[sundresh]

After further inspection, I don't think it's a caching problem. sccache isn't being used when building Swift source files. The CMake behavior actually changed slightly between the successful and failing builds:

Last successful build:

Build Experimental Overlay

Run cmake --build D:\r\_work\swift-build\swift-build/BinaryCache/ExperimentalOverlay
[1/8] Generating GYB source float.swift from float.swift.gyb
[2/8] Generating GYB source tgmath.swift from tgmath.swift.gyb
[3/8] Building C object clang/CMakeFiles/swift_Builtin_float.dir/__/linker-support/magic-symbols-for-install-name.c.obj
[4/8] Linking Swift static library clang\libswift_Builtin_float.lib
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swift-driver-new.exe'

<unknown>:0: warning: option '-explicit-module-build' is only supported in swift-driver

When I tried and failed to reproduce the problem on a separate branch, I ended up with the above build pattern.

First failing build:

Build Experimental Overlay

Run cmake --build D:\r\_work\swift-build\swift-build/BinaryCache/ExperimentalOverlay
[1/13] Generating GYB source float.swift from float.swift.gyb
[2/13] Generating GYB source tgmath.swift from tgmath.swift.gyb
[3/13] Building C object clang/CMakeFiles/swift_Builtin_float.dir/__/linker-support/magic-symbols-for-install-name.c.obj
[4/13] Building Swift object clang/_Builtin_float.swiftmodule clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swift-driver-new.exe'

<unknown>:0: warning: option '-explicit-module-build' is only supported in swift-driver

<unknown>:0: warning: option '-incremental' is only supported in swift-driver

[5/13] Linking Swift static library clang\libswift_Builtin_float.lib
FAILED: clang/libswift_Builtin_float.lib 
C:\Windows\system32\cmd.exe /C "cd . && D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swiftc.exe -target x86_64-unknown-windows-msvc -emit-library -static -o clang\libswift_Builtin_float.lib clang/CMakeFiles/swift_Builtin_float.dir/__/linker-support/magic-symbols-for-install-name.c.obj clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj  && cd ."
<unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swift-driver-new.exe'

<unknown>:0: error: no such file or directory: 'clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj'

When we add --verbose to cmake, here's what step [1/13] does to generate float.swift:

2026-01-24T08:05:55.5084412Z [1/13] C:\Windows\system32\cmd.exe /C "cd /D D:\r\_work\swift-build\swift-build\SourceCache\swift\Runtimes\Overlay\clang && D:\r\_work\_temp\-1934228423\cmake-3.29.9-windows-x86_64\bin\cmake.exe -E make_directory D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang && D:\r\_work\_temp\-1934228423\cmake-3.29.9-windows-x86_64\bin\cmake.exe -E env D:/r/_work/_tool/Python/3.10.1/x64/python3.exe D:/r/_work/swift-build/swift-build/SourceCache/swift/Runtimes/Overlay/../../utils//gyb -o D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift.tmp float.swift.gyb && D:\r\_work\_temp\-1934228423\cmake-3.29.9-windows-x86_64\bin\cmake.exe -E copy_if_different D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift.tmp D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift && D:\r\_work\_temp\-1934228423\cmake-3.29.9-windows-x86_64\bin\cmake.exe -E remove D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift.tmp"

It generates D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift

And then here's how step [4/13] builds clang/_Builtin_float.swiftmodule clang/CMakeFiles/swift_Builtin_float.dir/float.swift.obj (what do the two file paths together like that indicate?):

2026-01-24T08:05:56.1642507Z [4/13] D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swiftc.exe -target x86_64-unknown-windows-msvc -j 16 -num-threads 16 -c -DSWIFT_STDLIB_ENABLE_VECTOR_TYPES -DSWIFT_STDLIB_HAS_COMMANDLINE -parse-as-library -static -emit-module -emit-module-path clang\_Builtin_float.swiftmodule -module-name _Builtin_float -module-link-name swift_Builtin_float -g -debug-info-format=codeview -Xlinker /DEBUG -Xlinker /INCREMENTAL:NO -Xlinker /OPT:REF -Xlinker /OPT:ICF -O -swift-version 5 -libc MD -incremental -Xfrontend -define-availability -Xfrontend "SwiftStdlib 9999:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.1:macOS 10.15, iOS 13.0, watchOS 6.0, tvOS 13.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.2:macOS 10.15.4, iOS 13.4, watchOS 6.2, tvOS 13.4" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.3:macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.4:macOS 11.3, iOS 14.5, watchOS 7.4, tvOS 14.5" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.5:macOS 12.0, iOS 15.0, watchOS 8.0, tvOS 15.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.6:macOS 12.3, iOS 15.4, watchOS 8.5, tvOS 15.4" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.7:macOS 13.0, iOS 16.0, watchOS 9.0, tvOS 16.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.8:macOS 13.3, iOS 16.4, watchOS 9.4, tvOS 16.4" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.9:macOS 14.0, iOS 17.0, watchOS 10.0, tvOS 17.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 5.10:macOS 14.4, iOS 17.4, watchOS 10.4, tvOS 17.4, visionOS 1.1" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 6.0:macOS 15.0, iOS 18.0, watchOS 11.0, tvOS 18.0, visionOS 2.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 6.1:macOS 15.4, iOS 18.4, watchOS 11.4, tvOS 18.4, visionOS 2.4" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 6.2:macOS 26.0, iOS 26.0, watchOS 26.0, tvOS 26.0, visionOS 26.0" -Xfrontend -define-availability -Xfrontend "SwiftStdlib 6.3:macOS 9999, iOS 9999, watchOS 9999, tvOS 9999, visionOS 9999" -Xfrontend -define-availability -Xfrontend "SwiftCompatibilitySpan 5.0:macOS 10.14.4, iOS 12.2, watchOS 5.2, tvOS 12.2, visionOS 1.0" -Xfrontend -define-availability -Xfrontend "SwiftCompatibilitySpan 6.2:macOS 26.0, iOS 26.0, watchOS 26.0, tvOS 26.0, visionOS 26.0" -Xfrontend -define-availability -Xfrontend "Backtracing 6.2: macOS 26.0" -nostdlibimport -strict-memory-safety -Xfrontend -enable-lexical-lifetimes=false -Xfrontend -disable-implicit-concurrency-module-import -Xfrontend -disable-implicit-string-processing-module-import -Xfrontend -enforce-exclusivity=unchecked -Xfrontend -target-min-inlining-version -Xfrontend min -enable-experimental-feature NoncopyableGenerics2 -enable-experimental-feature SuppressedAssociatedTypes -enable-experimental-feature SE427NoInferenceOnExtension -enable-experimental-feature NonescapableTypes -enable-experimental-feature LifetimeDependence -enable-experimental-feature InoutLifetimeDependence -enable-experimental-feature LifetimeDependenceMutableAccessors -enable-upcoming-feature MemberImportVisibility -explicit-module-build -autolink-force-load -emit-module-path D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/_Builtin_float.swiftmodule/x86_64-unknown-windows-msvc.swiftmodule -vfsoverlay D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/Windows/clang/windows-sdk-overlay.yaml -Xcc -DSWIFT_STATIC_STDLIB -Xcc -fmodule-map-file=D:/r/_work/swift-build/swift-build/SourceCache/swift/Runtimes/Core/SwiftShims/swift/shims/module.modulemap -output-file-map clang\CMakeFiles\swift_Builtin_float.dir\Release\output-file-map.json -I D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalRuntime/Core -I D:/r/_work/swift-build/swift-build/SourceCache/swift/Runtimes/Core/SwiftShims/swift/shims D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift
2026-01-24T08:05:56.2182470Z <unknown>:0: warning: using (deprecated) legacy driver, Swift installation does not contain swift-driver at: 'D:\r\_work\swift-build\swift-build\BinaryCache\Library\Developer\Toolchains\6.3.0+Asserts\usr\bin\swift-driver-new.exe'
2026-01-24T08:05:56.2183816Z 
2026-01-24T08:05:56.2184228Z <unknown>:0: warning: option '-explicit-module-build' is only supported in swift-driver
2026-01-24T08:05:56.2184666Z 
2026-01-24T08:05:56.2184913Z <unknown>:0: warning: option '-incremental' is only supported in swift-driver
2026-01-24T08:05:56.2185381Z 

The build command has a -c but no -o. The one Swift source file passed in is D:/r/_work/swift-build/swift-build/BinaryCache/ExperimentalOverlay/clang/float.swift. I'm not sure if the .swiftmodule paths are supposed to tell it where to put a .obj file and/or are generated with the path to where the .obj file was built? I don't see a float.obj or float.swift.obj created anywhere else in D:\r\_work\swift-build\swift-build (I tried enumerating all files in that dir--the current dir at the time--in this build).

[sundresh]

So the problem must be in Configure Experimental Overlay. The difference between the successful build and the failing build is the successful build passes -D CMAKE_Swift_COMPILER_USE_OLD_DRIVER=YES to cmake -B while the failing build does not. This is due to one of the changes in the swift-build diff: e337d38

Maybe what happened is this was safe to remove from main, but not safe to remove from 6.3. I'm going to try putting this back in on the release/6.3 branch (in swift-toolchain.yml, since it's no longer in action.yml on main), and if that fixes it, revert all my other changes on that branch, then figure out a proper solution with the team (conditional vs. different file state on release/6.3)

[sundresh]

Ah, good. Build Experimental Overlay passed, but now we're failing at Build Experimental String Processing for the same reason: https://github.com/thebrowsercompany/swift-build/actions/runs/21313046319/job/61352059455

I think we just need a more general place to set CMAKE_Swift_COMPILER_USE_OLD_DRIVER=YES.