Compiling many Swift Testing tests is significantly slower than XCTest tests

[marcosgriselli]

Description

Compiling a large amount of Swift Testing tests is significantly slower than compiling the same amount of tests using XCTest on a clean build.

At 1000 tests in a single file building a test suite with Swift Testing is 24x slower than an almost identical suite using XCTest.

Here's a repo with a project to benchmark: https://github.com/marcosgriselli/swift-testing-performance. A few results from that project:

1000 Tests in a single file:

➜  swift-testing-performance git:(main) ✗ ./benchmark.py --test_count 1000 --runs 5
Benchmark 1: XCTest (1000 tests)
  Time (mean ± σ):      3.059 s ±  0.090 s    [User: 0.658 s, System: 0.214 s]
  Range (min … max):    2.965 s …  3.183 s    5 runs

Benchmark 2: SwiftTesting (1000 tests)
  Time (mean ± σ):     73.940 s ±  0.341 s    [User: 0.747 s, System: 0.285 s]
  Range (min … max):   73.603 s … 74.427 s    5 runs

Summary
  XCTest (1000 tests) ran
   24.17 ± 0.72 times faster than SwiftTesting (1000 tests)

Performance significantly improves when splitting the tests into different files, still 10x slower than XCTest.

1000 tests split in 10 files:

➜  swift-testing-performance git:(main) ./benchmark.py --test_count 1000 --test_files 10
Benchmark 1: XCTest (1000 tests in 10 files)
  Time (mean ± σ):      2.010 s ±  0.166 s    [User: 0.665 s, System: 0.216 s]
  Range (min … max):    1.855 s …  2.185 s    3 runs

Benchmark 2: SwiftTesting (1000 tests in 10 files)
  Time (mean ± σ):     20.769 s ±  0.351 s    [User: 0.792 s, System: 0.295 s]
  Range (min … max):   20.531 s … 21.171 s    3 runs

Summary
  XCTest (1000 tests in 10 files) ran
   10.33 ± 0.87 times faster than SwiftTesting (1000 tests in 10 files)

I'm aware of the macros compilation build time issues outside of swift-syntax and that the team is working on addressing these, is there anything we can do in the meantime to reduce the impact while adopting Swift Testing?

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

Run benchmarks on the shared repo to measure performance.

./benchmarks.py --test_count 1000 --test_files 1 --runs 5

./benchmarks.py --test_count 1000 --test_files 10 --runs 5

swift-testing version/commit hash

Xcode 16.0 (16A5171c)

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.109.2 Apple Swift version 6.0 (swiftlang-6.0.0.3.300 clang-1600.0.20.10)
Target: arm64-apple-macosx14.0
Darwin Marcoss-MBP.attlocal.net 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64

[stmontgomery]

Tracked internally as rdar://130478685

[stmontgomery]

Thank you for this report @marcosgriselli! The example project and data are very helpful. I am investigating this along with @rintaro.

[grynspan]

Does this need to move over to swift-syntax or swift?

[stmontgomery]

Does this need to move over to swift-syntax or swift?

At this point I do believe it should be transferred to the swiftlang/swift repo. I just tried doing that though, and it appears that you cannot transfer issues across organization boundaries, and due to the pending apple → swiftlang organization transition we'll need to wait until swift-testing makes the transition. We expect this to happen before too long.

[marcosgriselli]

@stmontgomery happy to close and create the issue there unless you want the transfer to keep the issue history clean

[stmontgomery]

We can wait to transfer. The relevant people know about the issue already.

[rintaro]

I'm actively looking into this issue:

• [Basic] Don't rewrite source buffer copy multiple times swift#75115
• [Macros] Optimize 'rangeContainsTokenLocWithGeneratedSource' swift#75130

So far, these patches improves the swift-testing test compilation performance by 2.6x or so (tested with a file with 500 @Test/#expect cases) . I'm exploring other areas for further improvements.

[grynspan]

@rintaro Does this seem more like a compiler or swift-syntax issue, to you? Is there something we ought to be doing in Swift Testing at this stage?