Skip to content

feat: attribute parallel Swift Testing tests through their tasks - #37

Draft
esnunes wants to merge 4 commits into
mainfrom
swift-testing-per-task-attribution
Draft

esnunes wants to merge 4 commits into
mainfrom
swift-testing-per-task-attribution

Conversation

@esnunes

@esnunes esnunes commented Nov 9, 2026

Copy link
Copy Markdown
Member

What

Swift Testing tests that run in parallel in one process now get per-test evidence. Code built with -sanitize-coverage=func,trace-pc-guard,pc-table (plus -sanitize=undefined, which the Swift driver requires for -sanitize-coverage and which adds no checks to Swift) calls into the package on every function entry. The observer credits the function to the test whose task is running it, read from a TaskLocal the .coverageAttribution trait binds. Child tasks, async let, Task {} and awaited actors inherit the context. What no test's task runs (GCD queues, threads, Task.detached, suite setup) goes to a per-process floor.

  • Each test case gets a function record in a new functions.bin (indexed by functions.tsv). It is kept apart from records.bin, so readers that predate it never see one.
  • The SanitizerCoverage callbacks live in the Swift module, which every test target links directly however the package is linked. They forward to one recording copy per process through a sized owner table (TEST_COVERAGE_ATTRIBUTION_OWNER_2). Copies of the package that predate the table are still forwarded to.
  • README: how to adopt it (the trait goes on every suite, not on individual tests), the record format, and the limitations.

Why

Swift Testing runs a process's tests in parallel by default. The process-wide counters can't tell overlapping tests apart, so those tests were marked overlapped and got no evidence. Tuist could only fall back to serial runs, which took the TuistUnitTests scheme from about 650 s to about 1,420 s of test execution.

Alternatives considered

  • Serializing Swift Testing tests: exact, but more than doubles test time.
  • Reading Swift Testing's Test.current on each function entry: building a Test value on every function entry is far too costly. A TaskLocal of a UInt32 is one lookup.
  • Per-thread attribution: wrong for async code, which hops between threads; the task is the unit that follows the test.

Impact

  • Nothing changes without function-entry tracking: counter records, the skip list, UI-test launches and the coverage report are as before.
  • With tracking, a test's evidence names functions, not lines. Counts can't be split between tests running the same function at once.
  • Tracked code calls back into the package, so only code linked into test targets that link the package may be built with it. Tuist decides which targets are eligible (tuist/tuist PR to follow).
  • A Swift Testing test without the trait, or a trait listed before it on a test, runs outside any test's context, so its work counts for every test of the process. The README asks for the trait on every suite.

Validation

  • swift test passes. The new Fixtures/Parallel runs 10 Swift Testing tests that wait at a barrier, so all 10 run at once. attributesParallelSwiftTestingTestsThroughTheirTasks checks, under SwiftPM, xcodebuild on macOS and the iOS simulator, that each test's function record holds exactly the functions its task ran, and none of another test's:

    • an actor hop;
    • task-group children;
    • a Task {};
    • a global another test built.

    It also checks that Task.detached and GCD work land in the floor and in no test, and that an unused function appears nowhere. The existing tests (counters, skip list, UI-test launches) pass unchanged.

  • End to end, with the tuist CLI branch, on the whole TuistUnitTests scheme (3,700 Swift Testing tests, parallel), against a serial run whose counter records are exact:

    • every Swift Testing test got evidence;
    • 99.95% of the serial run's product-file links were in the parallel evidence. The few missing ones are one-time initializers, which the counters also credit only to whichever test runs them first;
    • test time matched the parallel baseline (651 s against 723 s);
    • two identical runs gave 94% of tests the same file set, with a maximum difference of 4 files.

    Extra files beyond the serial run come from the shared floor, which only adds files, so selection runs more tests, never fewer.

🤖 Generated with Claude Code

esnunes and others added 3 commits November 7, 2026 02:29
Swift Testing runs a process's tests in parallel, which the process-wide
counters can't tell apart: overlapping tests were left without evidence.
Code built with -sanitize-coverage=func,trace-pc-guard,pc-table now calls
back on every function entry, and the observer attributes the function to
the test whose task runs it, read from the trait's TaskLocal: child tasks,
Task {} and awaited actors keep it. What no test's task runs goes to a
floor. Each test case gets a function record in functions.bin, apart from
records.bin so older readers never meet one.

The callbacks live in the Swift module, which every test target links
directly however the package is linked, and forward to one recording copy
per process through a sized owner table (TEST_COVERAGE_ATTRIBUTION_OWNER_2);
copies that predate it are still forwarded to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Traits listed before it on a test, and tests without it, run outside any
test's context, so their work counts for every test of the process.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The static analyzer flags using getenv's result after a setenv, which
process_owner() may call when it publishes the owner table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant