feat: report the tests the skip list actually skipped - #39

Follows #38 (Swift Testing skipping), which merged while this was in progress, so this targets main.

What changed

Each test process that reads a skip list now writes skipped.tsv under $TEST_COVERAGE_ATTRIBUTION_DIR/<pid>/. The file lists the tests that process actually removed (XCTest) or cancelled (Swift Testing). It has one line per test, tab-separated: framework\tmodule\tsuite\tname. The framework is xctest or swift-testing, and tests are named as the records name them, for example testSkippedThrowing for an XCTest test.

• XCTest names. Tests appear as XCTest names them (testSkippedThrowing for testSkippedThrowing() throws), the same as in records.bin.
• XCTest. suite_without_skipped_tests reports the tests it removed only once the removal has taken effect: after -setTests: runs, or once the empty replacement suite exists. If an XCTest lacks either, the tests run and nothing is reported. +defaultTestSuite is called several times, so each process dedupes its lines.
• Swift Testing. The trait calls a new C function, test_coverage_attribution_report_skipped, right before Test.cancel. A parameterized test is reported once. Below Swift 6.3 the trait cancels nothing, so it reports nothing.
• File contract. The file is created, empty, as soon as the process reads a non-empty skip list. An empty file therefore means "nothing skipped here", while a missing file means "this process didn't report". If a line can't be written, or a name contains a tab or line break, the file is removed. That way a reader never mistakes a partial list for a complete one. Each line is a single write to an O_APPEND descriptor, so a second copy of the library in the same process can't interleave with the first. Each copy opens the file from its constructor.
• Only with the directory. If only TEST_COVERAGE_ATTRIBUTION_SKIP_FILE is set, the tests are still skipped, but nothing is reported because there is nowhere to write. Tuist writes the skip file from coverage-based selection, and it sets TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_DIR whenever a run collects coverage evidence on macOS or the iOS simulator (TestCoverageEvidenceService.prepare). On a device the process can't write to the Mac anyway.
• Parallel testing and shards. The README documents what each process reports, based on what the fixture runs showed:
  • Serial: the hosted unit test process and the UI test runner each report their own tests.
  • Parallel: each clone reports the listed tests of the classes it builds suites for, including a class whose tests are all listed. A test can be reported by more than one process, so a reader should take the union.
  • -only-testing (a shard): only the selected classes get suites, so a listed test outside them isn't reported. It wasn't part of the run anyway.
• Docs. README: a new Output row, a new "What was skipped" bullet in Skipping, a Guarantees line and a Limitations note. AGENTS.md: a new rule for changing skipped.tsv.

Why

Before this, the reader could only see which tests the skip list requested, not which ones were skipped. A listed test still runs if:

• it's in a Swift Testing suite without the trait;
• the build uses Swift older than 6.3;
• a future Xcode drops XCTestSuite's private -setTests:, in which case a class with only some of its tests listed silently runs all of them.

DataDog's dd-sdk-swift-testing tags each test its test impact analysis skipped and tracks "forced runs" (listed tests that ran anyway). This gives Tuist the same signal: a forced run is a listed test that no process reports.

Alternatives considered

• Infer skips from the result bundle. A removed XCTest test is absent from the bundle, but so is a test that -only-testing or a crash left out. A Swift Testing test that wasn't cancelled before 6.3 shows up as passed, exactly like a test that wasn't listed. Positive evidence from the process that did the skipping is unambiguous.
• Report through the owner copy, like scopes. That would mean adding a field to the scope_functions struct, which copies of different versions read by address. A copy reading a field that an older owner doesn't have would crash. A shared append-only file avoids changing that ABI.
• Write the file only when something was skipped. Then "nothing skipped" and "failed to report" would look the same.
• A separate variable for the report location. Not needed, because Tuist always sets both variables together where a report can be written.

Impact

• skipped.tsv is a new contract file. Tuist's reader (CoverageObserverOutput) is not part of this PR and comes as a follow-up in tuist/tuist. Until it lands, Tuist ignores the file.
• A new public C function, test_coverage_attribution_report_skipped. No changes to records.bin or launches.tsv.
• Overlap with the in-flight unskippable-markers work (feat/unskippable-markers) is limited to the skip section of the observer file and the README. A forced run (a listed test the report leaves out) needs nothing extra.

Validation

• Xcode 26.6 (Swift 6.3), full suite: swift test passed, 12 tests, including the fixtures under SwiftPM, xcodebuild on macOS and on an iOS 26.3 simulator. Swift Testing cancellations are reported there.
• Xcode 26.3 (Swift 6.2): skipsTheListedTests (serial, parallel and shard), skipsTheListedSwiftTestingTests, runsEveryTestWithoutASkipList and needsObjCToLinkTheObserverFromAStaticArchive passed. The report has no Swift Testing lines, and the SwiftPM run writes an empty skipped.tsv.
  • The first 26.3 runs failed in the parallel case with Mach error -308 (ipc/mig) server died, and so did a run with the base branch's Sources. The cause was another session running the same Fixtures/UITests fixture on the same simulator, with the same bundle IDs, at the same time. Once those runs stopped overlapping, every case passed.
• mise run lint: clean (SwiftFormat, SwiftLint and the clang static analyzer).
• New test assertions:
  • SkipListTests reads skipped.tsv (new Skips.swift parser) for the serial, parallel and shard runs of the Fixtures/UITests simulator fixture.
  • skipsTheListedSwiftTestingTests asserts the SwiftPM report, and that the file exists even when nothing was cancelled.
  • The static-archive test asserts the XCTest report with -ObjC and no report without it.
  • The #if compiler(>=6.3) split is kept.
• Not covered: a failed write removing the file, and a name containing a tab. Both are exercised by reading only, not by a test.

🤖 Generated with Claude Code