feat: report the tests the skip list actually skipped

AGENTS.md

@@ -3,20 +3,21 @@
 A Swift package that test targets link to record which coverage counters each test moves (per-test coverage attribution). Tuist's CLI reduces the output to per-test evidence; the record layout is a contract with it (`TuistKit.CoverageObserverOutput` in tuist/tuist).
 
 ## Layout
-- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests; it lives in this file because `-ObjC` loads only archive members that define a class.
-- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed.
+- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests, reporting the ones it removed in `skipped.tsv`; it lives in this file because `-ObjC` loads only archive members that define a class.
+- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list, reporting them with `test_coverage_attribution_report_skipped`. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed.
 - `Tests/TestCoverageAttributionTests` - runs the fixtures with attribution on and parses what the observer wrote: `Records.swift` mirrors the `records.bin` layout and `FunctionTable.swift` maps counters to function names as Tuist's reader does.
   - `Fixtures/Example` - with SwiftPM, xcodebuild on macOS and xcodebuild on an iOS simulator.
   - `Fixtures/Concurrency` - the overlapping, nested-trait and failure cases, in a package of their own since Swift Testing runs suites in parallel across the process.
   - `Fixtures/StaticArchive` - links the observer from a static archive, with and without `-ObjC`.
-  - `Fixtures/UITests` - an app, its UI tests and its hosted unit tests. It's a Tuist project (`Project.swift`), because a UI test needs an app target and SwiftPM can't build apps; `Launches.swift` reads `launches.tsv`, and `SkipListTests.swift` runs the `SkipList` scheme serially, in parallel and as a shard and reads the result bundle.
+  - `Fixtures/UITests` - an app, its UI tests and its hosted unit tests. It's a Tuist project (`Project.swift`), because a UI test needs an app target and SwiftPM can't build apps; `Launches.swift` reads `launches.tsv`, and `SkipListTests.swift` runs the `SkipList` scheme serially, in parallel and as a shard and reads the result bundle and `skipped.tsv` (`Skips.swift`).
 
 ## Rules
 - Never crash or block the test process: every failure path returns and the tests run as if nothing were linked.
 - Never reset or write the counters.
 - Changing `records.bin`: update `write_record()`, `Records.swift`, the README and Tuist's reader together, and bump `kRecordVersion` when a record gains a field.
 - Changing `launches.tsv`: update `write_launch()`, `Launches.swift`, the README and Tuist's reader (`CoverageObserverOutput.parseLaunches`) together.
 - Changing the skip list format: update `load_skip_list()`, `SkipListTests.swift`, the README and Tuist's writer together.
+- Changing `skipped.tsv`: update `report_skipped()`, `Skips.swift`, the README and Tuist's reader together.
 - No `unsafeFlags` in `Package.swift`: they make the package unusable as a versioned dependency.
 
 ## Commands

README.md

@@ -94,10 +94,14 @@ TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE="$PWD/skip.txt" xcodebuild test
 
 - **Names.** Each line is `module/suite/name`, the way `-skip-testing` names a test and the records name it: the test bundle, the class without its Swift module, and the test as the result bundle names it (`testAppliesDiscount()`). The selector works too (`testAppliesDiscount`), including the bridged selectors of throwing and asynchronous Swift tests (`…AndReturnError:`, `…WithCompletionHandler:`).
 - **Only tests.** A line without exactly three components is ignored. Skip a whole target with `-skip-testing:CheckoutTests` instead. A suite is never skipped by naming it, because it can gain tests the list doesn't know about.
-- **Forgiving.** Blank lines and tests that don't exist are ignored. An unreadable file skips nothing. Skipping doesn't depend on `TEST_COVERAGE_ATTRIBUTION_DIR`.
+- **Forgiving.** Blank lines and tests that don't exist are ignored. An unreadable file skips nothing. Skipping doesn't depend on `TEST_COVERAGE_ATTRIBUTION_DIR`, but reporting what was skipped does.
 - **Where it applies.** `xcodebuild` passes the variable to the test process with the `TEST_RUNNER_` prefix: the app that hosts unit tests, and the runner of UI tests. It works with `test` and `test-without-building`, with `-only-testing`, and with parallel testing, where Xcode lists the tests, without the removed ones, before it hands them out to its clones. `swift test` reads it from the environment.
 - **What a skipped test looks like.** It's removed from the suite XCTest builds for its class, so it doesn't run and isn't in the result bundle. A class whose tests are all removed doesn't run its class `setUp` and `tearDown`. A class with tests left runs them as usual, class `setUp` included.
 - **Swift Testing.** In a suite with the `.coverageAttribution` trait, a listed test is cancelled before it starts, every argument of a parameterized test included, and reported as skipped, whether tests run in parallel or not. The line names it as the records do: the Swift module, the innermost suite, and the test's display name when it declares one (`Applies a discount`), its function otherwise (`appliesDiscount()`). A display name with a `/` can't be listed. This needs Swift 6.3 (Xcode 26.4) or later: before, a trait can skip a test only by not running it, which reports it as passed, so the listed tests run. Listed tests in suites without the trait run too.
+- **What was skipped.** A listed test can still run: a suite without the trait, Swift before 6.3, an XCTest without the private method partial skipping uses (see [Limitations](#limitations)). So with `TEST_COVERAGE_ATTRIBUTION_DIR` set, each process that read a skip list writes [`skipped.tsv`](#output), the tests it actually removed or cancelled. A listed test that no process reports ran, or was never part of the run. A test is reported once the removal took effect, by the process that removed it:
+  - **Serially**, the test process reports every listed test of the classes XCTest builds suites for and every Swift Testing test the trait cancels. A hosted unit test target and a UI test target run in processes of their own, so each reports its own tests.
+  - **With parallel testing**, the tests are spread over clones, and each reports the listed tests of the classes it builds suites for, a class whose tests are all listed included. A test can be reported by more than one process, such as one that only lists the tests, so read the union across processes.
+  - **With `-only-testing`**, as in a shard, XCTest builds suites only for the selected classes, so a listed test outside them isn't reported: it isn't part of the run.
 
 ## Output
 
@@ -109,6 +113,7 @@ The output is written under `$TEST_COVERAGE_ATTRIBUTION_DIR/<pid>/`, one directo
 | `<n>.data`, `<n>.names` | Image `n`'s raw `__llvm_prf_data` and `__llvm_prf_names` sections: what maps a counter to its function. |
 | `records.bin` | One record per scope. |
 | `launches.tsv` | One line per app process a UI test launched, written when `XCUIApplication.launch()` or `activate()` returns: the process's pid, the test bundle, the suite and the test, separated by tabs. A launch outside a test, such as in a class `setUp`, has an empty test name. A pid is listed once, with the scope that launched it, even when later tests activate the same process. Only written by a process that launches an app. |
+| `skipped.tsv` | One line per test the [skip list](#skipping-tests) had removed (XCTest) or cancelled (Swift Testing) in this process: the framework (`xctest` or `swift-testing`), the test bundle or Swift module, the suite and the test, separated by tabs, named as the records name them (an XCTest test as XCTest names it, `testAppliesDiscount` for `testAppliesDiscount() throws`). A test is listed once. Created, empty, as soon as the process reads a skip list, so an empty file means nothing was skipped. If a line can't be written, or a name holds a tab or a line break, the file is removed: a missing file means the process didn't report, not that it skipped nothing. Only written by a process with a skip list. |
 
 A scope is one of the following:
 - an XCTest test;
@@ -127,12 +132,13 @@ The deltas are what turns counters into lines. In the image's coverage mapping (
 
 - It never resets or writes the coverage counters: Xcode's and SwiftPM's own coverage reports are unchanged.
 - It never crashes or blocks the tests. Any failure stops recording, removes what it wrote for the process so a reader never gets partial output, and lets the tests run.
+- `skipped.tsv` is never partial: a test that was skipped is either listed, or the file is gone. Each copy of the package in the process appends its own lines to it, one write per line.
 - Only one copy records per process. If the package ends up linked more than once into the same process, for example into two test bundles loaded by the same host, the first copy to load does the recording and the others forward to it. The Objective-C runtime then logs `Class TestCoverageAttributionObserver is implemented in both …` once, which is harmless.
 
 ## Limitations
 
 - **Swift Testing needs the trait.** Tests in suites without it aren't attributed or skipped, tests that run at the same time get no evidence, and skipping them needs Swift 6.3. See [Usage](#usage).
-- **Skipping some of a class's tests uses a private method.** XCTest has no public way to remove tests from a class's suite, so the package uses `XCTestSuite`'s private `-setTests:`, which keeps the suite and the class `setUp` it runs. If a future Xcode removes it, a class with some tests listed runs all of them; a class with all of its tests listed is still skipped. A class that overrides `defaultTestSuite` without calling `super` isn't filtered.
+- **Skipping some of a class's tests uses a private method.** XCTest has no public way to remove tests from a class's suite, so the package uses `XCTestSuite`'s private `-setTests:`, which keeps the suite and the class `setUp` it runs. If a future Xcode removes it, a class with some tests listed runs all of them; a class with all of its tests listed is still skipped. Either way, `skipped.tsv` lists only the tests that were removed. A class that overrides `defaultTestSuite` without calling `super` isn't filtered.
 - **UI tests record which app process each test launched, not the app's counters.** The app's coverage comes from the profile Xcode writes for each app process (`<uuid>-<pid>.profraw` beside the run's `Coverage.profdata`). The pid is read from `XCUIApplication`'s private `processID` property; if a future Xcode removes it, nothing is written. Processes the app starts itself, such as extensions, aren't listed.
 - **Images loaded late aren't observed.** An image loaded after the first test starts, such as a framework the tests `dlopen`, is missing from the records.
 - **The package's own code shows up in the records.** The package is instrumented like the code under test, so its functions appear in the records, and a reader should ignore them.

Sources/TestCoverageAttribution/CoverageAttributionTrait.swift

@@ -5,7 +5,7 @@ import Testing
 
 /// Attributes the code each Swift Testing test executes to that test, by marking where each test
 /// starts and ends: Swift Testing has no observation center a library can join. It also skips the
-/// tests the skip list names, by cancelling them before they run.
+/// tests the skip list names, by cancelling them before they run, and reports the ones it cancels.
 ///
 /// The trait has to be added to every suite, and Swift Testing runs tests in parallel within a
 /// process by default, which leaves overlapping tests without evidence.
@@ -38,6 +38,7 @@ public struct CoverageAttributionTrait: TestTrait, SuiteTrait, TestScoping {
                 // `Test.cancel` reports the test as skipped. Before Swift 6.3 a trait can only
                 // skip a test by not running it, which reports it as passed, so it runs instead.
                 if test_coverage_attribution_is_skipped(module, suite, name) {
+                    test_coverage_attribution_report_skipped(module, suite, name)
                     try Test.cancel("Listed in TEST_COVERAGE_ATTRIBUTION_SKIP_FILE", sourceLocation: test.sourceLocation)
                 }
             #endif