feat: keep tests marked unskippable from being skipped

AGENTS.md

@@ -3,12 +3,12 @@
 A Swift package that test targets link to record which coverage counters each test moves (per-test coverage attribution). Tuist's CLI reduces the output to per-test evidence; the record layout is a contract with it (`TuistKit.CoverageObserverOutput` in tuist/tuist).
 
 ## Layout
-- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests; it lives in this file because `-ObjC` loads only archive members that define a class.
-- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed.
+- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests; it lives in this file because `-ObjC` loads only archive members that define a class. A class keeps listed tests it marks unskippable with class methods looked up by name (`unskippableTests`, `isUnskippable`, `skippableTests`), and the observer lists the unskippable tests that run in `unskippable.tsv`.
+- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list, unless the most specific `.unskippable`/`.skippable` trait (`SkippabilityTrait`) says they're unskippable; it reports those through `test_coverage_attribution_report_unskippable`. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed.
 - `Tests/TestCoverageAttributionTests` - runs the fixtures with attribution on and parses what the observer wrote: `Records.swift` mirrors the `records.bin` layout and `FunctionTable.swift` maps counters to function names as Tuist's reader does.
   - `Fixtures/Example` - with SwiftPM, xcodebuild on macOS and xcodebuild on an iOS simulator.
   - `Fixtures/Concurrency` - the overlapping, nested-trait and failure cases, in a package of their own since Swift Testing runs suites in parallel across the process.
-  - `Fixtures/StaticArchive` - links the observer from a static archive, with and without `-ObjC`.
+  - `Fixtures/StaticArchive` - links the observer from a static archive, with and without `-ObjC`, into a bundle with a Swift and an Objective-C test class.
   - `Fixtures/UITests` - an app, its UI tests and its hosted unit tests. It's a Tuist project (`Project.swift`), because a UI test needs an app target and SwiftPM can't build apps; `Launches.swift` reads `launches.tsv`, and `SkipListTests.swift` runs the `SkipList` scheme serially, in parallel and as a shard and reads the result bundle.
 
 ## Rules
@@ -17,6 +17,7 @@ A Swift package that test targets link to record which coverage counters each te
 - Changing `records.bin`: update `write_record()`, `Records.swift`, the README and Tuist's reader together, and bump `kRecordVersion` when a record gains a field.
 - Changing `launches.tsv`: update `write_launch()`, `Launches.swift`, the README and Tuist's reader (`CoverageObserverOutput.parseLaunches`) together.
 - Changing the skip list format: update `load_skip_list()`, `SkipListTests.swift`, the README and Tuist's writer together.
+- Changing `unskippable.tsv` or the XCTest marker names: update `write_unskippable()`/`xctest_is_unskippable()`, `SkipListTests.swift`, the README and Tuist's reader together.
 - No `unsafeFlags` in `Package.swift`: they make the package unusable as a versioned dependency.
 
 ## Commands

Fixtures/Example/Tests/ExampleTests/MultiplicationTests.swift

@@ -11,4 +11,26 @@ struct MultiplicationTests {
     @Test("Multiplying by zero") func multiplyByZero() {
         #expect(Calculator.multiply(0, 3) == 0)
     }
+
+    /// Runs even when the skip list names it.
+    @Test(.unskippable) func multiplyByOne() {
+        #expect(Calculator.multiply(1, 3) == 3)
+    }
+
+    /// Its tests run even when the skip list names them, except the one that opts back in. Nested,
+    /// so `.serialized` keeps it from running at the same time as the other tests.
+    @Suite(.unskippable)
+    struct UnskippableMultiplicationTests {
+        @Test func multiplyNegatives() {
+            #expect(Calculator.multiply(-2, -3) == 6)
+        }
+
+        @Test func multiplyByMinusOne() {
+            #expect(Calculator.multiply(-1, 3) == -3)
+        }
+
+        @Test(.skippable) func square() {
+            #expect(Calculator.multiply(3, 3) == 9)
+        }
+    }
 }

Fixtures/StaticArchive/ArchiveTests.swift

@@ -6,6 +6,8 @@ func sum(_ lhs: Int, _ rhs: Int) -> Int {
 
 /// Links the observer from a static archive and references nothing in it.
 final class ArchiveTests: XCTestCase {
+    @objc static let unskippableTests = ["testUnskippable"]
+
     func testSum() {
         XCTAssertEqual(sum(2, 3), 5)
     }
@@ -14,4 +16,7 @@ final class ArchiveTests: XCTestCase {
     func testSkipped() {
         XCTAssertNil(ProcessInfo.processInfo.environment["TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"])
     }
+
+    /// Listed with `-ObjC` too, but unskippable, so it runs either way.
+    func testUnskippable() {}
 }

Fixtures/StaticArchive/ObjCArchiveTests.m

@@ -0,0 +1,25 @@
+@import XCTest;
+
+/// Declares its unskippable tests from Objective-C. With `-ObjC` every test is listed: they run but
+/// `testSkippable`, which the class lets the list skip.
+@interface ObjCArchiveTests : XCTestCase
+@end
+
+@implementation ObjCArchiveTests
+
++ (BOOL)isUnskippable {
+    return YES;
+}
+
++ (NSArray<NSString *> *)skippableTests {
+    return @[@"testSkippable"];
+}
+
+- (void)testUnskippable {
+}
+
+- (void)testSkippable {
+    XCTAssertNil(NSProcessInfo.processInfo.environment[@"TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"]);
+}
+
+@end

Fixtures/StaticArchive/run.sh

@@ -22,14 +22,18 @@ xcrun clang -c -fobjc-arc -fmodules -isysroot "$SDK" \
     -I "$ROOT/Sources/TestCoverageAttributionObserver/include" \
     "$ROOT/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m" -o "$BUILD/observer.o"
 ar rcs "$BUILD/libTestCoverageAttributionObserver.a" "$BUILD/observer.o"
+# An Objective-C test class, in the bundle itself.
+xcrun clang -c -fobjc-arc -fmodules -isysroot "$SDK" -F "$FRAMEWORKS" \
+    "$FIXTURE/ObjCArchiveTests.m" -o "$BUILD/ObjCArchiveTests.o"
 /usr/libexec/PlistBuddy \
     -c "Add :CFBundleIdentifier string dev.tuist.ArchiveTests" \
     -c "Add :CFBundleExecutable string ArchiveTests" \
     -c "Add :CFBundlePackageType string BNDL" \
     "$BUILD/ArchiveTests.xctest/Contents/Info.plist" > /dev/null
 xcrun swiftc -sdk "$SDK" -F "$FRAMEWORKS" -I "$LIBRARIES" -L "$LIBRARIES" \
     -profile-generate -profile-coverage-mapping -emit-library -Xlinker -bundle \
-    "$FIXTURE/ArchiveTests.swift" -o "$BUILD/ArchiveTests.xctest/Contents/MacOS/ArchiveTests" \
+    "$FIXTURE/ArchiveTests.swift" "$BUILD/ObjCArchiveTests.o" \
+    -o "$BUILD/ArchiveTests.xctest/Contents/MacOS/ArchiveTests" \
     -framework XCTest -L "$BUILD" -lTestCoverageAttributionObserver "$@" \
     -Xlinker -rpath -Xlinker "$FRAMEWORKS" -Xlinker -rpath -Xlinker "$LIBRARIES"
 xcrun xctest "$BUILD/ArchiveTests.xctest"

Fixtures/UITests/AppTests/SkipListTests.swift

@@ -3,14 +3,18 @@ import Testing
 import XCTest
 
 // Unit tests hosted in the app, run by `AttributionTests` with a skip list that names every test
-// called `testSkipped…` and every test of `SkippedTests`. The listed tests pass when they run, so
-// a run with or without the list succeeds and the result bundle tells which ran; only a class
-// setUp that must not run crashes.
+// called `testSkipped…`, every test of `SkippedTests` and every test marked unskippable. The listed
+// tests pass when they run, so a run with or without the list succeeds and the result bundle tells
+// which ran; only a class setUp that must not run crashes.
 
-/// Some of its tests are listed: the rest run, after the class setUp.
+/// Some of its tests are listed: the rest run, after the class setUp, and so does the listed one it
+/// marks unskippable.
 final class PartlySkippedTests: XCTestCase {
     nonisolated(unsafe) static var classSetUpRan = false
 
+    /// Named as the skip list names it, and as its selector.
+    @objc static let unskippableTests = ["testUnskippable()", "testUnskippableThrowingAndReturnError:"]
+
     override static func setUp() {
         super.setUp()
         classSetUpRan = true
@@ -27,6 +31,34 @@ final class PartlySkippedTests: XCTestCase {
 
     /// Listed as `testSkippedAsync`; its selector is `testSkippedAsyncWithCompletionHandler:`.
     func testSkippedAsync() async {}
+
+    func testUnskippable() {
+        XCTAssertTrue(Self.classSetUpRan)
+    }
+
+    func testUnskippableThrowing() throws {}
+}
+
+/// All of its tests are listed, but the class is unskippable: they run, after the class setUp,
+/// except the one it lets the list skip.
+final class UnskippableTests: XCTestCase {
+    nonisolated(unsafe) static var classSetUpRan = false
+
+    @objc static let isUnskippable = true
+    @objc static let skippableTests = ["testSkippable"]
+
+    override static func setUp() {
+        super.setUp()
+        classSetUpRan = true
+    }
+
+    func testFirst() {
+        XCTAssertTrue(Self.classSetUpRan)
+    }
+
+    func testSecond() {}
+
+    func testSkippable() {}
 }
 
 /// All of its tests are listed: the class doesn't run, not even its class setUp.
@@ -46,14 +78,32 @@ final class UnlistedTests: XCTestCase {
 }
 
 /// The trait cancels the listed tests from Swift 6.3 on, which reports them as skipped; before,
-/// they run.
+/// they run. A listed test marked unskippable runs.
 @Suite(.coverageAttribution)
 struct SwiftTestingTests {
     @Test func runs() {}
 
     @Test func skipped() {}
 
     @Test(arguments: [1, 2]) func skippedWithArguments(value _: Int) {}
+
+    @Test(.unskippable) func unskippable() {}
+}
+
+/// Every test is listed. The suite is unskippable, so they run, except the one that opts back in
+/// and the tests of the nested suite that does.
+@Suite(.coverageAttribution, .unskippable)
+struct UnskippableSwiftTestingTests {
+    @Test func runs() {}
+
+    @Test(.skippable) func skipped() {}
+
+    @Suite(.skippable)
+    struct SkippableTests {
+        @Test func skipped() {}
+
+        @Test(.unskippable) func unskippable() {}
+    }
 }
 
 /// Listed too, but without the trait nothing reads the list, so it runs.

Fixtures/UITests/SkipUITests/SkipListUITests.swift

@@ -1,10 +1,12 @@
 import XCTest
 
 // UI tests run by `AttributionTests` with a skip list that names every test called `testSkipped`
-// and every test of `SkippedUITests`. The listed tests pass when they run, so the result bundle
+// or `testUnskippable` and every test of `SkippedUITests`. The listed tests pass when they run, so the result bundle
 // tells which ran; only a class setUp that must not run crashes.
 
 final class PartlySkippedUITests: XCTestCase {
+    @objc static let unskippableTests = ["testUnskippable"]
+
     func testLaunch() {
         let app = XCUIApplication()
         app.launch()
@@ -13,6 +15,8 @@ final class PartlySkippedUITests: XCTestCase {
     }
 
     func testSkipped() {}
+
+    func testUnskippable() {}
 }
 
 /// All of its tests are listed: the class doesn't run, not even its class setUp.

README.md

@@ -99,6 +99,49 @@ TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE="$PWD/skip.txt" xcodebuild test
 - **What a skipped test looks like.** It's removed from the suite XCTest builds for its class, so it doesn't run and isn't in the result bundle. A class whose tests are all removed doesn't run its class `setUp` and `tearDown`. A class with tests left runs them as usual, class `setUp` included.
 - **Swift Testing.** In a suite with the `.coverageAttribution` trait, a listed test is cancelled before it starts, every argument of a parameterized test included, and reported as skipped, whether tests run in parallel or not. The line names it as the records do: the Swift module, the innermost suite, and the test's display name when it declares one (`Applies a discount`), its function otherwise (`appliesDiscount()`). A display name with a `/` can't be listed. This needs Swift 6.3 (Xcode 26.4) or later: before, a trait can skip a test only by not running it, which reports it as passed, so the listed tests run. Listed tests in suites without the trait run too.
 
+### Unskippable tests
+
+Coverage only sees the code a test runs. A test whose real inputs are something else, such as fixtures, resources, the network or the environment, can break without any covered code changing. Mark it unskippable and it always runs, even when the skip list names it.
+
+**Swift Testing.** Add `.unskippable` to a test, or to a suite to cover every test in it and its nested suites, and `.skippable` to opt a test or a nested suite back in. The most specific one wins: a test's trait overrides its suite's, and a nested suite's overrides its parent's. The suite still needs the `.coverageAttribution` trait, since it's what skips listed tests.
+
+```swift
+@Suite(.coverageAttribution, .unskippable)
+struct FixtureTests {
+    @Test func parsesTheFixtures() { … }               // always runs
+    @Test(.skippable) func formatsTheDate() { … }      // can be skipped
+}
+```
+
+**XCTest.** Declare class methods with these names, which the package looks up at run time, so the test target needs nothing from the package. From Swift they need `@objc`.
+
+| Class method | Type | Meaning |
+| --- | --- | --- |
+| `unskippableTests` | `[String]` / `NSArray<NSString *> *` | The class's tests that always run. |
+| `isUnskippable` | `Bool` / `BOOL` | Every test of the class always runs. |
+| `skippableTests` | `[String]` / `NSArray<NSString *> *` | The tests the list can still skip in a class that's unskippable. |
+
+```swift
+final class FixtureTests: XCTestCase {
+    @objc static let unskippableTests = ["testParsesTheFixtures"]
+}
+
+final class NetworkTests: XCTestCase {
+    @objc static let isUnskippable = true
+    @objc static let skippableTests = ["testFormatsTheDate"]
+}
+```
+
+```objc
+@implementation NetworkTests
++ (BOOL)isUnskippable { return YES; }
+@end
+```
+
+A test is named as in the skip list: `testParsesTheFixtures`, `testParsesTheFixtures()` or its selector. The lists win over the class: a test in `unskippableTests` always runs, a test in `skippableTests` can be skipped, and any other follows `isUnskippable`. A subclass inherits these methods and can override them. A method with a different return type is ignored.
+
+**What the runner sees.** An unskippable test runs and is reported as usual. When the package records, each unskippable test that runs is listed in [`unskippable.tsv`](#output), so a runner that writes the skip list can stop listing it. A listed unskippable test is a *forced run*.
+
 ## Output
 
 The output is written under `$TEST_COVERAGE_ATTRIBUTION_DIR/<pid>/`, one directory per test process:
@@ -109,6 +152,7 @@ The output is written under `$TEST_COVERAGE_ATTRIBUTION_DIR/<pid>/`, one directo
 | `<n>.data`, `<n>.names` | Image `n`'s raw `__llvm_prf_data` and `__llvm_prf_names` sections: what maps a counter to its function. |
 | `records.bin` | One record per scope. |
 | `launches.tsv` | One line per app process a UI test launched, written when `XCUIApplication.launch()` or `activate()` returns: the process's pid, the test bundle, the suite and the test, separated by tabs. A launch outside a test, such as in a class `setUp`, has an empty test name. A pid is listed once, with the scope that launched it, even when later tests activate the same process. Only written by a process that launches an app. |
+| `unskippable.tsv` | One line per unskippable test that ran, written when it starts: the module, the suite and the test, named as in its record (an XCTest test as XCTest names it, `testParsesTheFixtures`), then `1` if the skip list named it (a forced run) or `0` if not, separated by tabs. A test is listed once per process. A name with a tab or a line break isn't written. Only written by a process that runs an unskippable test. |
 
 A scope is one of the following:
 - an XCTest test;

Sources/TestCoverageAttribution/CoverageAttributionTrait.swift

@@ -5,7 +5,8 @@ import Testing
 
 /// Attributes the code each Swift Testing test executes to that test, by marking where each test
 /// starts and ends: Swift Testing has no observation center a library can join. It also skips the
-/// tests the skip list names, by cancelling them before they run.
+/// tests the skip list names, by cancelling them before they run, unless they're marked
+/// `.unskippable` (``SkippabilityTrait``).
 ///
 /// The trait has to be added to every suite, and Swift Testing runs tests in parallel within a
 /// process by default, which leaves overlapping tests without evidence.
@@ -34,13 +35,17 @@ public struct CoverageAttributionTrait: TestTrait, SuiteTrait, TestScoping {
             // The name the result bundle reports the test under, which is the test's identity
             // everywhere else: its display name when it declares one, its function otherwise.
             let name = test.displayName ?? components.last ?? test.name
-            #if compiler(>=6.3)
-                // `Test.cancel` reports the test as skipped. Before Swift 6.3 a trait can only
-                // skip a test by not running it, which reports it as passed, so it runs instead.
-                if test_coverage_attribution_is_skipped(module, suite, name) {
+            let isListed = test_coverage_attribution_is_skipped(module, suite, name)
+            if !SkippabilityTrait.isSkippable(test) {
+                // It runs, listed or not, and the output says so, for the runner to stop listing it.
+                test_coverage_attribution_report_unskippable(module, suite, name, isListed)
+            } else if isListed {
+                #if compiler(>=6.3)
+                    // `Test.cancel` reports the test as skipped. Before Swift 6.3 a trait can only
+                    // skip a test by not running it, which reports it as passed, so it runs instead.
                     try Test.cancel("Listed in TEST_COVERAGE_ATTRIBUTION_SKIP_FILE", sourceLocation: test.sourceLocation)
-                }
-            #endif
+                #endif
+            }
             test_coverage_attribution_scope_begin(module, suite, name)
             defer { test_coverage_attribution_scope_end(module, suite, name) }
             try await Self.$isInScope.withValue(true) {