From 34236e153934cc7494c2b7519ab001dd6105b064 Mon Sep 17 00:00:00 2001 From: Eduardo Nunes Date: Mon, 9 Nov 2026 16:46:18 -0300 Subject: [PATCH] feat: keep tests marked unskippable from being skipped Coverage only sees the code a test runs, so a test whose real inputs are fixtures, resources, the network or the environment can break without any covered code changing. Test authors can now mark such tests so they always run, even when TEST_COVERAGE_ATTRIBUTION_SKIP_FILE lists them. - Swift Testing: `.unskippable` and `.skippable` traits (SkippabilityTrait), on a test or a suite. The most specific one wins, so a test or a nested suite can opt back in under an unskippable suite. The `.coverageAttribution` trait checks them before it cancels a listed test. - XCTest: class methods the observer looks up by name, so nothing links the package or XCTest: `unskippableTests`, `isUnskippable` and `skippableTests`. They work from Swift (`@objc static let`) and Objective-C, and with the observer linked from a static archive with -ObjC. - Each unskippable test that runs is listed in a new `unskippable.tsv` under $TEST_COVERAGE_ATTRIBUTION_DIR//, with whether the skip list named it (a forced run), so the runner that writes the list can stop listing it. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 7 +- .../ExampleTests/MultiplicationTests.swift | 22 +++ Fixtures/StaticArchive/ArchiveTests.swift | 5 + Fixtures/StaticArchive/ObjCArchiveTests.m | 25 ++++ Fixtures/StaticArchive/run.sh | 6 +- Fixtures/UITests/AppTests/SkipListTests.swift | 60 +++++++- .../UITests/SkipUITests/SkipListUITests.swift | 6 +- README.md | 44 ++++++ .../CoverageAttributionTrait.swift | 19 ++- .../SkippabilityTrait.swift | 39 +++++ .../TestCoverageAttributionObserver.m | 128 ++++++++++++++-- .../include/TestCoverageAttributionObserver.h | 4 + .../AttributionTests.swift | 49 ++++--- .../SkipListTests.swift | 137 +++++++++++++++--- 14 files changed, 488 insertions(+), 63 deletions(-) create mode 100644 Fixtures/StaticArchive/ObjCArchiveTests.m create mode 100644 Sources/TestCoverageAttribution/SkippabilityTrait.swift diff --git a/AGENTS.md b/AGENTS.md index d08db4a..d748114 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,12 +3,12 @@ A Swift package that test targets link to record which coverage counters each test moves (per-test coverage attribution). Tuist's CLI reduces the output to per-test evidence; the record layout is a contract with it (`TuistKit.CoverageObserverOutput` in tuist/tuist). ## Layout -- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests; it lives in this file because `-ObjC` loads only archive members that define a class. -- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed. +- `Sources/TestCoverageAttributionObserver` - Objective-C, Foundation only. XCTest is looked up at run time (class, protocol and selectors by name), so nothing links XCTest. It reads each instrumented image's `__llvm_prf_cnts` directly and never calls the LLVM profile runtime. Registers its XCTest observer from a constructor when the test bundle loads, and hooks `-[XCUIApplication launch]`/`-activate` once the bundle starts, to list the app processes UI tests launch. Its constructor also reads the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) and hooks `+[XCTestCase defaultTestSuite]` to remove the listed tests; it lives in this file because `-ObjC` loads only archive members that define a class. A class keeps listed tests it marks unskippable with class methods looked up by name (`unskippableTests`, `isUnskippable`, `skippableTests`), and the observer lists the unskippable tests that run in `unskippable.tsv`. +- `Sources/TestCoverageAttribution` - the Swift Testing trait (`.coverageAttribution`) that calls `test_coverage_attribution_scope_begin/end` around each test, and cancels the tests `test_coverage_attribution_is_skipped` finds in the skip list, unless the most specific `.unskippable`/`.skippable` trait (`SkippabilityTrait`) says they're unskippable; it reports those through `test_coverage_attribution_report_unskippable`. Cancelling needs `Test.cancel`, so it's compiled only with Swift 6.3 or later; before, listed tests run, since a test a trait doesn't run is reported as passed. - `Tests/TestCoverageAttributionTests` - runs the fixtures with attribution on and parses what the observer wrote: `Records.swift` mirrors the `records.bin` layout and `FunctionTable.swift` maps counters to function names as Tuist's reader does. - `Fixtures/Example` - with SwiftPM, xcodebuild on macOS and xcodebuild on an iOS simulator. - `Fixtures/Concurrency` - the overlapping, nested-trait and failure cases, in a package of their own since Swift Testing runs suites in parallel across the process. - - `Fixtures/StaticArchive` - links the observer from a static archive, with and without `-ObjC`. + - `Fixtures/StaticArchive` - links the observer from a static archive, with and without `-ObjC`, into a bundle with a Swift and an Objective-C test class. - `Fixtures/UITests` - an app, its UI tests and its hosted unit tests. It's a Tuist project (`Project.swift`), because a UI test needs an app target and SwiftPM can't build apps; `Launches.swift` reads `launches.tsv`, and `SkipListTests.swift` runs the `SkipList` scheme serially, in parallel and as a shard and reads the result bundle. ## Rules @@ -17,6 +17,7 @@ A Swift package that test targets link to record which coverage counters each te - Changing `records.bin`: update `write_record()`, `Records.swift`, the README and Tuist's reader together, and bump `kRecordVersion` when a record gains a field. - Changing `launches.tsv`: update `write_launch()`, `Launches.swift`, the README and Tuist's reader (`CoverageObserverOutput.parseLaunches`) together. - Changing the skip list format: update `load_skip_list()`, `SkipListTests.swift`, the README and Tuist's writer together. +- Changing `unskippable.tsv` or the XCTest marker names: update `write_unskippable()`/`xctest_is_unskippable()`, `SkipListTests.swift`, the README and Tuist's reader together. - No `unsafeFlags` in `Package.swift`: they make the package unusable as a versioned dependency. ## Commands diff --git a/Fixtures/Example/Tests/ExampleTests/MultiplicationTests.swift b/Fixtures/Example/Tests/ExampleTests/MultiplicationTests.swift index f112df4..ce3088f 100644 --- a/Fixtures/Example/Tests/ExampleTests/MultiplicationTests.swift +++ b/Fixtures/Example/Tests/ExampleTests/MultiplicationTests.swift @@ -11,4 +11,26 @@ struct MultiplicationTests { @Test("Multiplying by zero") func multiplyByZero() { #expect(Calculator.multiply(0, 3) == 0) } + + /// Runs even when the skip list names it. + @Test(.unskippable) func multiplyByOne() { + #expect(Calculator.multiply(1, 3) == 3) + } + + /// Its tests run even when the skip list names them, except the one that opts back in. Nested, + /// so `.serialized` keeps it from running at the same time as the other tests. + @Suite(.unskippable) + struct UnskippableMultiplicationTests { + @Test func multiplyNegatives() { + #expect(Calculator.multiply(-2, -3) == 6) + } + + @Test func multiplyByMinusOne() { + #expect(Calculator.multiply(-1, 3) == -3) + } + + @Test(.skippable) func square() { + #expect(Calculator.multiply(3, 3) == 9) + } + } } diff --git a/Fixtures/StaticArchive/ArchiveTests.swift b/Fixtures/StaticArchive/ArchiveTests.swift index c8bd498..fc553d4 100644 --- a/Fixtures/StaticArchive/ArchiveTests.swift +++ b/Fixtures/StaticArchive/ArchiveTests.swift @@ -6,6 +6,8 @@ func sum(_ lhs: Int, _ rhs: Int) -> Int { /// Links the observer from a static archive and references nothing in it. final class ArchiveTests: XCTestCase { + @objc static let unskippableTests = ["testUnskippable"] + func testSum() { XCTAssertEqual(sum(2, 3), 5) } @@ -14,4 +16,7 @@ final class ArchiveTests: XCTestCase { func testSkipped() { XCTAssertNil(ProcessInfo.processInfo.environment["TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"]) } + + /// Listed with `-ObjC` too, but unskippable, so it runs either way. + func testUnskippable() {} } diff --git a/Fixtures/StaticArchive/ObjCArchiveTests.m b/Fixtures/StaticArchive/ObjCArchiveTests.m new file mode 100644 index 0000000..085d5e0 --- /dev/null +++ b/Fixtures/StaticArchive/ObjCArchiveTests.m @@ -0,0 +1,25 @@ +@import XCTest; + +/// Declares its unskippable tests from Objective-C. With `-ObjC` every test is listed: they run but +/// `testSkippable`, which the class lets the list skip. +@interface ObjCArchiveTests : XCTestCase +@end + +@implementation ObjCArchiveTests + ++ (BOOL)isUnskippable { + return YES; +} + ++ (NSArray *)skippableTests { + return @[@"testSkippable"]; +} + +- (void)testUnskippable { +} + +- (void)testSkippable { + XCTAssertNil(NSProcessInfo.processInfo.environment[@"TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"]); +} + +@end diff --git a/Fixtures/StaticArchive/run.sh b/Fixtures/StaticArchive/run.sh index b0c3163..32b0c19 100755 --- a/Fixtures/StaticArchive/run.sh +++ b/Fixtures/StaticArchive/run.sh @@ -22,6 +22,9 @@ xcrun clang -c -fobjc-arc -fmodules -isysroot "$SDK" \ -I "$ROOT/Sources/TestCoverageAttributionObserver/include" \ "$ROOT/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m" -o "$BUILD/observer.o" ar rcs "$BUILD/libTestCoverageAttributionObserver.a" "$BUILD/observer.o" +# An Objective-C test class, in the bundle itself. +xcrun clang -c -fobjc-arc -fmodules -isysroot "$SDK" -F "$FRAMEWORKS" \ + "$FIXTURE/ObjCArchiveTests.m" -o "$BUILD/ObjCArchiveTests.o" /usr/libexec/PlistBuddy \ -c "Add :CFBundleIdentifier string dev.tuist.ArchiveTests" \ -c "Add :CFBundleExecutable string ArchiveTests" \ @@ -29,7 +32,8 @@ ar rcs "$BUILD/libTestCoverageAttributionObserver.a" "$BUILD/observer.o" "$BUILD/ArchiveTests.xctest/Contents/Info.plist" > /dev/null xcrun swiftc -sdk "$SDK" -F "$FRAMEWORKS" -I "$LIBRARIES" -L "$LIBRARIES" \ -profile-generate -profile-coverage-mapping -emit-library -Xlinker -bundle \ - "$FIXTURE/ArchiveTests.swift" -o "$BUILD/ArchiveTests.xctest/Contents/MacOS/ArchiveTests" \ + "$FIXTURE/ArchiveTests.swift" "$BUILD/ObjCArchiveTests.o" \ + -o "$BUILD/ArchiveTests.xctest/Contents/MacOS/ArchiveTests" \ -framework XCTest -L "$BUILD" -lTestCoverageAttributionObserver "$@" \ -Xlinker -rpath -Xlinker "$FRAMEWORKS" -Xlinker -rpath -Xlinker "$LIBRARIES" xcrun xctest "$BUILD/ArchiveTests.xctest" diff --git a/Fixtures/UITests/AppTests/SkipListTests.swift b/Fixtures/UITests/AppTests/SkipListTests.swift index bc0c99e..6003b94 100644 --- a/Fixtures/UITests/AppTests/SkipListTests.swift +++ b/Fixtures/UITests/AppTests/SkipListTests.swift @@ -3,14 +3,18 @@ import Testing import XCTest // Unit tests hosted in the app, run by `AttributionTests` with a skip list that names every test -// called `testSkipped…` and every test of `SkippedTests`. The listed tests pass when they run, so -// a run with or without the list succeeds and the result bundle tells which ran; only a class -// setUp that must not run crashes. +// called `testSkipped…`, every test of `SkippedTests` and every test marked unskippable. The listed +// tests pass when they run, so a run with or without the list succeeds and the result bundle tells +// which ran; only a class setUp that must not run crashes. -/// Some of its tests are listed: the rest run, after the class setUp. +/// Some of its tests are listed: the rest run, after the class setUp, and so does the listed one it +/// marks unskippable. final class PartlySkippedTests: XCTestCase { nonisolated(unsafe) static var classSetUpRan = false + /// Named as the skip list names it, and as its selector. + @objc static let unskippableTests = ["testUnskippable()", "testUnskippableThrowingAndReturnError:"] + override static func setUp() { super.setUp() classSetUpRan = true @@ -27,6 +31,34 @@ final class PartlySkippedTests: XCTestCase { /// Listed as `testSkippedAsync`; its selector is `testSkippedAsyncWithCompletionHandler:`. func testSkippedAsync() async {} + + func testUnskippable() { + XCTAssertTrue(Self.classSetUpRan) + } + + func testUnskippableThrowing() throws {} +} + +/// All of its tests are listed, but the class is unskippable: they run, after the class setUp, +/// except the one it lets the list skip. +final class UnskippableTests: XCTestCase { + nonisolated(unsafe) static var classSetUpRan = false + + @objc static let isUnskippable = true + @objc static let skippableTests = ["testSkippable"] + + override static func setUp() { + super.setUp() + classSetUpRan = true + } + + func testFirst() { + XCTAssertTrue(Self.classSetUpRan) + } + + func testSecond() {} + + func testSkippable() {} } /// All of its tests are listed: the class doesn't run, not even its class setUp. @@ -46,7 +78,7 @@ final class UnlistedTests: XCTestCase { } /// The trait cancels the listed tests from Swift 6.3 on, which reports them as skipped; before, -/// they run. +/// they run. A listed test marked unskippable runs. @Suite(.coverageAttribution) struct SwiftTestingTests { @Test func runs() {} @@ -54,6 +86,24 @@ struct SwiftTestingTests { @Test func skipped() {} @Test(arguments: [1, 2]) func skippedWithArguments(value _: Int) {} + + @Test(.unskippable) func unskippable() {} +} + +/// Every test is listed. The suite is unskippable, so they run, except the one that opts back in +/// and the tests of the nested suite that does. +@Suite(.coverageAttribution, .unskippable) +struct UnskippableSwiftTestingTests { + @Test func runs() {} + + @Test(.skippable) func skipped() {} + + @Suite(.skippable) + struct SkippableTests { + @Test func skipped() {} + + @Test(.unskippable) func unskippable() {} + } } /// Listed too, but without the trait nothing reads the list, so it runs. diff --git a/Fixtures/UITests/SkipUITests/SkipListUITests.swift b/Fixtures/UITests/SkipUITests/SkipListUITests.swift index 534275a..66ac6fa 100644 --- a/Fixtures/UITests/SkipUITests/SkipListUITests.swift +++ b/Fixtures/UITests/SkipUITests/SkipListUITests.swift @@ -1,10 +1,12 @@ import XCTest // UI tests run by `AttributionTests` with a skip list that names every test called `testSkipped` -// and every test of `SkippedUITests`. The listed tests pass when they run, so the result bundle +// or `testUnskippable` and every test of `SkippedUITests`. The listed tests pass when they run, so the result bundle // tells which ran; only a class setUp that must not run crashes. final class PartlySkippedUITests: XCTestCase { + @objc static let unskippableTests = ["testUnskippable"] + func testLaunch() { let app = XCUIApplication() app.launch() @@ -13,6 +15,8 @@ final class PartlySkippedUITests: XCTestCase { } func testSkipped() {} + + func testUnskippable() {} } /// All of its tests are listed: the class doesn't run, not even its class setUp. diff --git a/README.md b/README.md index 86f37a8..3ff1e0b 100644 --- a/README.md +++ b/README.md @@ -99,6 +99,49 @@ TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE="$PWD/skip.txt" xcodebuild test - **What a skipped test looks like.** It's removed from the suite XCTest builds for its class, so it doesn't run and isn't in the result bundle. A class whose tests are all removed doesn't run its class `setUp` and `tearDown`. A class with tests left runs them as usual, class `setUp` included. - **Swift Testing.** In a suite with the `.coverageAttribution` trait, a listed test is cancelled before it starts, every argument of a parameterized test included, and reported as skipped, whether tests run in parallel or not. The line names it as the records do: the Swift module, the innermost suite, and the test's display name when it declares one (`Applies a discount`), its function otherwise (`appliesDiscount()`). A display name with a `/` can't be listed. This needs Swift 6.3 (Xcode 26.4) or later: before, a trait can skip a test only by not running it, which reports it as passed, so the listed tests run. Listed tests in suites without the trait run too. +### Unskippable tests + +Coverage only sees the code a test runs. A test whose real inputs are something else, such as fixtures, resources, the network or the environment, can break without any covered code changing. Mark it unskippable and it always runs, even when the skip list names it. + +**Swift Testing.** Add `.unskippable` to a test, or to a suite to cover every test in it and its nested suites, and `.skippable` to opt a test or a nested suite back in. The most specific one wins: a test's trait overrides its suite's, and a nested suite's overrides its parent's. The suite still needs the `.coverageAttribution` trait, since it's what skips listed tests. + +```swift +@Suite(.coverageAttribution, .unskippable) +struct FixtureTests { + @Test func parsesTheFixtures() { … } // always runs + @Test(.skippable) func formatsTheDate() { … } // can be skipped +} +``` + +**XCTest.** Declare class methods with these names, which the package looks up at run time, so the test target needs nothing from the package. From Swift they need `@objc`. + +| Class method | Type | Meaning | +| --- | --- | --- | +| `unskippableTests` | `[String]` / `NSArray *` | The class's tests that always run. | +| `isUnskippable` | `Bool` / `BOOL` | Every test of the class always runs. | +| `skippableTests` | `[String]` / `NSArray *` | The tests the list can still skip in a class that's unskippable. | + +```swift +final class FixtureTests: XCTestCase { + @objc static let unskippableTests = ["testParsesTheFixtures"] +} + +final class NetworkTests: XCTestCase { + @objc static let isUnskippable = true + @objc static let skippableTests = ["testFormatsTheDate"] +} +``` + +```objc +@implementation NetworkTests ++ (BOOL)isUnskippable { return YES; } +@end +``` + +A test is named as in the skip list: `testParsesTheFixtures`, `testParsesTheFixtures()` or its selector. The lists win over the class: a test in `unskippableTests` always runs, a test in `skippableTests` can be skipped, and any other follows `isUnskippable`. A subclass inherits these methods and can override them. A method with a different return type is ignored. + +**What the runner sees.** An unskippable test runs and is reported as usual. When the package records, each unskippable test that runs is listed in [`unskippable.tsv`](#output), so a runner that writes the skip list can stop listing it. A listed unskippable test is a *forced run*. + ## Output The output is written under `$TEST_COVERAGE_ATTRIBUTION_DIR//`, one directory per test process: @@ -109,6 +152,7 @@ The output is written under `$TEST_COVERAGE_ATTRIBUTION_DIR//`, one directo | `.data`, `.names` | Image `n`'s raw `__llvm_prf_data` and `__llvm_prf_names` sections: what maps a counter to its function. | | `records.bin` | One record per scope. | | `launches.tsv` | One line per app process a UI test launched, written when `XCUIApplication.launch()` or `activate()` returns: the process's pid, the test bundle, the suite and the test, separated by tabs. A launch outside a test, such as in a class `setUp`, has an empty test name. A pid is listed once, with the scope that launched it, even when later tests activate the same process. Only written by a process that launches an app. | +| `unskippable.tsv` | One line per unskippable test that ran, written when it starts: the module, the suite and the test, named as in its record (an XCTest test as XCTest names it, `testParsesTheFixtures`), then `1` if the skip list named it (a forced run) or `0` if not, separated by tabs. A test is listed once per process. A name with a tab or a line break isn't written. Only written by a process that runs an unskippable test. | A scope is one of the following: - an XCTest test; diff --git a/Sources/TestCoverageAttribution/CoverageAttributionTrait.swift b/Sources/TestCoverageAttribution/CoverageAttributionTrait.swift index f8a3950..1b8fdab 100644 --- a/Sources/TestCoverageAttribution/CoverageAttributionTrait.swift +++ b/Sources/TestCoverageAttribution/CoverageAttributionTrait.swift @@ -5,7 +5,8 @@ import Testing /// Attributes the code each Swift Testing test executes to that test, by marking where each test /// starts and ends: Swift Testing has no observation center a library can join. It also skips the -/// tests the skip list names, by cancelling them before they run. +/// tests the skip list names, by cancelling them before they run, unless they're marked +/// `.unskippable` (``SkippabilityTrait``). /// /// The trait has to be added to every suite, and Swift Testing runs tests in parallel within a /// process by default, which leaves overlapping tests without evidence. @@ -34,13 +35,17 @@ public struct CoverageAttributionTrait: TestTrait, SuiteTrait, TestScoping { // The name the result bundle reports the test under, which is the test's identity // everywhere else: its display name when it declares one, its function otherwise. let name = test.displayName ?? components.last ?? test.name - #if compiler(>=6.3) - // `Test.cancel` reports the test as skipped. Before Swift 6.3 a trait can only - // skip a test by not running it, which reports it as passed, so it runs instead. - if test_coverage_attribution_is_skipped(module, suite, name) { + let isListed = test_coverage_attribution_is_skipped(module, suite, name) + if !SkippabilityTrait.isSkippable(test) { + // It runs, listed or not, and the output says so, for the runner to stop listing it. + test_coverage_attribution_report_unskippable(module, suite, name, isListed) + } else if isListed { + #if compiler(>=6.3) + // `Test.cancel` reports the test as skipped. Before Swift 6.3 a trait can only + // skip a test by not running it, which reports it as passed, so it runs instead. try Test.cancel("Listed in TEST_COVERAGE_ATTRIBUTION_SKIP_FILE", sourceLocation: test.sourceLocation) - } - #endif + #endif + } test_coverage_attribution_scope_begin(module, suite, name) defer { test_coverage_attribution_scope_end(module, suite, name) } try await Self.$isInScope.withValue(true) { diff --git a/Sources/TestCoverageAttribution/SkippabilityTrait.swift b/Sources/TestCoverageAttribution/SkippabilityTrait.swift new file mode 100644 index 0000000..99ab4b0 --- /dev/null +++ b/Sources/TestCoverageAttribution/SkippabilityTrait.swift @@ -0,0 +1,39 @@ +import Testing + +/// Whether the skip list (`TEST_COVERAGE_ATTRIBUTION_SKIP_FILE`) can skip a test: `.unskippable` +/// for a test whose real inputs coverage can't see, such as fixtures, resources, the network or +/// the environment, so it always runs, and `.skippable` to opt a test or suite back in under an +/// unskippable suite. +/// +/// On a suite it applies to every test in it and in its nested suites. The most specific one +/// wins: a test's overrides its suite's, and a nested suite's overrides its parent's. It takes +/// effect in suites with the `.coverageAttribution` trait, which is what skips listed tests. +public struct SkippabilityTrait: TestTrait, SuiteTrait { + /// Whether the skip list can skip the test. + public let isSkippable: Bool + + public var isRecursive: Bool { + true + } + + /// Whether the skip list can skip `test`, from the most specific trait that says. Swift + /// Testing puts a suite's recursive traits before its children's own, outermost first, so the + /// last one in `test.traits` is the most specific. + static func isSkippable(_ test: Test) -> Bool { + (test.traits.last { $0 is Self } as? Self)?.isSkippable ?? true + } +} + +extension Trait where Self == SkippabilityTrait { + /// The test, or every test in the suite, always runs, even when the skip list names it. See + /// ``SkippabilityTrait``. + public static var unskippable: Self { + Self(isSkippable: false) + } + + /// The skip list can skip the test, or the tests in the suite, under a suite marked + /// `.unskippable`. See ``SkippabilityTrait``. + public static var skippable: Self { + Self(isSkippable: true) + } +} diff --git a/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m b/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m index 116d61e..563cf92 100644 --- a/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m +++ b/Sources/TestCoverageAttributionObserver/TestCoverageAttributionObserver.m @@ -24,11 +24,13 @@ // .names the image's raw __llvm_prf_names section // records.bin one record per scope, see write_record() // launches.tsv one line per process a UI test launched, see record_launch() +// unskippable.tsv one line per unskippable test that ran, see write_unskippable() // // Skipping: $TEST_COVERAGE_ATTRIBUTION_SKIP_FILE (through xcodebuild, // TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE), an absolute path, names a file of tests not to // run: XCTest tests are removed before XCTest runs them, and the trait cancels Swift Testing tests. -// See load_skip_list(). It works with or without recording. +// See load_skip_list(). It works with or without recording. Tests marked unskippable always run, see +// xctest_is_unskippable() and the trait's `.unskippable`. #import #import "TestCoverageAttributionObserver.h" #include @@ -73,6 +75,9 @@ static NSString *current_suite; static NSString *current_test; static NSMutableSet *listed_launches; +static FILE *unskippable; +static bool unskippable_failed; +static NSMutableSet *reported_unskippable; static const uint8_t *section(const struct mach_header_64 *h, const char *name, unsigned long *size) { const uint8_t *p = getsectiondata(h, "__DATA", name, size); @@ -338,6 +343,41 @@ static void observe_launches(void) { }); } +// Unskippable tests: the tests marked to run even when the skip list names them, so a runner that +// writes the list can stop listing them. Each unskippable test that runs is appended to +// unskippable.tsv once, when it starts, as `module\tsuite\tname\tlisted`, named as its record +// names it; `listed` is 1 when the skip list names it, which makes the run a forced one, and 0 +// otherwise. A name with a tab or a line break isn't written, since it can't be told apart. +// Called with the lock held. +static void write_unskippable(const char *module, const char *suite, const char *name, bool listed) { + if (unskippable_failed || !out_dir[0] || !module || !suite || !name) return; + if (strpbrk(module, "\t\r\n") || strpbrk(suite, "\t\r\n") || strpbrk(name, "\t\r\n")) return; + NSString *key = [NSString stringWithFormat:@"%s\t%s\t%s", module, suite, name]; + if (!key || [reported_unskippable containsObject:key]) return; + if (!unskippable) { + char file[PATH_MAX]; + snprintf(file, sizeof file, "%s/unskippable.tsv", out_dir); + unskippable = fopen(file, "a"); + if (!unskippable) { + unskippable_failed = true; + return; + } + } + if (!reported_unskippable) reported_unskippable = [NSMutableSet set]; + [reported_unskippable addObject:key]; + if (fprintf(unskippable, "%s\t%s\t%s\t%d\n", module, suite, name, listed ? 1 : 0) < 0 || fflush(unskippable) != 0) { + fclose(unskippable); + unskippable = NULL; + unskippable_failed = true; + } +} + +static void report_unskippable(const char *module, const char *suite, const char *name, bool listed) { + pthread_mutex_lock(&lock); + write_unskippable(module, suite, name, listed); + pthread_mutex_unlock(&lock); +} + // A process records through one copy of the library, the first to load: a second copy (another // test bundle, or a framework that links it too) would observe the same tests twice. The owner // publishes its scope functions in the environment, keyed by the process, and the other copies @@ -345,9 +385,10 @@ static void observe_launches(void) { typedef struct { void (*begin)(const char *, const char *, const char *); void (*end)(const char *, const char *, const char *); + void (*report_unskippable)(const char *, const char *, const char *, bool); } scope_functions; -static const scope_functions own_functions = {scope_begin, scope_end}; +static const scope_functions own_functions = {scope_begin, scope_end, report_unskippable}; static const char *const kOwnerVariable = "TEST_COVERAGE_ATTRIBUTION_OWNER"; static const scope_functions *process_owner(void) { @@ -374,6 +415,11 @@ void test_coverage_attribution_scope_end(const char *module, const char *suite, if (functions) functions->end(module, suite, name); } +void test_coverage_attribution_report_unskippable(const char *module, const char *suite, const char *name, bool listed) { + const scope_functions *functions = owner(); + if (functions) functions->report_unskippable(module, suite, name, listed); +} + // XCTest's names: `-[Suite test]` for Objective-C, `-[Module.Suite test]` for Swift; the // selector of a throwing or asynchronous Swift test ends in `AndReturnError:` or // `WithCompletionHandler:`. The reader normalizes the selector; this only splits. @@ -412,14 +458,17 @@ static bool is_class_suite(NSString *name) { static NSSet *skipped_tests; static IMP original_default_test_suite; -static NSString *skip_identifier(NSString *module, NSString *suite, NSString *name) { +// A test's name without what tells its forms apart: `testAdd()`, `testAddAndReturnError:` and +// `testAddWithCompletionHandler:` are all `testAdd`. +static NSString *normalized_test_name(NSString *name) { for (NSString *suffix in @[@"()", @"AndReturnError:", @"WithCompletionHandler:"]) { - if ([name hasSuffix:suffix]) { - name = [name substringToIndex:name.length - suffix.length]; - break; - } + if ([name hasSuffix:suffix]) return [name substringToIndex:name.length - suffix.length]; } - return [NSString stringWithFormat:@"%@/%@/%@", module, suite, name]; + return name; +} + +static NSString *skip_identifier(NSString *module, NSString *suite, NSString *name) { + return [NSString stringWithFormat:@"%@/%@/%@", module, suite, normalized_test_name(name)]; } static void load_skip_list(void) { @@ -445,6 +494,52 @@ bool test_coverage_attribution_is_skipped(const char *module, const char *suite, return identifier && [skipped_tests containsObject:identifier]; } +// Unskippable XCTest tests: a class marks them with class methods the observer looks up by name, so +// the test target needs nothing from the package to declare them, from Swift (`@objc static let`) +// or Objective-C: +// +unskippableTests NSArray of NSString, tests that always run, in any form the skip list takes; +// +skippableTests NSArray of NSString, tests the skip list can skip under +isUnskippable; +// +isUnskippable BOOL, every test of the class always runs. +// The most specific wins: a test the lists name follows them, unskippableTests first, and any other +// follows the class. A subclass inherits them, and overrides them as usual. A method with another +// return type is ignored. + +// The class method's return type, or 0 when the class has no such method. +static char class_method_type(Class cls, SEL selector) { + Method method = class_getClassMethod(cls, selector); + if (!method) return 0; + char type[8] = {0}; + method_getReturnType(method, type, sizeof type); + return type[0]; +} + +static bool class_lists_test(Class cls, NSString *selector_name, NSString *name) { + SEL selector = NSSelectorFromString(selector_name); + if (class_method_type(cls, selector) != '@') return false; + id tests = ((id (*)(Class, SEL))objc_msgSend)(cls, selector); + if (![tests isKindOfClass:[NSArray class]]) return false; + for (id test in tests) { + if ([test isKindOfClass:[NSString class]] && [normalized_test_name(test) isEqualToString:name]) return true; + } + return false; +} + +static bool class_is_unskippable(Class cls) { + SEL selector = NSSelectorFromString(@"isUnskippable"); + // BOOL is `bool` on arm64 and `signed char` on x86_64. + char type = class_method_type(cls, selector); + if (type != 'B' && type != 'c') return false; + return ((BOOL (*)(Class, SEL))objc_msgSend)(cls, selector); +} + +static bool xctest_is_unskippable(Class cls, NSString *method) { + NSString *name = normalized_test_name(method); + if (!cls || !name) return false; + if (class_lists_test(cls, @"unskippableTests", name)) return true; + if (class_lists_test(cls, @"skippableTests", name)) return false; + return class_is_unskippable(cls); +} + // A class's suite holds its tests. When some are listed, the suite is kept and its tests replaced // through XCTestSuite's private -setTests:, since the suite is what runs the class setUp and // tearDown; an XCTest without it runs every test. When all are listed, an empty suite takes its @@ -459,7 +554,9 @@ static id suite_without_skipped_tests(Class test_class, id suite) { if ([test isKindOfClass:test_case]) { NSString *test_suite, *method; split_xctest_name([test name], &test_suite, &method); - if ([skipped_tests containsObject:skip_identifier(module, test_suite, method)]) continue; + if ([skipped_tests containsObject:skip_identifier(module, test_suite, method)] + && !xctest_is_unskippable([test class], method)) + continue; } [kept addObject:test]; } @@ -522,6 +619,19 @@ - (void)testCaseWillStart:(id)testCase { split_xctest_name([testCase name], &suite, &method); [self gapForSuite:suite]; [self setScopeSuite:suite test:method]; + [self reportIfUnskippable:testCase suite:suite method:method]; +} + +// The skip list is matched as when the test's class built its suite, with its bundle as the module. +- (void)reportIfUnskippable:(id)testCase suite:(NSString *)suite method:(NSString *)method { + @try { + Class test_class = [testCase class]; + if (!xctest_is_unskippable(test_class, method)) return; + NSString *module = [NSBundle bundleForClass:test_class].bundleURL.lastPathComponent.stringByDeletingPathExtension ?: @""; + bool listed = [skipped_tests containsObject:skip_identifier(module, suite, method)]; + report_unskippable(bundle_name, suite.UTF8String, method.UTF8String, listed); + } @catch (NSException *exception) { + } } - (void)testCaseDidFinish:(id)testCase { diff --git a/Sources/TestCoverageAttributionObserver/include/TestCoverageAttributionObserver.h b/Sources/TestCoverageAttributionObserver/include/TestCoverageAttributionObserver.h index 2aede35..077cd58 100644 --- a/Sources/TestCoverageAttributionObserver/include/TestCoverageAttributionObserver.h +++ b/Sources/TestCoverageAttributionObserver/include/TestCoverageAttributionObserver.h @@ -19,6 +19,10 @@ void test_coverage_attribution_scope_end(const char *module, const char *suite, /// XCTest tests are. bool test_coverage_attribution_is_skipped(const char *module, const char *suite, const char *name); +/// Lists the test in `unskippable.tsv`: it's marked to run even when the skip list names it, and +/// `listed` says whether the list does. Does nothing unless the process collects attribution. +void test_coverage_attribution_report_unskippable(const char *module, const char *suite, const char *name, bool listed); + #ifdef __cplusplus } #endif diff --git a/Tests/TestCoverageAttributionTests/AttributionTests.swift b/Tests/TestCoverageAttributionTests/AttributionTests.swift index 27612a7..dc8c810 100644 --- a/Tests/TestCoverageAttributionTests/AttributionTests.swift +++ b/Tests/TestCoverageAttributionTests/AttributionTests.swift @@ -67,7 +67,8 @@ struct AttributionTests { /// Linked from a static archive, as Tuist links packages by default, an XCTest-only target /// references nothing in the package, so the linker drops the observer unless the target links /// with `-ObjC`, which loads every archive member that defines an Objective-C class. The skip - /// list is read by the same member: with `-ObjC`, the listed test doesn't run. + /// list is read by the same member: with `-ObjC`, the listed test doesn't run, but the listed + /// tests a Swift or an Objective-C class marks unskippable do, and `unskippable.tsv` lists them. @Test(arguments: [false, true]) func needsObjCToLinkTheObserverFromAStaticArchive(objC: Bool) throws { let scratch = FileManager.default.temporaryDirectory.appending(path: "TestCoverageAttribution-\(UUID().uuidString)") @@ -78,7 +79,12 @@ struct AttributionTests { environment["TEST_COVERAGE_ATTRIBUTION_DIR"] = output.path() if objC { let skipList = scratch.appending(path: "skip.txt") - try "ArchiveTests/ArchiveTests/testSkipped()\n".write(to: skipList, atomically: true, encoding: .utf8) + try """ + ArchiveTests/ArchiveTests/testSkipped() + ArchiveTests/ArchiveTests/testUnskippable() + ArchiveTests/ObjCArchiveTests/testUnskippable + ArchiveTests/ObjCArchiveTests/testSkippable + """.write(to: skipList, atomically: true, encoding: .utf8) environment["TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"] = skipList.path() } @@ -93,6 +99,13 @@ struct AttributionTests { #expect((test != nil) == objC) #expect(test?.ran("3sum") ?? !objC) #expect(!records.contains { $0.name == "testSkipped" }) + let unskippable = records.filter { $0.kind == .xcTest && $0.name == "testUnskippable" }.map(\.suite).sorted() + #expect(unskippable == (objC ? ["ArchiveTests", "ObjCArchiveTests"] : [])) + #expect(!records.contains { $0.name == "testSkippable" }) + #expect(try Self.unskippableTests(in: output) == (objC ? [ + "ArchiveTests\tArchiveTests\ttestUnskippable\t1", + "ArchiveTests\tObjCArchiveTests\ttestUnskippable\t1", + ] : [])) } /// Two tests that run at the same time are both marked overlapped, including the one that @@ -189,21 +202,6 @@ struct AttributionTests { return try Record.all(in: output) } - /// The coverage report of the last SwiftPM run of `Fixtures/Example`. - private func swiftPMCoverageReport() throws -> Data { - let output = try Self.execute( - URL(filePath: "/usr/bin/xcrun"), - [ - "swift", "test", "--package-path", Self.packageRoot.appending(path: "Fixtures/Example").path(), - "--scratch-path", Self.packageRoot.appending(path: ".build/fixtures/Example/swiftpm").path(), - "--show-codecov-path", - ], - environment: Self.cleanEnvironment, - ) - let path = try #require(output.split(separator: "\n").last.map(String.init)) - return try Data(contentsOf: URL(filePath: path)) - } - /// This process's environment without the observer's variables, which the package's own test /// run may have set. static var cleanEnvironment: [String: String] { @@ -325,3 +323,20 @@ struct AttributionTests { ) } } + +extension AttributionTests { + /// The coverage report of the last SwiftPM run of `Fixtures/Example`. + private func swiftPMCoverageReport() throws -> Data { + let output = try Self.execute( + URL(filePath: "/usr/bin/xcrun"), + [ + "swift", "test", "--package-path", Self.packageRoot.appending(path: "Fixtures/Example").path(), + "--scratch-path", Self.packageRoot.appending(path: ".build/fixtures/Example/swiftpm").path(), + "--show-codecov-path", + ], + environment: Self.cleanEnvironment, + ) + let path = try #require(output.split(separator: "\n").last.map(String.init)) + return try Data(contentsOf: URL(filePath: path)) + } +} diff --git a/Tests/TestCoverageAttributionTests/SkipListTests.swift b/Tests/TestCoverageAttributionTests/SkipListTests.swift index c0ed63d..815e928 100644 --- a/Tests/TestCoverageAttributionTests/SkipListTests.swift +++ b/Tests/TestCoverageAttributionTests/SkipListTests.swift @@ -16,6 +16,11 @@ extension AttributionTests { /// with Swift 6.3 or later, the listed Swift Testing tests of a suite with the trait are /// skipped, every argument of a parameterized one included; before, they run. A listed Swift /// Testing test without the trait runs. + /// + /// Listed tests marked unskippable run, and `unskippable.tsv` lists them as listed: XCTest + /// tests a class names in `unskippableTests`, every test of a class with `isUnskippable` but + /// those it names in `skippableTests`, and Swift Testing tests marked `.unskippable` or in a + /// suite that is, unless a test or a nested suite opts back in with `.skippable`. @Test(arguments: SkipListRun.allCases) func skipsTheListedTests(run: SkipListRun) throws { let scratch = FileManager.default.temporaryDirectory.appending(path: "TestCoverageAttribution-\(UUID().uuidString)") @@ -28,56 +33,130 @@ extension AttributionTests { AppTests/PartlySkippedTests/testSkipped() AppTests/PartlySkippedTests/testSkippedThrowing() AppTests/PartlySkippedTests/testSkippedAsync + AppTests/PartlySkippedTests/testUnskippable() + AppTests/PartlySkippedTests/testUnskippableThrowing() AppTests/SkippedTests/testFirst() AppTests/SkippedTests/testSecond() + AppTests/UnskippableTests/testFirst() + AppTests/UnskippableTests/testSecond() + AppTests/UnskippableTests/testSkippable() AppTests/SwiftTestingTests/skipped() AppTests/SwiftTestingTests/skippedWithArguments(value:) + AppTests/SwiftTestingTests/unskippable() + AppTests/UnskippableSwiftTestingTests/runs() + AppTests/UnskippableSwiftTestingTests/skipped() + AppTests/SkippableTests/skipped() + AppTests/SkippableTests/unskippable() AppTests/UntraitedSwiftTestingTests/runs() AppTests/UnlistedTests AppTests/MissingTests/testMissing() SkipUITests/PartlySkippedUITests/testSkipped() + SkipUITests/PartlySkippedUITests/testUnskippable() SkipUITests/SkippedUITests/testSkipped() """.write(to: skipList, atomically: true, encoding: .utf8) let resultBundle = scratch.appending(path: "Tests.xcresult") - try runSkipListScheme(run, skipList: skipList, resultBundle: resultBundle) + let output = scratch.appending(path: "output") + try FileManager.default.createDirectory(at: output, withIntermediateDirectories: true) + try runSkipListScheme(run, skipList: skipList, resultBundle: resultBundle, output: output) + let results = try Self.testResults(in: resultBundle) + #expect(results.map(\.description).sorted() == Self.expectedResults(run)) + let unskippable = try Self.unskippableTests(in: output) + #expect(unskippable == Self.expectedUnskippable(run)) + } + + /// The tests `skipsTheListedTests` expects in the result bundle, and how they end. A shard runs + /// its `-only-testing` classes alone, and no Swift Testing test. + private static func expectedResults(_ run: SkipListRun) -> [String] { #if compiler(>=6.3) let listedSwiftTesting = "Skipped" #else let listedSwiftTesting = "Passed" #endif - let results = try Self.testResults(in: resultBundle) - #expect(results.map(\.description).sorted() == (run == .shard ? [ + var results = [ "AppTests/PartlySkippedTests/testRuns() Passed", + "AppTests/PartlySkippedTests/testUnskippable() Passed", + "AppTests/PartlySkippedTests/testUnskippableThrowing() Passed", + "AppTests/UnskippableTests/testFirst() Passed", + "AppTests/UnskippableTests/testSecond() Passed", "SkipUITests/PartlySkippedUITests/testLaunch() Passed", - ] : [ - "AppTests/PartlySkippedTests/testRuns() Passed", - "AppTests/SwiftTestingTests/runs() Passed", - "AppTests/SwiftTestingTests/skipped() \(listedSwiftTesting)", - "AppTests/SwiftTestingTests/skippedWithArguments(value:) \(listedSwiftTesting)", - "AppTests/UnlistedTests/testRuns() Passed", - "AppTests/UntraitedSwiftTestingTests/runs() Passed", - "SkipUITests/PartlySkippedUITests/testLaunch() Passed", - ])) + "SkipUITests/PartlySkippedUITests/testUnskippable() Passed", + ] + if run != .shard { + results += [ + "AppTests/SwiftTestingTests/runs() Passed", + "AppTests/SwiftTestingTests/skipped() \(listedSwiftTesting)", + "AppTests/SwiftTestingTests/skippedWithArguments(value:) \(listedSwiftTesting)", + "AppTests/SwiftTestingTests/unskippable() Passed", + "AppTests/UnlistedTests/testRuns() Passed", + "AppTests/UnskippableSwiftTestingTests/SkippableTests/skipped() \(listedSwiftTesting)", + "AppTests/UnskippableSwiftTestingTests/SkippableTests/unskippable() Passed", + "AppTests/UnskippableSwiftTestingTests/runs() Passed", + "AppTests/UnskippableSwiftTestingTests/skipped() \(listedSwiftTesting)", + "AppTests/UntraitedSwiftTestingTests/runs() Passed", + ] + } + return results.sorted() + } + + /// The lines `skipsTheListedTests` expects in `unskippable.tsv`, every test listed, named as + /// the records name them: XCTest tests as XCTest names them, without `()` or a bridged suffix, + /// and Swift Testing tests under their innermost suite. + private static func expectedUnskippable(_ run: SkipListRun) -> [String] { + var lines = [ + "AppTests\tPartlySkippedTests\ttestUnskippable\t1", + "AppTests\tPartlySkippedTests\ttestUnskippableThrowing\t1", + "AppTests\tUnskippableTests\ttestFirst\t1", + "AppTests\tUnskippableTests\ttestSecond\t1", + "SkipUITests\tPartlySkippedUITests\ttestUnskippable\t1", + ] + if run != .shard { + lines += [ + "AppTests\tSkippableTests\tunskippable()\t1", + "AppTests\tSwiftTestingTests\tunskippable()\t1", + "AppTests\tUnskippableSwiftTestingTests\truns()\t1", + ] + } + return lines.sorted() } /// Built with Swift 6.3 or later, the trait cancels a listed Swift Testing test before it - /// starts, so it leaves no record; before, it runs. The list names it by its display name. + /// starts, so it leaves no record; before, it runs. The list names it by its display name. A + /// test marked `.unskippable`, or in a suite that is, runs, unless it opts back in with + /// `.skippable`, and `unskippable.tsv` lists it, listed or not. @Test func skipsTheListedSwiftTestingTests() throws { + var unskippable: [String] = [] let records = try runFixture( .swiftPM, attributing: true, - skipList: "ExampleTests/MultiplicationTests/Multiplying by zero\n", - ) + skipList: """ + ExampleTests/MultiplicationTests/Multiplying by zero + ExampleTests/MultiplicationTests/multiplyByOne() + ExampleTests/UnskippableMultiplicationTests/multiplyNegatives() + ExampleTests/UnskippableMultiplicationTests/square() + """, + ) { unskippable = try Self.unskippableTests(in: $0) } let names = records.filter { $0.kind == .swiftTesting }.map(\.name) #expect(names.contains("multiply()")) + // Unskippable, the test or its suite: it runs. + #expect(names.contains("multiplyByOne()")) + #expect(names.contains("multiplyNegatives()")) + #expect(names.contains("multiplyByMinusOne()")) #if compiler(>=6.3) #expect(!names.contains("Multiplying by zero")) + // Opts back in under the unskippable suite. + #expect(!names.contains("square()")) #else #expect(names.contains("Multiplying by zero")) + #expect(names.contains("square()")) #endif + #expect(unskippable == [ + "ExampleTests\tMultiplicationTests\tmultiplyByOne()\t1", + "ExampleTests\tUnskippableMultiplicationTests\tmultiplyByMinusOne()\t0", + "ExampleTests\tUnskippableMultiplicationTests\tmultiplyNegatives()\t1", + ]) } /// Without a skip list, every test runs, those the list would name included. @@ -109,19 +188,20 @@ extension AttributionTests { "AppTests/PartlySkippedTests/testSkipped() Passed", "AppTests/PartlySkippedTests/testSkippedAsync() Passed", "AppTests/PartlySkippedTests/testSkippedThrowing() Passed", + "AppTests/PartlySkippedTests/testUnskippable() Passed", + "AppTests/PartlySkippedTests/testUnskippableThrowing() Passed", ]) } /// Generates `Fixtures/UITests` and runs its `SkipList` scheme on an iPhone simulator, the way - /// `run` says, with the skip list. - private func runSkipListScheme(_ run: SkipListRun, skipList: URL, resultBundle: URL) throws { + /// `run` says, with the skip list, writing its output to `output`. + private func runSkipListScheme(_ run: SkipListRun, skipList: URL, resultBundle: URL, output: URL) throws { let fixture = Self.packageRoot.appending(path: "Fixtures/UITests") let derivedData = Self.packageRoot.appending(path: ".build/fixtures/UITests/skip-list") let simulator = try Self.iPhoneSimulator(fixture: Self.packageRoot.appending(path: "Fixtures/Example")) try Self.execute(URL(filePath: "/usr/bin/xcrun"), ["simctl", "bootstatus", simulator, "-b"]) try Self.execute(URL(filePath: "/usr/bin/env"), ["tuist", "generate", "--no-open"], directory: fixture) - var environment = Self.cleanEnvironment - environment["TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE"] = skipList.path() + let environment = Self.skipListEnvironment(skipList: skipList, output: output) let destination = ["-destination", "id=\(simulator)"] let parallel = ["-parallel-testing-enabled", "YES", "-parallel-testing-worker-count", "2"] @@ -160,7 +240,7 @@ extension AttributionTests { ["xcodebuild", "test-without-building", "-xctestrun", products.appending(path: testRun).path()] + destination + ["-resultBundlePath", resultBundle.path(), "-quiet"] + parallel + [ "-only-testing:AppTests/PartlySkippedTests", "-only-testing:AppTests/SkippedTests", - "-only-testing:SkipUITests/PartlySkippedUITests", + "-only-testing:AppTests/UnskippableTests", "-only-testing:SkipUITests/PartlySkippedUITests", ], environment: environment, directory: fixture, @@ -169,10 +249,27 @@ extension AttributionTests { } } + /// The environment xcodebuild passes the skip list and the output directory to the test processes in. + private static func skipListEnvironment(skipList: URL, output: URL) -> [String: String] { + cleanEnvironment.merging([ + "TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_SKIP_FILE": skipList.path(), + "TEST_RUNNER_TEST_COVERAGE_ATTRIBUTION_DIR": output.path(), + ]) { $1 } + } + /// How long one xcodebuild invocation may take before it is stopped and its output reported, /// well under the suite's time limit. static let xcodebuildTimeout: TimeInterval = 360 + /// The lines of every process's `unskippable.tsv` under `output`, sorted. + static func unskippableTests(in output: URL) throws -> [String] { + try FileManager.default.contentsOfDirectory(at: output, includingPropertiesForKeys: nil).flatMap { process in + let file = process.appending(path: "unskippable.tsv") + guard FileManager.default.fileExists(atPath: file.path()) else { return [String]() } + return try String(contentsOf: file, encoding: .utf8).split(separator: "\n").map(String.init) + }.sorted() + } + /// A test case in a result bundle: `module/suite/name`, and how it ended. struct TestResult: CustomStringConvertible { let identifier: String