feat: keep tests marked unskippable from being skipped - #40

What

Test authors can mark XCTest and Swift Testing tests, or whole classes and suites, as unskippable. Those tests always run, even when TEST_COVERAGE_ATTRIBUTION_SKIP_FILE lists them.

Swift Testing. Two new traits, .unskippable and .skippable (SkippabilityTrait). They're recursive, and the most specific one wins: a test's trait overrides its suite's, and a nested suite's overrides its parent's. The .coverageAttribution trait checks them before it calls Test.cancel.

@Suite(.coverageAttribution, .unskippable)
struct FixtureTests {
    @Test func parsesTheFixtures() { … }            // always runs
    @Test(.skippable) func formatsTheDate() { … }   // can be skipped
}

XCTest. Class methods that the observer looks up by name, so the test target links nothing and the observer still never links XCTest:

The lists win over the class, and unskippableTests wins over skippableTests. Names take any form the skip list takes (testFoo, testFoo() or the selector). A subclass inherits these methods. A method with an unexpected return type is ignored (checked through its type encoding). From Swift, declare them with @objc static let …; from Objective-C, as + (BOOL)isUnskippable and the like.

Output for Tuist. Each unskippable test that runs is appended once per process to a new $TEST_COVERAGE_ATTRIBUTION_DIR/<pid>/unskippable.tsv, as module\tsuite\tname\tlisted. Names match records.bin, so an XCTest test is named by its selector. listed is 1 when the skip list named the test, which makes it a forced run. The README documents the format.

Why

Coverage-based selection only sees the code a test runs. Some tests depend on inputs coverage can't see: fixtures, resources, the network, the environment. Such a test can break without any covered code changing, so authors need a way to keep it from ever being skipped. DataDog's dd-sdk-swift-testing has the same feature (Tag.dd.tia.unskippable, tiaSkippable, and itr.forced_run for a listed test that ran anyway), and this follows its semantics: the most specific marker wins.

Alternatives considered

• A Swift Testing Tag instead of a trait. test.tags merges every level into a set, which loses which level declared a tag, and "most specific wins" needs that. A dedicated trait is also explicit about what it does. The trait relies on Swift Testing putting a suite's recursive traits before a child's own traits, outermost first (Runner.Plan._recursivelyApplyTraits), so the last SkippabilityTrait in test.traits is the most specific. The tests cover this, including a nested suite that opts back in.
• A parameter on .coverageAttribution (.coverageAttribution(skippable: false)). It needs a third state for "no opinion", so that a plain .coverageAttribution on a nested suite doesn't reset its parent. That's subtle, and it adds a second scope-providing trait to every marked test.
• A Swift protocol for XCTest classes. It would give compile-time checking, but an Objective-C class couldn't easily conform to it, and an XCTest-only target that doesn't import the package couldn't use it. Plain selector names work everywhere, including with the package linked statically through -ObjC. The cost: if you forget @objc in Swift, nothing happens. The README calls this out.
• A flag bit in records.bin instead of a new file. It would only work while recording, and Swift Testing would need a new scope_end signature. A separate TSV keeps records.bin and kRecordVersion unchanged, matches launches.tsv, and doesn't depend on counters.

Impact

• Without markers, nothing changes: no marker means skippable, as before.
• The marker is enforced in the test process, so an unskippable test runs even if a runner lists it.
• New public API: SkippabilityTrait, .unskippable, .skippable, and the C function test_coverage_attribution_report_unskippable. The scope-functions table that copies of the library share inside one process gains a third member.
• Tuist's reader isn't part of this PR. TuistKit.CoverageObserverOutput needs to parse unskippable.tsv, so selection can stop listing those tests and show forced runs. That's a follow-up in tuist/tuist.
• Overlap with the skipped-tests report (feat: report the tests the skip list actually skipped #39). Both change the cancel block in CoverageAttributionTrait, suite_without_skipped_tests, the observer's output list, SkipListTests.swift and the README. Where the two changes are the same plumbing (the output directory for the UI test fixture, skipListEnvironment, moving swiftPMCoverageReport() into an extension), I used the same text so it merges cleanly. Whichever lands second rebases. An unskippable listed test is neither removed nor cancelled, so it shouldn't appear in that report.
• This was meant to stack on feat: skip the Swift Testing tests a skip list names #38, but feat: skip the Swift Testing tests a skip list names #38 had already been merged into main (0.4.0) when this started, so this PR targets main.

Validation

• Fixtures/UITests (iOS simulator, SkipList scheme), serial, parallel and shard runs, checked against the result bundle and unskippable.tsv:
  • an unskippable XCTest method, listed by its result-bundle name and by its selector;
  • an unskippable XCTest class whose skippableTests test is still skipped (and the class setUp runs);
  • an unskippable UI test;
  • a Swift Testing .unskippable test;
  • an unskippable suite whose .skippable test, and whose .skippable nested suite, are still skipped, with an .unskippable test inside that nested suite that runs.
  • The listed tests without markers are still skipped.
• skipsTheListedSwiftTestingTests (SwiftPM, Fixtures/Example): an unskippable test, an unskippable nested suite and a .skippable opt-out, plus unskippable.tsv with both listed values. The #if compiler(>=6.3) expectations are kept.
• needsObjCToLinkTheObserverFromAStaticArchive: a Swift class (unskippableTests) and a new Objective-C class (isUnskippable plus skippableTests) in a bundle that links the observer from a static archive. With -ObjC the markers are honoured and reported; without it everything runs.
• Ran:
  • Xcode 26.6 (Swift 6.3.3): full swift test. Everything passed except skipsTheListedTests, which failed only on a wrong expectation of mine (XCTest names a throwing test testFoo, not by its AndReturnError: selector). After fixing it, skipsTheListedTests passed for serial, parallel and shard.
  • Xcode 26.3 (Swift 6.2.4): skipsTheListedTests (all three runs), skipsTheListedSwiftTestingTests, runsEveryTestWithoutASkipList and needsObjCToLinkTheObserverFromAStaticArchive passed.
  • mise run lint is clean (SwiftFormat, SwiftLint, clang static analyzer).
  • An earlier run failed with "Process spawn via launchd failed" / "Mach error -308" while another worktree used the same simulator. Those were infrastructure errors; the runs above had the simulator to themselves.

🤖 Generated with Claude Code