Skip to content

Commit 788f8ec

Browse files
grynspangrynspan
committed
[ST-NNNN] Data-dependent test serialization
1 parent 69dd6f6 commit 788f8ec

1 file changed

Lines changed: 366 additions & 0 deletions

File tree

‎proposals/testing/NNNN-serialization-dependencies.md‎

Lines changed: 366 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,366 @@
1+
# Data-dependent test serialization
2+
3+
* Proposal: [ST-NNNN](NNNN-serialization-dependencies.md)
4+
* Authors: [Jonathan Grynspan](https://github.com/grynspan)
5+
* Review Manager: TBD
6+
* Status: **Awaiting review**
7+
* Bug: rdar://135288463
8+
* Implementation: [swiftlang/swift-testing#1232](https://github.com/swiftlang/swift-testing/pull/1232)
9+
* Review: ([pre-pitch](https://forums.swift.org/t/pre-pitch-data-dependent-test-serialization/81251))
10+
([pitch](https://forums.swift.org/...))
11+
12+
## Introduction
13+
14+
This proposal introduces new variants of the `.serialized` trait to allow
15+
finer-grained control over test serialization/parallelization.
16+
17+
> [!NOTE]
18+
> This proposal uses the term "data dependency" to describe shared or global
19+
> mutable state that a test may rely upon. This term is unrelated to
20+
> "[dependency injection](https://en.wikipedia.org/wiki/Dependency_injection)",
21+
> a commonly-used pattern when writing tests, and this feature isn't directly
22+
> related to that pattern.
23+
24+
## Motivation
25+
26+
By default, Swift Testing runs all tests in parallel. We believe this is
27+
generally the right choice as it allows tests to complete more quickly and can
28+
help expose hidden dependencies between tests.
29+
30+
Some tests are dependent on shared/global mutable state like environment
31+
variables or singletons and cannot run serially. For example, if these two tests
32+
run in parallel, they may stomp on each other's state despite being valid in
33+
isolation:
34+
35+
```swift
36+
@Test func `Xterm color emulation`() {
37+
Environment.set("TERM", to: "xterm-256")
38+
#expect(Terminal.isColorEnabled)
39+
}
40+
41+
@Test func `VT-100 emulation`() {
42+
Environment.set("TERM", to: "vt100")
43+
#expect(!Terminal.isColorEnabled)
44+
}
45+
```
46+
47+
This proposal introduces new API to Swift Testing to let test authors document
48+
data dependencies like this one. Swift Testing can then order affected tests
49+
serially while still allowing other tests to run in parallel.
50+
51+
## Proposed solution
52+
53+
We propose introducing new overloads of the existing `.serialized` trait that
54+
take values describing data dependencies. When these overloads are applied to a
55+
test, that test will run serially with respect to other tests that share the
56+
same data dependency, ensuring that those tests do not interfere with each
57+
other.
58+
59+
### Deprecating the existing trait
60+
61+
The existing `.serialized` trait only applies serialization within the context
62+
of the test it is applied to. If it is applied to a suite, it serializes all
63+
test functions in that suite (including those recursively contained in nested
64+
test suites). If it is applied to a parameterized test function, it serializes
65+
all the test cases of that test function.
66+
67+
However, two unrelated test suites that both have the `.serialized` trait
68+
applied may still run in parallel with respect to _each other_. This behavior
69+
has proven surprising to test authors who expect `.serialized` to apply
70+
more-or-less globally. As such, we propose changing the behavior of the existing
71+
`.serialized` trait to match that of the new unbounded-data-dependency trait and
72+
to mark it to-be-deprecated. A change in behavior will not make the existing
73+
trait any less correct, but will allow it to behave the way test authors
74+
generally expect; deprecation will guide test authors toward the new trait whose
75+
name is more expressive.
76+
77+
## Detailed design
78+
79+
A new nested type is added to `ParallelizationTrait` which describes a data
80+
dependency:
81+
82+
```swift
83+
public struct Dependency: Sendable, CustomStringConvertible {}
84+
```
85+
86+
A new trait factory function is added to `ParallelizationTrait`:
87+
88+
```swift
89+
extension Trait where Self == ParallelizationTrait {
90+
/// Constructs a trait that describes a test's dependency on shared state
91+
/// using a key path.
92+
///
93+
/// - Parameters:
94+
/// - keyPath: The key path representing the dependency.
95+
///
96+
/// - Returns: An instance of ``ParallelizationTrait`` that marks any test it
97+
/// is applied to as dependent on `keyPath`.
98+
///
99+
/// Use this trait when you write a test function is dependent on global
100+
/// mutable state and you can describe that state using a key path.
101+
///
102+
/// ```swift
103+
/// @Test(.serialized(for: \FoodTruck.shared.freezer.door))
104+
/// func `Freezer door works`() {
105+
/// let freezer = FoodTruck.shared.freezer
106+
/// freezer.openDoor()
107+
/// #expect(freezer.isOpen)
108+
/// freezer.closeDoor()
109+
/// #expect(!freezer.isOpen)
110+
/// }
111+
/// ```
112+
///
113+
/// The testing library may combine dependencies represented by key paths with
114+
/// common prefixes. For example, the testing library treats the following key
115+
/// paths as equivalent for the purposes of serialization:
116+
///
117+
/// ```swift
118+
/// let first = \T.x[0]
119+
/// let second = \T.x[1]
120+
/// ```
121+
///
122+
/// ## See Also
123+
///
124+
/// - ``ParallelizationTrait``
125+
public static func serialized<R, V>(for keyPath: KeyPath<R, V>) -> Self
126+
}
127+
```
128+
129+
When applied to a test suite, this trait is recursively inherited by nested
130+
suites and test functions. When a test function has one of these traits applied
131+
to it, it runs in serial with respect to _all_ other tests in the same process
132+
that have the _same_ data dependency, but may run in parallel with tests that
133+
have other data dependencies. For example:
134+
135+
```swift
136+
@Test(.serialized(for: A)) func a1() {}
137+
@Test(.serialized(for: A)) func a2() {}
138+
@Test(.serialized(for: B)) func b() {}
139+
```
140+
141+
`a1()` and `a2()` will run serially with respect to each other, but `b()` is
142+
allowed to run in parallel with both `a1()` and `a2()`.
143+
144+
### Declaring an unbounded data dependency
145+
146+
Another overload of `.serialized` is added along with a corresponding typealias
147+
to describe its type, which can be referred to in argument position as `*`:
148+
149+
```swift
150+
extension ParallelizationTrait.Dependency {
151+
/// An unbounded dependency.
152+
///
153+
/// An unbounded dependency is a dependency on the complete state of the
154+
/// current process. To specify an unbounded dependency when using
155+
/// ``Trait/serialized(for:)-(Self.Unbounded)``, pass a reference
156+
/// to this function.
157+
///
158+
/// ```swift
159+
/// @Test(.serialized(for: *))
160+
/// func `All food truck environment variables`() { ... }
161+
/// ```
162+
///
163+
/// If a test has more than one dependency, the testing library automatically
164+
/// treats it as if it is dependent on the program's complete state.
165+
///
166+
/// ## See Also
167+
///
168+
/// - ``ParallelizationTrait``
169+
public static func *(/*...*/)
170+
171+
/// A type describing unbounded dependencies.
172+
///
173+
/// An unbounded dependency is a dependency on the complete state of the
174+
/// current process. To specify an unbounded dependency when using
175+
/// ``Trait/serialized(for:)-(Self.Dependency.Unbounded)``, pass a reference
176+
/// to the `*` operator.
177+
///
178+
/// ```swift
179+
/// @Test(.serialized(for: *))
180+
/// func `All food truck environment variables`() { ... }
181+
/// ```
182+
///
183+
/// If a test has more than one dependency, the testing library automatically
184+
/// treats it as if it is dependent on the program's complete state.
185+
///
186+
/// ## See Also
187+
///
188+
/// - ``ParallelizationTrait``
189+
public typealias Unbounded = /* ... */
190+
}
191+
192+
extension Trait where Self == ParallelizationTrait {
193+
/// Constructs a trait that describes a dependency on the complete state of
194+
/// the current process.
195+
///
196+
/// - Returns: An instance of ``ParallelizationTrait`` that adds a dependency
197+
/// on the complete state of the current process to any test it is applied
198+
/// to.
199+
///
200+
/// Pass `*` to ``serialized(for:)-(Self.Dependency.Unbounded)`` when you
201+
/// write a test function is dependent on global mutable state in the current
202+
/// process that cannot be fully described or that isn't known until runtime.
203+
///
204+
/// ```swift
205+
/// @Test(.serialized(for: *))
206+
/// func `All food truck environment variables`() { ... }
207+
/// ```
208+
///
209+
/// If a test has more than one dependency, the testing library automatically
210+
/// treats it as if it is dependent on the program's complete state.
211+
///
212+
/// ## See Also
213+
///
214+
/// - ``ParallelizationTrait``
215+
public static func serialized(for _: Self.Dependency.Unbounded) -> Self
216+
}
217+
```
218+
219+
And the existing `.serialized` trait is marked to-be-deprecated:
220+
221+
```swift
222+
extension Trait where Self == ParallelizationTrait {
223+
/// A trait that serializes the test to which it is applied.
224+
///
225+
/// ## See Also
226+
///
227+
/// - ``ParallelizationTrait``
228+
@available(swift, deprecated: 100000.0, renamed: "serialized(for: *)")
229+
public static var serialized: Self { get }
230+
}
231+
```
232+
233+
A test author can declare that a test has a data dependency on _all_ observable
234+
state in the program by writing `.serialized(for: *)`. This overload of the
235+
trait is useful when a test has complex requirements that cannot be fully
236+
described statically. For example:
237+
238+
```swift
239+
@Test(.serialized(for: *)) func monkey() {
240+
// https://www.folklore.org/Monkey_Lives.html
241+
let possibleActions = [
242+
writeToStandardError,
243+
readFromStandardInput,
244+
modifyEnvironment,
245+
// ...
246+
]
247+
for i in 0 ..< 1000 {
248+
let action = possibleActions.randomElement()
249+
action.perform()
250+
}
251+
}
252+
```
253+
254+
The `.serialized` trait's behavior will change to match that of
255+
`.serialized(for: *)` as described earlier in this proposal. In cases where the
256+
existing behavior is desireable for a given suite, any key path that isn't used
257+
elsewhere can be used as the data dependency to the same effect:
258+
259+
```swift
260+
private enum LocallySerialized {}
261+
262+
@Suite(.serialized(for: \LocallySerialized.self))
263+
struct S {
264+
// ...
265+
}
266+
```
267+
268+
### Declaring multiple data dependencies
269+
270+
If a test has multiple distinct data dependencies, it runs in serial with all
271+
other tests that have _any_ of those data dependencies. For example:
272+
273+
```swift
274+
@Test(.serialized(for: A)) func a() {}
275+
@Test(.serialized(for: A), .serialized(for: B)) func ab() {}
276+
@Test(.serialized(for: B)) func b() {}
277+
```
278+
279+
In this case, `a()` and `ab()` must run serially, and `b()` and `ab()` must run
280+
serially, but `a()` and `b()` can run in parallel with each other.
281+
282+
There is a class of deadlock bugs that can occur when tests have moderately
283+
complex interrelated data dependencies. For example:
284+
285+
```swift
286+
@Test(.serialized(for: A), .serialized(for: B)) func ab() {}
287+
@Test(.serialized(for: B), .serialized(for: C)) func bc() {}
288+
@Test(.serialized(for: C), .serialized(for: A)) func ca() {}
289+
```
290+
291+
The execution order for `ab()`, `bc()`, and `ca()` is unspecified, and it is
292+
possible for each of the three tests to end up scheduled to run after the others
293+
(i.e. a deadlock can occur). To avoid that deadlock, Swift Testing cuts the
294+
Gordian knot and treats any test with more than one data dependency as having an
295+
_unbounded_ data dependency instead. In this example, `ab()`, `bc()`, and `ca()`
296+
will run serially with respect to each other.
297+
298+
## Source compatibility
299+
300+
This change is additive only and does not affect source compatibility (but note
301+
again the change in behaviour for the existing `.serialized` trait).
302+
303+
## Integration with supporting tools
304+
305+
This change does not affect supporting tools or the JSON event stream schema.
306+
307+
## Future directions
308+
309+
- **Adding other kinds of data dependency.**
310+
We anticipate that key paths are sufficient to describe most, if not all, data
311+
dependencies. The community may find use for other "key" types, which we can
312+
evaluate on a case-by-case basis.
313+
314+
- **Formally deprecating `.serialized`.**
315+
A future proposal will move this trait from to-be-deprecated to formally
316+
deprecated.
317+
318+
## Alternatives considered
319+
320+
- **Leaving the behavior of `.serialized` unchanged.**
321+
This interface frequently confuses test authors who expect it to apply across
322+
all tests with the same trait. Changing it would resolve this issue for those
323+
test authors while not affecting the correctness of existing tests that use it
324+
(if they are affected, it implies a concurrency bug already exists in those
325+
tests).
326+
327+
- **Inferring data dependencies from source inspection.**
328+
In the general case, computing the set of data dependencies in a particular
329+
program is undecidable.
330+
331+
- **Using tags to describe data dependencies.**
332+
Tags have the benefit of providing a usable dependency namespace. However,
333+
tags on their own have no "magic powers", and using them here was confusing
334+
for users. A test author might declare one test with the trait
335+
`.serialized(for: .tag)` and another test with the trait `.tags(.tag)`, and
336+
then assume that the latter is serialized with respect to the former (when, in
337+
fact, simply _having_ a tag does not imply serialization).
338+
339+
- **Using a new tag-like type and macro to describe data dependencies.**
340+
We briefly considered creating a new `@Dependency` macro that behaved exactly
341+
the same way as `@Tag`, but produced an instance of `Dependency` instead of an
342+
instance of `Tag`. The only distinction between the two types would be that a
343+
test author could use one with `.serialized(for:)` and use the other with
344+
`.tags(_:)`. In practice, this would likely be just as confusing as using tags
345+
directly. Consider the following test functions which, _when read_, appear
346+
ambiguous even though one relies on a `Tag` and the other on a `Dependency`:
347+
```swift
348+
@Test(.tags(.usesEnvironment)) func f() { ... }
349+
@Test(.serialized(for: .usesEnvironment)) func g() { ... }
350+
```
351+
352+
- **Using Swift types to describe data dependencies.**
353+
Swift types could also provide a usable dependency namespace, but they proved
354+
confusing in their own way. If a test author declares a test with the trait
355+
`.serialized(for: T.self)` and then adds test functions in an extension to
356+
`T`, those test functions are not implied to be serialized with respect to
357+
each other or the earlier test.
358+
359+
- **Using unsafe pointers to describe data dependencies.**
360+
Pointers allow describing data dependencies on C and C++ API. For example, the
361+
global `stderr` and `environ` variables could be used describe the standard
362+
error stream and the process environment block, respectively. However, unsafe
363+
pointers are _unsafe_. Accessing any pointer originating in Swift begs the
364+
question of why you wouldn't just use a more idiomatic (and presumably safer)
365+
Swift API; accessing a pointer originating in C tends to generate errors about
366+
concurrency safety in Swift 6.

0 commit comments

Comments
 (0)