Skip to content

Commit 9209cf3

Browse files
grynspangrynspan
committed
Rethink how we capture expectation conditions and their subexpressions.
This PR completely rewrites how we capture expectation conditions. For example, given the following expectation: ```swift ``` We currently detect that there is a binary operation and emit code that calls the binary operator as a closure and passes the left-hand value and right-hand value, then checks that the result of the operation is `true`. This is sufficient for simpler expressions like that one, but more complex ones (including any that involve `try` or `await` keywords) cannot be expanded correctly. With this PR, such expressions _can_ generally be expanded correctly. The change involves rewriting the macro condition as a closure to which is passed a local, mutable "context" value. Subexpressions of the condition expression are then rewritten by walking the syntax tree of the expression (using typical swift-syntax API) and replacing them with calls into the context value that pass in the value and related state. If the expectation ultimately fails, the collected data is transformed into an instance of the SPI type `Expression` that contains the source code of the expression and interesting subexpressions as well as the runtime values of those subexpressions. Nodes in the syntax tree are identified by a unique ID which is composed of the swift-syntax ID for that node as well as all its parent nodes in a compact bitmask format. These IDs can be transformed into graph/trie key paths when expression/subexpression relationships need to be reconstructed on failure, meaning that a single rewritten node doesn't otherwise need to know its "place" in the overall expression. There remain a few caveats (that also generally affect the current implementation): - Mutating member functions are syntactically indistinguishable from non-mutating ones and miscompile when rewritten; - Expressions involving move-only types are also indistinguishable, but need lifetime management to be rewritten correctly; and - Expressions where the `try` or `await` keyword is _outside_ the `#expect` macro cannot be expanded correctly because the macro cannot see those keywords during expansion. The first issue might be resolvable in the future using pointer tricks, although I don't hold a lot of hope for it. The second issue is probably resolved by non-escaping types. The third issue is an area of active exploration for us and the macros/swift-syntax team.
1 parent fd29262 commit 9209cf3

44 files changed

Lines changed: 2552 additions & 1949 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Package.swift‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,13 @@ let package = Package(
186186
path: "Tests/_MemorySafeTestingTests",
187187
swiftSettings: .packageSettings + [.strictMemorySafety()]
188188
),
189+
.testTarget(
190+
name: "SubexpressionShowcase",
191+
dependencies: [
192+
"Testing",
193+
],
194+
swiftSettings: .packageSettings
195+
),
189196

190197
.macro(
191198
name: "TestingMacros",
@@ -393,6 +400,8 @@ extension Array where Element == PackageDescription.SwiftSetting {
393400

394401
.enableUpcomingFeature("InferIsolatedConformances"),
395402

403+
.enableExperimentalFeature("Lifetimes"),
404+
396405
// When building as a package, the macro plugin always builds as an
397406
// executable rather than a library.
398407
.define("SWT_NO_LIBRARY_MACRO_PLUGINS"),

‎Sources/Testing/ABI/Encoded/ABI.EncodedExpectation.swift‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2025 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
extension ABI {
12+
/// A type implementing the JSON encoding of ``Expectation`` for the ABI entry
13+
/// point and event stream output.
14+
///
15+
/// This type is not part of the public interface of the testing library. It
16+
/// assists in converting values to JSON; clients that consume this JSON are
17+
/// expected to write their own decoders.
18+
///
19+
/// - Warning: Expectations are not yet part of the JSON schema.
20+
struct EncodedExpectation<V>: Sendable where V: ABI.Version {
21+
/// The expression evaluated by this expectation.
22+
///
23+
/// - Warning: Expressions are not yet part of the JSON schema.
24+
var _expression: EncodedExpression<V>
25+
26+
init(encoding expectation: borrowing Expectation, in eventContext: borrowing Event.Context) {
27+
_expression = EncodedExpression<V>(encoding: expectation.evaluatedExpression, in: eventContext)
28+
}
29+
}
30+
}
31+
32+
// MARK: - Codable
33+
34+
extension ABI.EncodedExpectation: Codable {}

‎Sources/Testing/ABI/Encoded/ABI.EncodedExpression.swift‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
//
2+
// This source file is part of the Swift.org open source project
3+
//
4+
// Copyright (c) 2025 Apple Inc. and the Swift project authors
5+
// Licensed under Apache License v2.0 with Runtime Library Exception
6+
//
7+
// See https://swift.org/LICENSE.txt for license information
8+
// See https://swift.org/CONTRIBUTORS.txt for Swift project authors
9+
//
10+
11+
extension ABI {
12+
/// A type implementing the JSON encoding of ``Expression`` for the ABI entry
13+
/// point and event stream output.
14+
///
15+
/// This type is not part of the public interface of the testing library. It
16+
/// assists in converting values to JSON; clients that consume this JSON are
17+
/// expected to write their own decoders.
18+
///
19+
/// - Warning: Expressions are not yet part of the JSON schema.
20+
struct EncodedExpression<V>: Sendable where V: ABI.Version {
21+
/// The source code of the original captured expression.
22+
var sourceCode: String
23+
24+
/// A string representation of the runtime value of this expression.
25+
///
26+
/// If the runtime value of this expression has not been evaluated, the
27+
/// value of this property is `nil`.
28+
var runtimeValue: String?
29+
30+
/// The fully-qualified name of the type of value represented by
31+
/// `runtimeValue`, or `nil` if that value has not been captured.
32+
var runtimeTypeName: String?
33+
34+
/// Any child expressions within this expression.
35+
var children: [EncodedExpression]?
36+
37+
init(encoding expression: borrowing __Expression, in eventContext: borrowing Event.Context) {
38+
sourceCode = expression.sourceCode
39+
runtimeValue = expression.runtimeValue.map(String.init(describingForTest:))
40+
runtimeTypeName = expression.runtimeValue.map(\.typeInfo.fullyQualifiedName)
41+
if !expression.subexpressions.isEmpty {
42+
children = expression.subexpressions.map { [eventContext = copy eventContext] subexpression in
43+
Self(encoding: subexpression, in: eventContext)
44+
}
45+
}
46+
}
47+
}
48+
}
49+
50+
// MARK: - Codable
51+
52+
extension ABI.EncodedExpression: Codable {}

‎Sources/Testing/ABI/Encoded/ABI.EncodedIssue.swift‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,11 @@ extension ABI {
5858
/// - Warning: Errors are not yet part of the JSON schema.
5959
var _error: EncodedError<V>?
6060

61+
/// The expectation associated with this issue, if applicable.
62+
///
63+
/// - Warning: Expectations are not yet part of the JSON schema.
64+
var _expectation: EncodedExpectation<V>?
65+
6166
init(encoding issue: borrowing Issue, in eventContext: borrowing Event.Context) {
6267
// >= v0
6368
isKnown = issue.isKnown
@@ -80,6 +85,9 @@ extension ABI {
8085
if let error = issue.error {
8186
_error = EncodedError(encoding: error, in: eventContext)
8287
}
88+
if case let .expectationFailed(expectation) = issue.kind {
89+
_expectation = EncodedExpectation(encoding: expectation, in: eventContext)
90+
}
8391
}
8492
}
8593
}

‎Sources/Testing/ABI/Encoded/ABI.EncodedMessage.swift‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,11 +64,19 @@ extension ABI {
6464
/// The symbol associated with this message.
6565
var symbol: Symbol
6666

67+
/// How much to indent this message when presenting it.
68+
///
69+
/// - Warning: This property is not yet part of the JSON schema.
70+
var _indentation: Int?
71+
6772
/// The human-readable, unformatted text associated with this message.
6873
var text: String
6974

7075
init(encoding message: borrowing Event.HumanReadableOutputRecorder.Message) {
7176
symbol = Symbol(encoding: message.symbol ?? .default)
77+
if message.indentation > 0 {
78+
_indentation = message.indentation
79+
}
7280
text = message.conciseStringValue ?? message.stringValue
7381
}
7482
}

‎Sources/Testing/CMakeLists.txt‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ add_library(Testing
1818
ABI/Encoded/ABI.EncodedBacktrace.swift
1919
ABI/Encoded/ABI.EncodedError.swift
2020
ABI/Encoded/ABI.EncodedEvent.swift
21+
ABI/Encoded/ABI.EncodedExpectation.swift
22+
ABI/Encoded/ABI.EncodedExpression.swift
2123
ABI/Encoded/ABI.EncodedInstant.swift
2224
ABI/Encoded/ABI.EncodedIssue.swift
2325
ABI/Encoded/ABI.EncodedMessage.swift
@@ -49,6 +51,7 @@ add_library(Testing
4951
Expectations/Expectation.swift
5052
Expectations/Expectation+Macro.swift
5153
Expectations/ExpectationChecking+Macro.swift
54+
Expectations/ExpectationContext.swift
5255
Issues/Confirmation.swift
5356
Issues/ErrorSnapshot.swift
5457
Issues/Issue.swift
@@ -71,7 +74,7 @@ add_library(Testing
7174
SourceAttribution/Backtrace+Symbolication.swift
7275
SourceAttribution/CustomTestStringConvertible.swift
7376
SourceAttribution/Expression.swift
74-
SourceAttribution/Expression+Macro.swift
77+
SourceAttribution/ExpressionID.swift
7578
SourceAttribution/SourceBounds.swift
7679
SourceAttribution/SourceContext.swift
7780
SourceAttribution/SourceLocation.swift

‎Sources/Testing/Events/Recorder/Event.ConsoleOutputRecorder.swift‎

Lines changed: 25 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,22 @@ private let _ansiEscapeCodePrefix = "\u{001B}["
129129
private let _resetANSIEscapeCode = "\(_ansiEscapeCodePrefix)0m"
130130

131131
extension Event.Symbol {
132+
/// Get the string value to use for a message with no associated symbol.
133+
///
134+
/// - Parameters:
135+
/// - options: Options to use when writing the symbol.
136+
///
137+
/// - Returns: A string representation of "no symbol" appropriate for writing
138+
/// to a stream.
139+
fileprivate static func placeholderStringValue(options: Event.ConsoleOutputRecorder.Options) -> String {
140+
#if os(macOS) || (os(iOS) && targetEnvironment(macCatalyst))
141+
if options.useSFSymbols {
142+
return " "
143+
}
144+
#endif
145+
return " "
146+
}
147+
132148
/// Get the string value for this symbol with the given write options.
133149
///
134150
/// - Parameters:
@@ -171,7 +187,7 @@ extension Event.Symbol {
171187
case .attachment:
172188
return "\(_ansiEscapeCodePrefix)94m\(symbolCharacter)\(_resetANSIEscapeCode)"
173189
case .details:
174-
return symbolCharacter
190+
return "\(symbolCharacter)"
175191
}
176192
}
177193
return "\(symbolCharacter)"
@@ -305,18 +321,12 @@ extension Event.ConsoleOutputRecorder {
305321
/// - Returns: Whether any output was produced and written to this instance's
306322
/// destination.
307323
@discardableResult public func record(_ event: borrowing Event, in context: borrowing Event.Context) -> Bool {
308-
let messages = _humanReadableOutputRecorder.record(event, in: context)
309-
310-
// Padding to use in place of a symbol for messages that don't have one.
311-
var padding = " "
312-
#if os(macOS) || (os(iOS) && targetEnvironment(macCatalyst))
313-
if options.useSFSymbols {
314-
padding = " "
315-
}
316-
#endif
324+
let symbolPlaceholder = Event.Symbol.placeholderStringValue(options: options)
317325

326+
let messages = _humanReadableOutputRecorder.record(event, in: context)
318327
let lines = messages.lazy.map { [test = context.test] message in
319-
let symbol = message.symbol?.stringValue(options: options) ?? padding
328+
let symbol = message.symbol?.stringValue(options: options) ?? symbolPlaceholder
329+
let indentation = String(repeating: " ", count: message.indentation)
320330

321331
if case .details = message.symbol {
322332
// Special-case the detail symbol to apply grey to the entire line of
@@ -325,17 +335,17 @@ extension Event.ConsoleOutputRecorder {
325335
// to the indentation provided by the symbol.
326336
var lines = message.stringValue.split(omittingEmptySubsequences: false, whereSeparator: \.isNewline)
327337
lines = CollectionOfOne(lines[0]) + lines.dropFirst().map { line in
328-
"\(padding) \(line)"
338+
"\(indentation)\(symbolPlaceholder) \(line)"
329339
}
330340
let stringValue = lines.joined(separator: "\n")
331341
if options.useANSIEscapeCodes, options.ansiColorBitDepth > 1 {
332-
return "\(_ansiEscapeCodePrefix)90m\(symbol) \(stringValue)\(_resetANSIEscapeCode)\n"
342+
return "\(_ansiEscapeCodePrefix)90m\(symbol) \(indentation)\(stringValue)\(_resetANSIEscapeCode)\n"
333343
} else {
334-
return "\(symbol) \(stringValue)\n"
344+
return "\(symbol) \(indentation)\(stringValue)\n"
335345
}
336346
} else {
337347
let colorDots = test.map { self.colorDots(for: $0.tags) } ?? ""
338-
return "\(symbol) \(colorDots)\(message.stringValue)\n"
348+
return "\(symbol) \(indentation)\(colorDots)\(message.stringValue)\n"
339349
}
340350
}
341351

‎Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift‎

Lines changed: 18 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,15 @@ extension Event {
2929
/// The symbol associated with this message, if any.
3030
var symbol: Symbol?
3131

32+
/// How much to indent this message when presenting it.
33+
///
34+
/// The way in which this additional indentation is rendered is
35+
/// implementation-defined. Typically, the greater the value of this
36+
/// property, the more whitespace characters are inserted.
37+
///
38+
/// Rendering of indentation is optional.
39+
var indentation = 0
40+
3241
/// The human-readable message.
3342
var stringValue: String
3443

@@ -503,20 +512,18 @@ extension Event.HumanReadableOutputRecorder {
503512
additionalMessages.append(_formattedComment(knownIssueComment))
504513
}
505514

506-
if verbosity > 0, case let .expectationFailed(expectation) = issue.kind {
515+
if verbosity >= 0, case let .expectationFailed(expectation) = issue.kind {
507516
let expression = expectation.evaluatedExpression
508-
func addMessage(about expression: __Expression) {
509-
let description = expression.expandedDebugDescription()
510-
additionalMessages.append(Message(symbol: .details, stringValue: description))
511-
}
512-
let subexpressions = expression.subexpressions
513-
if subexpressions.isEmpty {
514-
addMessage(about: expression)
515-
} else {
516-
for subexpression in subexpressions {
517-
addMessage(about: subexpression)
517+
func addMessage(about expression: __Expression, depth: Int) {
518+
let description = expression.expandedDescription(verbose: verbosity > 0)
519+
if description != expression.sourceCode {
520+
additionalMessages.append(Message(symbol: .details, indentation: depth, stringValue: description))
521+
}
522+
for subexpression in expression.subexpressions {
523+
addMessage(about: subexpression, depth: depth + 1)
518524
}
519525
}
526+
addMessage(about: expression, depth: 0)
520527
}
521528

522529
let atSourceLocation = issue.sourceLocation.map { " at \($0)" } ?? ""

‎Sources/Testing/ExitTests/ExitTest.swift‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -455,17 +455,16 @@ extension ExitTest {
455455
/// This function contains the common implementation for all
456456
/// `await #expect(processExitsWith:) { }` invocations regardless of calling
457457
/// convention.
458-
func callExitTest(
458+
nonisolated(nonsending) func callExitTest(
459459
identifiedBy exitTestID: (UInt64, UInt64, UInt64, UInt64),
460460
encodingCapturedValues capturedValues: [ExitTest.CapturedValue],
461461
processExitsWith expectedExitCondition: ExitTest.Condition,
462462
observing observedValues: [any PartialKeyPath<ExitTest.Result> & Sendable],
463-
expression: __Expression,
463+
sourceCode: @escaping @autoclosure @Sendable () -> KeyValuePairs<__ExpressionID, String>,
464464
comments: @autoclosure () -> [Comment],
465465
isRequired: Bool,
466-
isolation: isolated (any Actor)? = #isolation,
467466
sourceLocation: SourceLocation
468-
) async -> Result<ExitTest.Result?, any Error> {
467+
) async -> Result<ExitTest.Result?, ExpectationFailedError> {
469468
guard let configuration = Configuration.current ?? Configuration.all.first else {
470469
preconditionFailure("A test must be running on the current task to use #expect(processExitsWith:).")
471470
}
@@ -518,11 +517,14 @@ func callExitTest(
518517
}
519518

520519
// Plumb the exit test's result through the general expectation machinery.
521-
let expression = __Expression(String(describingForTest: expectedExitCondition))
522-
return __checkValue(
520+
let expectationContext = __ExpectationContext<Bool>(
521+
sourceCode: [.root: String(describingForTest: expectedExitCondition)],
522+
runtimeValues: [.root: { Expression.Value(reflecting: result.exitStatus) }]
523+
)
524+
return check(
523525
expectedExitCondition.isApproximatelyEqual(to: result.exitStatus),
524-
expression: expression,
525-
expressionWithCapturedRuntimeValues: expression.capturingRuntimeValues(result.exitStatus),
526+
expectationContext: expectationContext,
527+
mismatchedErrorDescription: nil,
526528
comments: comments(),
527529
isRequired: isRequired,
528530
sourceLocation: sourceLocation

‎Sources/Testing/Expectations/Expectation+Macro.swift‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@
6666
/// running in the current task and an instance of ``ExpectationFailedError`` is
6767
/// thrown.
6868
@freestanding(expression) public macro require<T>(
69-
_ optionalValue: T?,
69+
_ optionalValue: consuming T?,
7070
_ comment: @autoclosure () -> Comment? = nil,
7171
sourceLocation: SourceLocation = #_sourceLocation
72-
) -> T = #externalMacro(module: "TestingMacros", type: "RequireMacro")
72+
) -> T = #externalMacro(module: "TestingMacros", type: "UnwrapMacro") where T: ~Copyable
7373

7474
/// Unwrap an optional boolean value or, if it is `nil`, fail and throw an
7575
/// error.
@@ -89,7 +89,7 @@
8989
/// running in the current task and an instance of ``ExpectationFailedError`` is
9090
/// thrown.
9191
///
92-
/// This overload of ``require(_:_:sourceLocation:)-6w9oo`` checks if
92+
/// This overload of ``require(_:_:sourceLocation:)-5l63q`` checks if
9393
/// `optionalValue` may be ambiguous (i.e. it is unclear if the developer
9494
/// intended to check for a boolean value or unwrap an optional boolean value)
9595
/// and provides additional compile-time diagnostics when it is.
@@ -118,16 +118,16 @@ public macro require(
118118
/// running in the current task and an instance of ``ExpectationFailedError`` is
119119
/// thrown.
120120
///
121-
/// This overload of ``require(_:_:sourceLocation:)-6w9oo`` is used when a
121+
/// This overload of ``require(_:_:sourceLocation:)-5l63q`` is used when a
122122
/// non-optional, non-`Bool` value is passed to `#require()`. It emits a warning
123123
/// diagnostic indicating that the expectation is redundant.
124124
@freestanding(expression)
125125
@_documentation(visibility: private)
126126
public macro require<T>(
127-
_ optionalValue: T,
127+
_ optionalValue: consuming T,
128128
_ comment: @autoclosure () -> Comment? = nil,
129129
sourceLocation: SourceLocation = #_sourceLocation
130-
) -> T = #externalMacro(module: "TestingMacros", type: "NonOptionalRequireMacro")
130+
) -> T = #externalMacro(module: "TestingMacros", type: "NonOptionalRequireMacro") where T: ~Copyable
131131

132132
// MARK: - Matching errors by type
133133

0 commit comments

Comments
 (0)