From 7893f3f527428dd53aec1e66ef87dc1fd0e0e083 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Thu, 23 Jan 2025 18:23:17 -0500 Subject: [PATCH 1/2] [SWT-NNNN] Attachments Test authors frequently need to include out-of-band data with tests that can be used to diagnose issues when a test fails. This proposal introduces a new API called "attachments" (analogous to the same-named feature in XCTest) as well as the infrastructure necessary to create new attachments and handle them in tools like VS Code. Read the full proposal [here](). --- Documentation/ABI/JSON.md | 7 +- Documentation/Proposals/NNNN-attachments.md | 436 ++++++++++++++++++ .../_AttachableImageContainer.swift | 11 +- .../Attachable+Encodable+NSSecureCoding.swift | 10 +- .../Attachments/Attachable+Encodable.swift | 23 +- .../Attachable+NSSecureCoding.swift | 13 +- .../Attachments/Attachment+URL.swift | 7 +- .../Attachments/Data+Attachable.swift | 11 +- .../Attachments/EncodingFormat.swift | 2 +- .../Attachments/_AttachableURLContainer.swift | 5 +- .../ABI/Encoded/ABI.EncodedEvent.swift | 8 +- Sources/Testing/Attachments/Attachable.swift | 36 +- .../Attachments/AttachableContainer.swift | 18 +- Sources/Testing/Attachments/Attachment.swift | 57 ++- Sources/Testing/Testing.docc/Attachments.md | 32 ++ Sources/Testing/Testing.docc/Documentation.md | 4 + .../Testing.docc/MigratingFromXCTest.md | 60 +++ Tests/TestingTests/AttachmentTests.swift | 28 +- 18 files changed, 685 insertions(+), 83 deletions(-) create mode 100644 Documentation/Proposals/NNNN-attachments.md create mode 100644 Sources/Testing/Testing.docc/Attachments.md diff --git a/Documentation/ABI/JSON.md b/Documentation/ABI/JSON.md index 2fac0c3fb..9dd3f59d3 100644 --- a/Documentation/ABI/JSON.md +++ b/Documentation/ABI/JSON.md @@ -188,19 +188,24 @@ sufficient information to display the event in a human-readable format. "kind": , "instant": , ; when the event occurred ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") + ["attachment": ,] ; the attachment (if kind is "valueAttached") "messages": , ["testID": ,] } ::= "runStarted" | "testStarted" | "testCaseStarted" | "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | - "runEnded" ; additional event kinds may be added in the future + "runEnded" | "valueAttached"; additional event kinds may be added in the future ::= { "isKnown": , ; is this a known issue or not? ["sourceLocation": ,] ; where the issue occurred, if known } + ::= { + "path": , ; the absolute path to the attachment on disk +} + ::= { "symbol": , "text": , ; the human-readable text of this message diff --git a/Documentation/Proposals/NNNN-attachments.md b/Documentation/Proposals/NNNN-attachments.md new file mode 100644 index 000000000..ce3d5efde --- /dev/null +++ b/Documentation/Proposals/NNNN-attachments.md @@ -0,0 +1,436 @@ +# Attachments + +* Proposal: [SWT-NNNN](NNNN-attachments.md) +* Authors: [Jonathan Grynspan](https://github.com/grynspan) +* Status: **Awaiting review** +* Bug: [swiftlang/swift-testing#714](https://github.com/swiftlang/swift-testing/issues/714) +* Implementation: [swiftlang/swift-testing#796](https://github.com/swiftlang/swift-testing/pull/796) +* Review: ([pitch](https://forums.swift.org/...)) + +## Introduction + +Test authors frequently need to include out-of-band data with tests that can be +used to diagnose issues when a test fails. This proposal introduces a new API +called "attachments" (analogous to the same-named feature in XCTest) as well as +the infrastructure necessary to create new attachments and handle them in tools +like VS Code. + +## Motivation + +When a test fails, especially in a remote environment like CI, it can often be +difficult to determine what exactly has gone wrong. Data that was produced +during the test can be useful, but there is currently no mechanism in Swift +Testing to output arbitrary data other than via `stdout`/`stderr` or via an +artificially-generated issue. A dedicated interface for attaching arbitrary +information to a test would allow test authors to gather relevant information +from a test in a structured way. + +## Proposed solution + +We propose introducing a new type to Swift Testing, `Attachment`, that represents +some arbitrary "attachment" to associate with a test. Along with `Attachment`, +we will introduce a new protocol, `Attachable`, to which types can conform to +indicate they can be attached to a test. + +Default conformances to `Attachable` will be provided for standard library types +that can reasonably be attached. We will also introduce a **cross-import overlay** +with Foundation—that is, a tertiary module that is automatically imported when +a test target imports both Foundation _and_ Swift Testing—that includes +additional conformances for Foundation types such as `Data` and `URL` and +provides support for attaching values that also conform to `Encodable` or +`NSSecureCoding`. + +## Detailed design + +The `Attachment` type is defined as follows: + +```swift +/// A type describing values that can be attached to the output of a test run +/// and inspected later by the user. +/// +/// Attachments are included in test reports in Xcode or written to disk when +/// tests are run at the command line. To create an attachment, you need a value +/// of some type that conforms to ``Attachable``. Initialize an instance of +/// ``Attachment`` with that value and, optionally, a preferred filename to use +/// when writing to disk. +public struct Attachment: ~Copyable where AttachableValue: Attachable & ~Copyable { + /// A filename to use when writing this attachment to a test report or to a + /// file on disk. + /// + /// The value of this property is used as a hint to the testing library. The + /// testing library may substitute a different filename as needed. If the + /// value of this property has not been explicitly set, the testing library + /// will attempt to generate its own value. + public var preferredName: String { get } + + /// The value of this attachment. + public var attachableValue: AttachableValue { get } + + /// Initialize an instance of this type that encloses the given attachable + /// value. + /// + /// - Parameters: + /// - attachableValue: The value that will be attached to the output of the + /// test run. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the testing library attempts to + /// derive a reasonable filename for the attached value. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + public init( + _ attachableValue: consuming AttachableValue, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) + + /// Attach this instance to the current test. + /// + /// - Parameters: + /// - sourceLocation: The source location of the call to this function. + /// + /// When attaching a value of a type that does not conform to both + /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and + /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), + /// the testing library encodes it as data immediately. If the value cannot be + /// encoded and an error is thrown, that error is recorded as an issue in the + /// current test and the attachment is not written to the test report or to + /// disk. + /// + /// An attachment can only be attached once. + public consuming func attach(sourceLocation: SourceLocation = #_sourceLocation) + + /// Call a function and pass a buffer representing the value of this + /// instance's ``attachableValue-2tnj5`` property to it. + /// + /// - Parameters: + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. This function calls the + /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's + /// ``attachableValue-2tnj5`` property. + @inlinable public borrowing func withUnsafeBytes( + _ body: (UnsafeRawBufferPointer) throws -> R + ) throws -> R +} + +extension Attachment: Copyable where AttachableValue: Copyable {} +extension Attachment: Sendable where AttachableValue: Sendable {} +``` + +With `Attachment` comes `Attachable`, a protocol to which "attachable values" +conform: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run. +/// +/// To attach an attachable value to a test report or test run output, use it to +/// initialize a new instance of ``Attachment``, then call +/// ``Attachment/attach(sourceLocation:)``. An attachment can only be attached +/// once. +/// +/// The testing library provides default conformances to this protocol for a +/// variety of standard library types. Most user-defined types do not need to +/// conform to this protocol. +/// +/// A type should conform to this protocol if it can be represented as a +/// sequence of bytes that would be diagnostically useful if a test fails. If a +/// type cannot conform directly to this protocol (such as a non-final class or +/// a type declared in a third-party module), you can create a container type +/// that conforms to ``AttachableContainer`` to act as a proxy. +public protocol Attachable: ~Copyable { + /// An estimate of the number of bytes of memory needed to store this value as + /// an attachment. + /// + /// The testing library uses this property to determine if an attachment + /// should be held in memory or should be immediately persisted to storage. + /// Larger attachments are more likely to be persisted, but the algorithm the + /// testing library uses is an implementation detail and is subject to change. + /// + /// The value of this property is approximately equal to the number of bytes + /// that will actually be needed, or `nil` if the value cannot be computed + /// efficiently. The default implementation of this property returns `nil`. + /// + /// - Complexity: O(1) unless `Self` conforms to `Collection`, in which case + /// up to O(_n_) where _n_ is the length of the collection. + var estimatedAttachmentByteCount: Int? { get } + + /// Call a function and pass a buffer representing this instance to it. + /// + /// - Parameters: + /// - attachment: The attachment that is requesting a buffer (that is, the + /// attachment containing this instance.) + /// - body: A function to call. A temporary buffer containing a data + /// representation of this instance is passed to it. + /// + /// - Returns: Whatever is returned by `body`. + /// + /// - Throws: Whatever is thrown by `body`, or any error that prevented the + /// creation of the buffer. + /// + /// The testing library uses this function when writing an attachment to a + /// test report or to a file on disk. The format of the buffer is + /// implementation-defined, but should be "idiomatic" for this type: for + /// example, if this type represents an image, it would be appropriate for + /// the buffer to contain an image in PNG format, JPEG format, etc., but it + /// would not be idiomatic for the buffer to contain a textual description of + /// the image. + borrowing func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R + + /// Generate a preferred name for the given attachment. + /// + /// - Parameters: + /// - attachment: The attachment that needs to be named. + /// - suggestedName: A suggested name to use as the basis of the preferred + /// name. This string was provided by the developer when they initialized + /// `attachment`. + /// + /// - Returns: The preferred name for `attachment`. + /// + /// The testing library uses this function to determine the best name to use + /// when adding `attachment` to a test report or persisting it to storage. The + /// default implementation of this function returns `suggestedName` without + /// any changes. + borrowing func preferredName(for attachment: borrowing Attachment, basedOn suggestedName: String) -> String +} +``` + +Default conformances to `Attachable` are provided for: + +- `Array`, `ContiguousArray`, and `ArraySlice` +- `String` and `Substring` +- `Data` (if Foundation is also imported) + +Default _implementations_ are provided for types when they conform to +`Attachable` and either `Encodable` or `NSSecureCoding` (or both.) To use these +conformances, Foundation must be imported because `JSONEncoder` and +`PropertyListEncoder` are members of Foundation, not the Swift standard library. + +Some types cannot conform directly to `Attachable` because they require +additional information to encode correctly, or because they are not directly +`Sendable` or `Copyable`. A second protocol, `AttachableContainer`, is provided +that refines `Attachable`: + +```swift +/// A protocol describing a type that can be attached to a test report or +/// written to disk when a test is run and which contains another value that it +/// stands in for. +/// +/// To attach an attachable value to a test report or test run output, use it to +/// initialize a new instance of ``Attachment``, then call +/// ``Attachment/attach(sourceLocation:)``. An attachment can only be attached +/// once. +/// +/// A type can conform to this protocol if it represents another type that +/// cannot directly conform to ``Attachable``, such as a non-final class or a +/// type declared in a third-party module. +public protocol AttachableContainer: Attachable, ~Copyable { + /// The type of the attachable value represented by this type. + associatedtype AttachableValue + + /// The attachable value represented by this instance. + var attachableValue: AttachableValue { get } +} + +extension Attachment where AttachableValue: AttachableContainer & ~Copyable { + /// The value of this attachment. + /// + /// When the attachable value's type conforms to ``AttachableContainer``, the + /// value of this property equals the container's underlying attachable value. + /// To access the attachable value as an instance of `T` (where `T` conforms + /// to ``AttachableContainer``), specify the type explicitly: + /// + /// ```swift + /// let attachableValue = attachment.attachableValue as T + /// ``` + public var attachableValue: AttachableValue.AttachableValue { get } +} +``` + +The cross-import overlay with Foundation also provides the following convenience +interface for attaching the contents of a file or directory on disk: + +```swift +extension Attachment where AttachableValue == _AttachableURLContainer { + /// Initialize an instance of this type with the contents of the given URL. + /// + /// - Parameters: + /// - url: The URL containing the attachment's data. + /// - preferredName: The preferred name of the attachment when writing it to + /// a test report or to disk. If `nil`, the name of the attachment is + /// derived from the last path component of `url`. + /// - sourceLocation: The source location of the call to this initializer. + /// This value is used when recording issues associated with the + /// attachment. + /// + /// - Throws: Any error that occurs attempting to read from `url`. + public init( + contentsOf url: URL, + named preferredName: String? = nil, + sourceLocation: SourceLocation = #_sourceLocation + ) async throws +} +``` + +`_AttachableURLContainer` is a type that conforms to `AttachableContainer` and +encloses the URL and corresponding mapped data. As an implementation detail, it +is omitted from this proposal for brevity. + +## Source compatibility + +This proposal is additive and has no impact on existing code. + +## Integration with supporting tools + +We will add a new command-line argument to the `swift test` command in Swift +Package Manager: + +```sh +--attachments-path Path where attachments should be saved. +``` + +If specified, an attachment will be written to that path when its `attach()` +method is called. If not specified, attachments are not saved to disk. Tools +that indirectly use Swift Testing through `swift test` can specify a path (e.g. +to a directory created inside the system's temporary directory), then move or +delete the created files as needed. + +The JSON event stream ABI will be amended correspondingly: + +```diff +--- a/Documentation/ABI/JSON.md ++++ b/Documentation/ABI/JSON.md + ::= { + "kind": , + "instant": , ; when the event occurred + ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") ++ ["attachment": ,] ; the attachment (if kind is "valueAttached") + "messages": , + ["testID": ,] + } + + ::= "runStarted" | "testStarted" | "testCaseStarted" | + "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | +- "runEnded" ; additional event kinds may be added in the future ++ "runEnded" | "valueAttached"; additional event kinds may be added in the future + ++ ::= { ++ "path": , ; the absolute path to the attachment on disk ++} +``` + +As these changes are additive only, the JSON schema version does not need to be +incremented to support them. We are separately planning to increment the JSON +schema version to support other features; these changes will apply to the newer +version too. + +## Future directions + +- Attachment lifetime management: XCTest's attachments allow for specifying a + "lifetime", with two lifetimes currently available: + + ```objc + typedef NS_ENUM(NSInteger, XCTAttachmentLifetime) { + XCTAttachmentLifetimeKeepAlways = 0, + XCTAttachmentLifetimeDeleteOnSuccess = 1 + }; + ``` + + If a test passes, it is probably not necessary to keep its attachments saved + to disk. The exact "shape" this feature should take in Swift Testing is not + yet clear. + +- Image attachments: it is often useful to be able to attach images to tests, + however there is no cross-platform solution for this functionality. An + experimental implementation that allows attaching an instance of `CGImage` (on + Apple platforms) is available in Swift Testing's repository and shows what it + might look like for us to provide this functionality. + +- Additional conformances for types in other modules: in order to keep Swift + Testing's dependency graph as small as possible, we cannot link it to + arbitrary packages such as (for example) swift-collections even if it would be + useful to do so. That means we can't directly provide conformances to + `Attachable` for types in those modules. Adding additional cross-import + overlays would allow us to provide those conformances when both Swift Testing + and those packages are imported at the same time. + + This functionality may require changes in Swift Package Manager that are + beyond the scope of this proposal. + +- Adopting `RawSpan` instead of `UnsafeRawBufferPointer`: `RawSpan` represents a + safer alternative to `UnsafeRawBufferPointer`, but it is not yet available + everywhere we'd need it in the standard library, and our minimum deployment + targets on Apple's platforms do not allow us to require the use of `RawSpan` + (as no shipping version of Apple's platforms includes it.) + +- Adding an associated `Metadata` type to `Attachable` allowing for inclusion of + arbitrary out-of-band data to attachments: we see several uses for such a + feature: + + - Fine-grained control of the serialization format used for `Encodable` types; + - Metrics (scaling factor, rotation, etc.) for images; and + - Compression algorithms to use for attached files and directories. + + The exact shape of this interface needs further consideration, but it could be + added in the future without disrupting the interface we are proposing here. + [swiftlang/swift-testing#824](https://github.com/swiftlang/swift-testing/pull/824) + includes an experimental implementation of this feature. + +## Alternatives considered + +- Doing nothing: there's sufficient demand for this feature that we know we want + to address it. + +- Reusing the existing `XCTAttachment` API from XCTest: while this would + _probably_ have saved me a lot of typing, `XCTAttachment` is an Objective-C + class and is only available on Apple's platforms. The open-source + swift-corelibs-xctest package does not include it or an equivalent interface. + As well, this would create a dependency on XCTest in Swift Testing that does + not currently exist. + +- Implementing `Attachment` as a non-generic type and eagerly serializing + non-sendable or move-only attachable values: an earlier implementation did + exactly this, but it forced us to include an existential box in `Attachment` + to store the attachable value, and that would preclude ever supporting + attachments in Embedded Swift. + +- Having `Attachment` take a byte buffer rather than an attachable value, or + having it take a closure that returns a byte buffer: this would just raise the + problem of attaching arbitrary values up to the test author's layer, and that + would no doubt produce a lot of duplicate implementations of "turn this value + into a byte buffer" while also worsening the interface's ergonomics. + +- Adding a `var contentType: UTType { get set }` property to `Attachment` or to + `Attachable`: `XCTAttachment` lets you specify a Uniform Type Identifier that + tells Xcode the type of data. Uniform Type Identifiers are proprietary and not + available on Linux or Windows, and adding that property would force us to also + add a public dependency on the `UniformTypeIdentifiers` framework and, + indirectly, on Foundation, which would prevent Foundation from authoring tests + using Swift Testing in the future due to the resulting circular dependency. + + We considered using a MIME type instead, but there is no portable mechanism + for turning a MIME type into a path extension, which is ultimately what we + need when writing an attachment to persistent storage. + + Instead, `Attachable` includes the function `preferredName(for:basedOn:)` that + allows an implementation (such as that of `Encodable & Attachable`) to add a + path extension to the filename specified by the test author if needed. + +## Acknowledgments + +Thanks to Stuart Montgomery and Brian Croom for goading me into finally writing +this proposal! + +Thanks to Wil Addario-Turner for his feedback, in particular around `UTType` and +MIME type support. + +Thanks to Honza Dvorsky for his earlier work on attachments in XCTest and his +ideas on how to improve Swift Testing's implementation. diff --git a/Sources/Overlays/_Testing_CoreGraphics/Attachments/_AttachableImageContainer.swift b/Sources/Overlays/_Testing_CoreGraphics/Attachments/_AttachableImageContainer.swift index 9db225826..752b47d0c 100644 --- a/Sources/Overlays/_Testing_CoreGraphics/Attachments/_AttachableImageContainer.swift +++ b/Sources/Overlays/_Testing_CoreGraphics/Attachments/_AttachableImageContainer.swift @@ -32,11 +32,10 @@ import UniformTypeIdentifiers /// such a requirement, and all image types we care about are non-final /// classes. Thus, the compiler will steadfastly refuse to allow non-final /// classes to conform to the `Attachable` protocol. We could get around this -/// by changing the signature of `withUnsafeBufferPointer()` so that the -/// generic parameter to `Attachment` is not `Self`, but that would defeat -/// much of the purpose of making `Attachment` generic in the first place. -/// (And no, the language does not let us write `where T: Self` anywhere -/// useful.) +/// by changing the signature of `withUnsafeBytes()` so that the generic +/// parameter to `Attachment` is not `Self`, but that would defeat much of +/// the purpose of making `Attachment` generic in the first place. (And no, +/// the language does not let us write `where T: Self` anywhere useful.) /// A wrapper type for image types such as `CGImage` and `NSImage` that can be /// attached indirectly. @@ -132,7 +131,7 @@ extension _AttachableImageContainer: AttachableContainer { image } - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { let data = NSMutableData() // Convert the image to a CGImage. diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable+NSSecureCoding.swift b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable+NSSecureCoding.swift index 80c75b5e9..46a1e11e6 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable+NSSecureCoding.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable+NSSecureCoding.swift @@ -9,7 +9,7 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing public import Foundation // This implementation is necessary to let the compiler disambiguate when a type @@ -18,11 +18,13 @@ public import Foundation // (which explicitly document what happens when a type conforms to both // protocols.) -@_spi(Experimental) +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } extension Attachable where Self: Encodable & NSSecureCoding { @_documentation(visibility: private) - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { - try _Testing_Foundation.withUnsafeBufferPointer(encoding: self, for: attachment, body) + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + try _Testing_Foundation.withUnsafeBytes(encoding: self, for: attachment, body) } } #endif diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable.swift b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable.swift index cfae97ca7..683888801 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+Encodable.swift @@ -9,12 +9,12 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing private import Foundation -/// A common implementation of ``withUnsafeBufferPointer(for:_:)`` that is -/// used when a type conforms to `Encodable`, whether or not it also conforms -/// to `NSSecureCoding`. +/// A common implementation of ``withUnsafeBytes(for:_:)`` that is used when a +/// type conforms to `Encodable`, whether or not it also conforms to +/// `NSSecureCoding`. /// /// - Parameters: /// - attachableValue: The value to encode. @@ -27,7 +27,7 @@ private import Foundation /// /// - Throws: Whatever is thrown by `body`, or any error that prevented the /// creation of the buffer. -func withUnsafeBufferPointer(encoding attachableValue: borrowing E, for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R where E: Attachable & Encodable { +func withUnsafeBytes(encoding attachableValue: borrowing E, for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R where E: Attachable & Encodable { let format = try EncodingFormat(for: attachment) let data: Data @@ -53,7 +53,10 @@ func withUnsafeBufferPointer(encoding attachableValue: borrowing E, for at // Implement the protocol requirements generically for any encodable value by // encoding to JSON. This lets developers provide trivial conformance to the // protocol for types that already support Codable. -@_spi(Experimental) + +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } extension Attachable where Self: Encodable { /// Encode this value into a buffer using either [`PropertyListEncoder`](https://developer.apple.com/documentation/foundation/propertylistencoder) /// or [`JSONEncoder`](https://developer.apple.com/documentation/foundation/jsonencoder), @@ -86,8 +89,12 @@ extension Attachable where Self: Encodable { /// _and_ [`NSSecureCoding`](https://developer.apple.com/documentation/foundation/nssecurecoding), /// the default implementation of this function uses the value's conformance /// to `Encodable`. - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { - try _Testing_Foundation.withUnsafeBufferPointer(encoding: self, for: attachment, body) + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + try _Testing_Foundation.withUnsafeBytes(encoding: self, for: attachment, body) } } #endif diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+NSSecureCoding.swift b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+NSSecureCoding.swift index c6916ec39..4acbf4960 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+NSSecureCoding.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/Attachable+NSSecureCoding.swift @@ -9,13 +9,16 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing public import Foundation // As with Encodable, implement the protocol requirements for // NSSecureCoding-conformant classes by default. The implementation uses // NSKeyedArchiver for encoding. -@_spi(Experimental) + +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } extension Attachable where Self: NSSecureCoding { /// Encode this object using [`NSKeyedArchiver`](https://developer.apple.com/documentation/foundation/nskeyedarchiver) /// into a buffer, then call a function and pass that buffer to it. @@ -46,7 +49,11 @@ extension Attachable where Self: NSSecureCoding { /// _and_ [`NSSecureCoding`](https://developer.apple.com/documentation/foundation/nssecurecoding), /// the default implementation of this function uses the value's conformance /// to `Encodable`. - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { let format = try EncodingFormat(for: attachment) var data = try NSKeyedArchiver.archivedData(withRootObject: self, requiringSecureCoding: true) diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/Attachment+URL.swift b/Sources/Overlays/_Testing_Foundation/Attachments/Attachment+URL.swift index dbf7e2688..af27b6c1a 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/Attachment+URL.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/Attachment+URL.swift @@ -9,7 +9,7 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing public import Foundation #if !SWT_NO_PROCESS_SPAWNING && os(Windows) @@ -32,7 +32,6 @@ extension URL { } } -@_spi(Experimental) extension Attachment where AttachableValue == _AttachableURLContainer { #if SWT_TARGET_OS_APPLE /// An operation queue to use for asynchronously reading data from disk. @@ -51,6 +50,10 @@ extension Attachment where AttachableValue == _AttachableURLContainer { /// attachment. /// /// - Throws: Any error that occurs attempting to read from `url`. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public init( contentsOf url: URL, named preferredName: String? = nil, diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/Data+Attachable.swift b/Sources/Overlays/_Testing_Foundation/Attachments/Data+Attachable.swift index f931e5824..ce7b719a9 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/Data+Attachable.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/Data+Attachable.swift @@ -9,12 +9,17 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing public import Foundation -@_spi(Experimental) +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } extension Data: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { try withUnsafeBytes(body) } } diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/EncodingFormat.swift b/Sources/Overlays/_Testing_Foundation/Attachments/EncodingFormat.swift index bbbe934ab..49499a8c2 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/EncodingFormat.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/EncodingFormat.swift @@ -9,7 +9,7 @@ // #if canImport(Foundation) -@_spi(Experimental) import Testing +import Testing import Foundation /// An enumeration describing the encoding formats we support for `Encodable` diff --git a/Sources/Overlays/_Testing_Foundation/Attachments/_AttachableURLContainer.swift b/Sources/Overlays/_Testing_Foundation/Attachments/_AttachableURLContainer.swift index 38f21d4d3..24f187f42 100644 --- a/Sources/Overlays/_Testing_Foundation/Attachments/_AttachableURLContainer.swift +++ b/Sources/Overlays/_Testing_Foundation/Attachments/_AttachableURLContainer.swift @@ -9,7 +9,7 @@ // #if canImport(Foundation) -@_spi(Experimental) public import Testing +public import Testing public import Foundation /// A wrapper type representing file system objects and URLs that can be @@ -17,7 +17,6 @@ public import Foundation /// /// You do not need to use this type directly. Instead, initialize an instance /// of ``Attachment`` using a file URL. -@_spi(Experimental) public struct _AttachableURLContainer: Sendable { /// The underlying URL. var url: URL @@ -36,7 +35,7 @@ extension _AttachableURLContainer: AttachableContainer { url } - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { try data.withUnsafeBytes(body) } diff --git a/Sources/Testing/ABI/Encoded/ABI.EncodedEvent.swift b/Sources/Testing/ABI/Encoded/ABI.EncodedEvent.swift index b8bafdde1..73e7db2ac 100644 --- a/Sources/Testing/ABI/Encoded/ABI.EncodedEvent.swift +++ b/Sources/Testing/ABI/Encoded/ABI.EncodedEvent.swift @@ -27,7 +27,7 @@ extension ABI { case testStarted case testCaseStarted case issueRecorded - case valueAttached = "_valueAttached" + case valueAttached case testCaseEnded case testEnded case testSkipped @@ -50,9 +50,7 @@ extension ABI { /// /// The value of this property is `nil` unless the value of the /// ``kind-swift.property`` property is ``Kind-swift.enum/valueAttached``. - /// - /// - Warning: Attachments are not yet part of the JSON schema. - var _attachment: EncodedAttachment? + var attachment: EncodedAttachment? /// Human-readable messages associated with this event that can be presented /// to the user. @@ -82,7 +80,7 @@ extension ABI { issue = EncodedIssue(encoding: recordedIssue, in: eventContext) case let .valueAttached(attachment): kind = .valueAttached - _attachment = EncodedAttachment(encoding: attachment, in: eventContext) + self.attachment = EncodedAttachment(encoding: attachment, in: eventContext) case .testCaseEnded: if eventContext.test?.isParameterized == false { return nil diff --git a/Sources/Testing/Attachments/Attachable.swift b/Sources/Testing/Attachments/Attachable.swift index 4a1d775a5..ceddb11ca 100644 --- a/Sources/Testing/Attachments/Attachable.swift +++ b/Sources/Testing/Attachments/Attachable.swift @@ -25,7 +25,10 @@ /// type cannot conform directly to this protocol (such as a non-final class or /// a type declared in a third-party module), you can create a container type /// that conforms to ``AttachableContainer`` to act as a proxy. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } public protocol Attachable: ~Copyable { /// An estimate of the number of bytes of memory needed to store this value as /// an attachment. @@ -41,6 +44,10 @@ public protocol Attachable: ~Copyable { /// /// - Complexity: O(1) unless `Self` conforms to `Collection`, in which case /// up to O(_n_) where _n_ is the length of the collection. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } var estimatedAttachmentByteCount: Int? { get } /// Call a function and pass a buffer representing this instance to it. @@ -63,7 +70,11 @@ public protocol Attachable: ~Copyable { /// the buffer to contain an image in PNG format, JPEG format, etc., but it /// would not be idiomatic for the buffer to contain a textual description of /// the image. - borrowing func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } + borrowing func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R /// Generate a preferred name for the given attachment. /// @@ -79,6 +90,10 @@ public protocol Attachable: ~Copyable { /// when adding `attachment` to a test report or persisting it to storage. The /// default implementation of this function returns `suggestedName` without /// any changes. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } borrowing func preferredName(for attachment: borrowing Attachment, basedOn suggestedName: String) -> String } @@ -99,7 +114,7 @@ extension Attachable where Self: Collection, Element == UInt8 { count } - // We do not provide an implementation of withUnsafeBufferPointer(for:_:) here + // We do not provide an implementation of withUnsafeBytes(for:_:) here // because there is no way in the standard library to statically detect if a // collection can provide contiguous storage (_HasContiguousBytes is not API.) // If withContiguousStorageIfAvailable(_:) fails, we don't want to make a @@ -118,30 +133,26 @@ extension Attachable where Self: StringProtocol { // Implement the protocol requirements for byte arrays and buffers so that // developers can attach raw data when needed. -@_spi(Experimental) extension Array: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { try withUnsafeBytes(body) } } -@_spi(Experimental) extension ContiguousArray: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { try withUnsafeBytes(body) } } -@_spi(Experimental) extension ArraySlice: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { try withUnsafeBytes(body) } } -@_spi(Experimental) extension String: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { var selfCopy = self return try selfCopy.withUTF8 { utf8 in try body(UnsafeRawBufferPointer(utf8)) @@ -149,9 +160,8 @@ extension String: Attachable { } } -@_spi(Experimental) extension Substring: Attachable { - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { var selfCopy = self return try selfCopy.withUTF8 { utf8 in try body(UnsafeRawBufferPointer(utf8)) diff --git a/Sources/Testing/Attachments/AttachableContainer.swift b/Sources/Testing/Attachments/AttachableContainer.swift index aced49c0f..134965e0e 100644 --- a/Sources/Testing/Attachments/AttachableContainer.swift +++ b/Sources/Testing/Attachments/AttachableContainer.swift @@ -20,16 +20,22 @@ /// A type can conform to this protocol if it represents another type that /// cannot directly conform to ``Attachable``, such as a non-final class or a /// type declared in a third-party module. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } public protocol AttachableContainer: Attachable, ~Copyable { -#if hasFeature(SuppressedAssociatedTypes) - /// The type of the attachable value represented by this type. - associatedtype AttachableValue: ~Copyable -#else /// The type of the attachable value represented by this type. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } associatedtype AttachableValue -#endif /// The attachable value represented by this instance. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } var attachableValue: AttachableValue { get } } diff --git a/Sources/Testing/Attachments/Attachment.swift b/Sources/Testing/Attachments/Attachment.swift index ef7ae5537..914133cbb 100644 --- a/Sources/Testing/Attachments/Attachment.swift +++ b/Sources/Testing/Attachments/Attachment.swift @@ -18,7 +18,10 @@ private import _TestingInternals /// of some type that conforms to ``Attachable``. Initialize an instance of /// ``Attachment`` with that value and, optionally, a preferred filename to use /// when writing to disk. -@_spi(Experimental) +/// +/// @Metadata { +/// @Available(Swift, introduced: 6.2) +/// } public struct Attachment: ~Copyable where AttachableValue: Attachable & ~Copyable { /// Storage for ``attachableValue-7dyjv``. fileprivate var _attachableValue: AttachableValue @@ -51,6 +54,10 @@ public struct Attachment: ~Copyable where AttachableValue: Atta /// testing library may substitute a different filename as needed. If the /// value of this property has not been explicitly set, the testing library /// will attempt to generate its own value. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var preferredName: String { let suggestedName = if let _preferredName, !_preferredName.isEmpty { _preferredName @@ -90,6 +97,10 @@ extension Attachment where AttachableValue: ~Copyable { /// - sourceLocation: The source location of the call to this initializer. /// This value is used when recording issues associated with the /// attachment. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public init(_ attachableValue: consuming AttachableValue, named preferredName: String? = nil, sourceLocation: SourceLocation = #_sourceLocation) { self._attachableValue = attachableValue self._preferredName = preferredName @@ -97,7 +108,7 @@ extension Attachment where AttachableValue: ~Copyable { } } -@_spi(Experimental) @_spi(ForToolsIntegrationOnly) +@_spi(ForToolsIntegrationOnly) extension Attachment where AttachableValue == AnyAttachable { /// Create a type-erased attachment from an instance of ``Attachment``. /// @@ -125,7 +136,7 @@ extension Attachment where AttachableValue == AnyAttachable { // Swift's type system requires that this type be at least as visible as // `Event.Kind.valueAttached(_:)`, otherwise it would be declared private. // } -@_spi(Experimental) @_spi(ForToolsIntegrationOnly) +@_spi(ForToolsIntegrationOnly) public struct AnyAttachable: AttachableContainer, Copyable, Sendable { #if !SWT_NO_LAZY_ATTACHMENTS public typealias AttachableValue = any Attachable & Sendable /* & Copyable rdar://137614425 */ @@ -143,7 +154,7 @@ public struct AnyAttachable: AttachableContainer, Copyable, Sendable { attachableValue.estimatedAttachmentByteCount } - public func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + public func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { func open(_ attachableValue: T, for attachment: borrowing Attachment) throws -> R where T: Attachable & Sendable & Copyable { let temporaryAttachment = Attachment( _attachableValue: attachableValue, @@ -151,7 +162,7 @@ public struct AnyAttachable: AttachableContainer, Copyable, Sendable { _preferredName: attachment._preferredName, sourceLocation: attachment.sourceLocation ) - return try temporaryAttachment.withUnsafeBufferPointer(body) + return try temporaryAttachment.withUnsafeBytes(body) } return try open(attachableValue, for: attachment) } @@ -173,6 +184,7 @@ public struct AnyAttachable: AttachableContainer, Copyable, Sendable { // MARK: - Describing an attachment extension Attachment where AttachableValue: ~Copyable { + @_documentation(visibility: private) public var description: String { let typeInfo = TypeInfo(describing: AttachableValue.self) return #""\#(preferredName)": instance of '\#(typeInfo.unqualifiedName)'"# @@ -180,6 +192,9 @@ extension Attachment where AttachableValue: ~Copyable { } extension Attachment: CustomStringConvertible { + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var description: String { #""\#(preferredName)": \#(String(describingForTest: attachableValue))"# } @@ -187,9 +202,12 @@ extension Attachment: CustomStringConvertible { // MARK: - Getting an attachable value from an attachment -@_spi(Experimental) extension Attachment where AttachableValue: ~Copyable { /// The value of this attachment. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } @_disfavoredOverload public var attachableValue: AttachableValue { _read { yield _attachableValue @@ -197,7 +215,6 @@ extension Attachment where AttachableValue: ~Copyable { } } -@_spi(Experimental) extension Attachment where AttachableValue: AttachableContainer & ~Copyable { /// The value of this attachment. /// @@ -209,6 +226,10 @@ extension Attachment where AttachableValue: AttachableContainer & ~Copyable { /// ```swift /// let attachableValue = attachment.attachableValue as T /// ``` + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public var attachableValue: AttachableValue.AttachableValue { _read { yield attachableValue.attachableValue @@ -250,9 +271,13 @@ extension Attachment where AttachableValue: ~Copyable { /// disk. /// /// An attachment can only be attached once. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } public consuming func attach(sourceLocation: SourceLocation = #_sourceLocation) { do { - let attachmentCopy = try withUnsafeBufferPointer { buffer in + let attachmentCopy = try withUnsafeBytes { buffer in let attachableContainer = AnyAttachable(attachableValue: Array(buffer)) return Attachment( _attachableValue: attachableContainer, @@ -286,10 +311,14 @@ extension Attachment where AttachableValue: ~Copyable { /// /// The testing library uses this function when writing an attachment to a /// test report or to a file on disk. This function calls the - /// ``Attachable/withUnsafeBufferPointer(for:_:)`` function on this - /// attachment's ``attachableValue-2tnj5`` property. - @inlinable public borrowing func withUnsafeBufferPointer(_ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { - try attachableValue.withUnsafeBufferPointer(for: self, body) + /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's + /// ``attachableValue-2tnj5`` property. + /// + /// @Metadata { + /// @Available(Swift, introduced: 6.2) + /// } + @inlinable public borrowing func withUnsafeBytes(_ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + try attachableValue.withUnsafeBytes(for: self, body) } } @@ -321,7 +350,7 @@ extension Attachment where AttachableValue: ~Copyable { /// This function is provided as a convenience to allow tools authors to write /// attachments to persistent storage the same way that Swift Package Manager /// does. You are not required to use this function. - @_spi(Experimental) @_spi(ForToolsIntegrationOnly) + @_spi(ForToolsIntegrationOnly) public borrowing func write(toFileInDirectoryAtPath directoryPath: String) throws -> String { try write( toFileInDirectoryAtPath: directoryPath, @@ -391,7 +420,7 @@ extension Attachment where AttachableValue: ~Copyable { // There should be no code path that leads to this call where the attachable // value is nil. - try withUnsafeBufferPointer { buffer in + try withUnsafeBytes { buffer in try file!.write(buffer) } diff --git a/Sources/Testing/Testing.docc/Attachments.md b/Sources/Testing/Testing.docc/Attachments.md new file mode 100644 index 000000000..cb75eea0f --- /dev/null +++ b/Sources/Testing/Testing.docc/Attachments.md @@ -0,0 +1,32 @@ +# Attachments + + + +Attach values to tests to help diagnose issues and gather feedback. + +## Overview + +Attach values such as strings and files to tests. Implement the ``Attachable`` +protocol to create your own attachable types. + +## Topics + +### Attaching values to tests + +- ``Attachment`` +- ``Attachable`` +- ``AttachableContainer`` + + diff --git a/Sources/Testing/Testing.docc/Documentation.md b/Sources/Testing/Testing.docc/Documentation.md index 901c0e3a6..cc4001889 100644 --- a/Sources/Testing/Testing.docc/Documentation.md +++ b/Sources/Testing/Testing.docc/Documentation.md @@ -69,3 +69,7 @@ their problems. ### Test customization - + +### Data collection + +- diff --git a/Sources/Testing/Testing.docc/MigratingFromXCTest.md b/Sources/Testing/Testing.docc/MigratingFromXCTest.md index 44d91b5b0..9e7842ecb 100644 --- a/Sources/Testing/Testing.docc/MigratingFromXCTest.md +++ b/Sources/Testing/Testing.docc/MigratingFromXCTest.md @@ -742,6 +742,66 @@ suite serially: For more information, see . +### Attach values + +In XCTest, you can create an instance of [`XCTAttachment`](https://developer.apple.com/documentation/xctest/xctattachment) +representing arbitrary data, files, property lists, encodable objects, images, +and other types of information that would be useful to have available if a test +fails. Swift Testing has an ``Attachment`` type that serves much the same +purpose. + +To attach a value from a test to the output of a test run, that value must +conform to the ``Attachable`` protocol. The testing library provides default +conformances for various standard library and Foundation types. + +If you want to attach a value of another type, and that type already conforms to +[`Encodable`](https://developer.apple.com/documentation/swift/encodable) or to +[`NSSecureCoding`](https://developer.apple.com/documentation/foundation/nssecurecoding), +the testing library automatically provides a default implementation: + +@Row { + @Column { + ```swift + // Before + import Foundation + + class Tortilla: NSSecureCoding { /* ... */ } + + func testTortillaIntegrity() async { + let tortilla = Tortilla(diameter: .large) + ... + let attachment = XCTAttachment( + archivableObject: tortilla + ) + self.add(attachment) + } + ``` + } + @Column { + ```swift + // After + import Foundation + + struct Tortilla: Codable, Attachable { /* ... */ } + + @Test func tortillaIntegrity() async { + let tortilla = Tortilla(diameter: .large) + ... + let attachment = Attachment(tortilla) + attachment.attach() + } + ``` + } +} + +If you have a type that does not (or cannot) conform to `Encodable` or +`NSSecureCoding`, or if you want fine-grained control over how it is serialized +when attaching it to a test, you can provide your own implementation of +``Attachable/withUnsafeBytes(for:_:)``. + + + ## See Also - diff --git a/Tests/TestingTests/AttachmentTests.swift b/Tests/TestingTests/AttachmentTests.swift index 0a220552a..b56ece4d9 100644 --- a/Tests/TestingTests/AttachmentTests.swift +++ b/Tests/TestingTests/AttachmentTests.swift @@ -8,11 +8,11 @@ // See https://swift.org/CONTRIBUTORS.txt for Swift project authors // -@testable @_spi(Experimental) @_spi(ForToolsIntegrationOnly) import Testing +@testable @_spi(ForToolsIntegrationOnly) import Testing private import _TestingInternals #if canImport(Foundation) import Foundation -@_spi(Experimental) import _Testing_Foundation +import _Testing_Foundation #endif #if canImport(CoreGraphics) import CoreGraphics @@ -265,7 +265,7 @@ struct AttachmentTests { #expect(attachment.preferredName == temporaryFileName) #expect(throws: Never.self) { - try attachment.withUnsafeBufferPointer { buffer in + try attachment.withUnsafeBytes { buffer in #expect(buffer.count == data.count) } } @@ -297,7 +297,7 @@ struct AttachmentTests { } #expect(attachment.preferredName == "\(temporaryDirectoryName).zip") - try! attachment.withUnsafeBufferPointer { buffer in + try! attachment.withUnsafeBytes { buffer in #expect(buffer.count > 32) #expect(buffer[0] == UInt8(ascii: "P")) #expect(buffer[1] == UInt8(ascii: "K")) @@ -391,7 +391,7 @@ struct AttachmentTests { } func open(_ attachment: borrowing Attachment) throws where T: Attachable { - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { bytes in + try attachment.attachableValue.withUnsafeBytes(for: attachment) { bytes in #expect(bytes.first == args.firstCharacter.asciiValue) let decodedStringValue = try args.decode(Data(bytes)) #expect(decodedStringValue == "stringly speaking") @@ -414,7 +414,7 @@ struct AttachmentTests { let attachableValue = MySecureCodingAttachable(string: "stringly speaking") let attachment = Attachment(attachableValue, named: "loremipsum.json") #expect(throws: CocoaError.self) { - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { _ in } + try attachment.attachableValue.withUnsafeBytes(for: attachment) { _ in } } } @@ -423,7 +423,7 @@ struct AttachmentTests { let attachableValue = MySecureCodingAttachable(string: "stringly speaking") let attachment = Attachment(attachableValue, named: "loremipsum.gif") #expect(throws: CocoaError.self) { - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { _ in } + try attachment.attachableValue.withUnsafeBytes(for: attachment) { _ in } } } #endif @@ -435,7 +435,7 @@ extension AttachmentTests { func test(_ value: some Attachable) throws { #expect(value.estimatedAttachmentByteCount == 6) let attachment = Attachment(value) - try attachment.withUnsafeBufferPointer { buffer in + try attachment.withUnsafeBytes { buffer in #expect(buffer.elementsEqual("abc123".utf8)) #expect(buffer.count == 6) } @@ -528,7 +528,7 @@ extension AttachmentTests { let image = try Self.cgImage.get() let attachment = Attachment(image, named: "diamond") #expect(attachment.attachableValue === image) - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { buffer in + try attachment.attachableValue.withUnsafeBytes(for: attachment) { buffer in #expect(buffer.count > 32) } attachment.attach() @@ -540,7 +540,7 @@ extension AttachmentTests { let image = try Self.cgImage.get() let attachment = Attachment(image, named: "diamond", as: type, encodingQuality: quality) #expect(attachment.attachableValue === image) - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { buffer in + try attachment.attachableValue.withUnsafeBytes(for: attachment) { buffer in #expect(buffer.count > 32) } if let ext = type?.preferredFilenameExtension { @@ -553,7 +553,7 @@ extension AttachmentTests { @Test func cannotAttachCGImageWithNonImageType() async { await #expect(exitsWith: .failure) { let attachment = Attachment(try Self.cgImage.get(), named: "diamond", as: .mp3) - try attachment.attachableValue.withUnsafeBufferPointer(for: attachment) { _ in } + try attachment.attachableValue.withUnsafeBytes(for: attachment) { _ in } } } #endif @@ -567,7 +567,7 @@ struct MyAttachable: Attachable, ~Copyable { var string: String var errorToThrow: (any Error)? - func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { if let errorToThrow { throw errorToThrow } @@ -585,7 +585,7 @@ extension MyAttachable: Sendable {} struct MySendableAttachable: Attachable, Sendable { var string: String - func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { #expect(attachment.attachableValue.string == string) var string = string return try string.withUTF8 { buffer in @@ -597,7 +597,7 @@ struct MySendableAttachable: Attachable, Sendable { struct MySendableAttachableWithDefaultByteCount: Attachable, Sendable { var string: String - func withUnsafeBufferPointer(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { + func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R { var string = string return try string.withUTF8 { buffer in try body(.init(buffer)) From bd29a044b7ddaf686ba4c1fdb717e92577c3cb33 Mon Sep 17 00:00:00 2001 From: Jonathan Grynspan Date: Mon, 3 Mar 2025 16:32:48 -0500 Subject: [PATCH 2/2] Move proposal to Swift Evolution --- Documentation/Proposals/NNNN-attachments.md | 436 -------------------- 1 file changed, 436 deletions(-) delete mode 100644 Documentation/Proposals/NNNN-attachments.md diff --git a/Documentation/Proposals/NNNN-attachments.md b/Documentation/Proposals/NNNN-attachments.md deleted file mode 100644 index ce3d5efde..000000000 --- a/Documentation/Proposals/NNNN-attachments.md +++ /dev/null @@ -1,436 +0,0 @@ -# Attachments - -* Proposal: [SWT-NNNN](NNNN-attachments.md) -* Authors: [Jonathan Grynspan](https://github.com/grynspan) -* Status: **Awaiting review** -* Bug: [swiftlang/swift-testing#714](https://github.com/swiftlang/swift-testing/issues/714) -* Implementation: [swiftlang/swift-testing#796](https://github.com/swiftlang/swift-testing/pull/796) -* Review: ([pitch](https://forums.swift.org/...)) - -## Introduction - -Test authors frequently need to include out-of-band data with tests that can be -used to diagnose issues when a test fails. This proposal introduces a new API -called "attachments" (analogous to the same-named feature in XCTest) as well as -the infrastructure necessary to create new attachments and handle them in tools -like VS Code. - -## Motivation - -When a test fails, especially in a remote environment like CI, it can often be -difficult to determine what exactly has gone wrong. Data that was produced -during the test can be useful, but there is currently no mechanism in Swift -Testing to output arbitrary data other than via `stdout`/`stderr` or via an -artificially-generated issue. A dedicated interface for attaching arbitrary -information to a test would allow test authors to gather relevant information -from a test in a structured way. - -## Proposed solution - -We propose introducing a new type to Swift Testing, `Attachment`, that represents -some arbitrary "attachment" to associate with a test. Along with `Attachment`, -we will introduce a new protocol, `Attachable`, to which types can conform to -indicate they can be attached to a test. - -Default conformances to `Attachable` will be provided for standard library types -that can reasonably be attached. We will also introduce a **cross-import overlay** -with Foundation—that is, a tertiary module that is automatically imported when -a test target imports both Foundation _and_ Swift Testing—that includes -additional conformances for Foundation types such as `Data` and `URL` and -provides support for attaching values that also conform to `Encodable` or -`NSSecureCoding`. - -## Detailed design - -The `Attachment` type is defined as follows: - -```swift -/// A type describing values that can be attached to the output of a test run -/// and inspected later by the user. -/// -/// Attachments are included in test reports in Xcode or written to disk when -/// tests are run at the command line. To create an attachment, you need a value -/// of some type that conforms to ``Attachable``. Initialize an instance of -/// ``Attachment`` with that value and, optionally, a preferred filename to use -/// when writing to disk. -public struct Attachment: ~Copyable where AttachableValue: Attachable & ~Copyable { - /// A filename to use when writing this attachment to a test report or to a - /// file on disk. - /// - /// The value of this property is used as a hint to the testing library. The - /// testing library may substitute a different filename as needed. If the - /// value of this property has not been explicitly set, the testing library - /// will attempt to generate its own value. - public var preferredName: String { get } - - /// The value of this attachment. - public var attachableValue: AttachableValue { get } - - /// Initialize an instance of this type that encloses the given attachable - /// value. - /// - /// - Parameters: - /// - attachableValue: The value that will be attached to the output of the - /// test run. - /// - preferredName: The preferred name of the attachment when writing it to - /// a test report or to disk. If `nil`, the testing library attempts to - /// derive a reasonable filename for the attached value. - /// - sourceLocation: The source location of the call to this initializer. - /// This value is used when recording issues associated with the - /// attachment. - public init( - _ attachableValue: consuming AttachableValue, - named preferredName: String? = nil, - sourceLocation: SourceLocation = #_sourceLocation - ) - - /// Attach this instance to the current test. - /// - /// - Parameters: - /// - sourceLocation: The source location of the call to this function. - /// - /// When attaching a value of a type that does not conform to both - /// [`Sendable`](https://developer.apple.com/documentation/swift/sendable) and - /// [`Copyable`](https://developer.apple.com/documentation/swift/copyable), - /// the testing library encodes it as data immediately. If the value cannot be - /// encoded and an error is thrown, that error is recorded as an issue in the - /// current test and the attachment is not written to the test report or to - /// disk. - /// - /// An attachment can only be attached once. - public consuming func attach(sourceLocation: SourceLocation = #_sourceLocation) - - /// Call a function and pass a buffer representing the value of this - /// instance's ``attachableValue-2tnj5`` property to it. - /// - /// - Parameters: - /// - body: A function to call. A temporary buffer containing a data - /// representation of this instance is passed to it. - /// - /// - Returns: Whatever is returned by `body`. - /// - /// - Throws: Whatever is thrown by `body`, or any error that prevented the - /// creation of the buffer. - /// - /// The testing library uses this function when writing an attachment to a - /// test report or to a file on disk. This function calls the - /// ``Attachable/withUnsafeBytes(for:_:)`` function on this attachment's - /// ``attachableValue-2tnj5`` property. - @inlinable public borrowing func withUnsafeBytes( - _ body: (UnsafeRawBufferPointer) throws -> R - ) throws -> R -} - -extension Attachment: Copyable where AttachableValue: Copyable {} -extension Attachment: Sendable where AttachableValue: Sendable {} -``` - -With `Attachment` comes `Attachable`, a protocol to which "attachable values" -conform: - -```swift -/// A protocol describing a type that can be attached to a test report or -/// written to disk when a test is run. -/// -/// To attach an attachable value to a test report or test run output, use it to -/// initialize a new instance of ``Attachment``, then call -/// ``Attachment/attach(sourceLocation:)``. An attachment can only be attached -/// once. -/// -/// The testing library provides default conformances to this protocol for a -/// variety of standard library types. Most user-defined types do not need to -/// conform to this protocol. -/// -/// A type should conform to this protocol if it can be represented as a -/// sequence of bytes that would be diagnostically useful if a test fails. If a -/// type cannot conform directly to this protocol (such as a non-final class or -/// a type declared in a third-party module), you can create a container type -/// that conforms to ``AttachableContainer`` to act as a proxy. -public protocol Attachable: ~Copyable { - /// An estimate of the number of bytes of memory needed to store this value as - /// an attachment. - /// - /// The testing library uses this property to determine if an attachment - /// should be held in memory or should be immediately persisted to storage. - /// Larger attachments are more likely to be persisted, but the algorithm the - /// testing library uses is an implementation detail and is subject to change. - /// - /// The value of this property is approximately equal to the number of bytes - /// that will actually be needed, or `nil` if the value cannot be computed - /// efficiently. The default implementation of this property returns `nil`. - /// - /// - Complexity: O(1) unless `Self` conforms to `Collection`, in which case - /// up to O(_n_) where _n_ is the length of the collection. - var estimatedAttachmentByteCount: Int? { get } - - /// Call a function and pass a buffer representing this instance to it. - /// - /// - Parameters: - /// - attachment: The attachment that is requesting a buffer (that is, the - /// attachment containing this instance.) - /// - body: A function to call. A temporary buffer containing a data - /// representation of this instance is passed to it. - /// - /// - Returns: Whatever is returned by `body`. - /// - /// - Throws: Whatever is thrown by `body`, or any error that prevented the - /// creation of the buffer. - /// - /// The testing library uses this function when writing an attachment to a - /// test report or to a file on disk. The format of the buffer is - /// implementation-defined, but should be "idiomatic" for this type: for - /// example, if this type represents an image, it would be appropriate for - /// the buffer to contain an image in PNG format, JPEG format, etc., but it - /// would not be idiomatic for the buffer to contain a textual description of - /// the image. - borrowing func withUnsafeBytes(for attachment: borrowing Attachment, _ body: (UnsafeRawBufferPointer) throws -> R) throws -> R - - /// Generate a preferred name for the given attachment. - /// - /// - Parameters: - /// - attachment: The attachment that needs to be named. - /// - suggestedName: A suggested name to use as the basis of the preferred - /// name. This string was provided by the developer when they initialized - /// `attachment`. - /// - /// - Returns: The preferred name for `attachment`. - /// - /// The testing library uses this function to determine the best name to use - /// when adding `attachment` to a test report or persisting it to storage. The - /// default implementation of this function returns `suggestedName` without - /// any changes. - borrowing func preferredName(for attachment: borrowing Attachment, basedOn suggestedName: String) -> String -} -``` - -Default conformances to `Attachable` are provided for: - -- `Array`, `ContiguousArray`, and `ArraySlice` -- `String` and `Substring` -- `Data` (if Foundation is also imported) - -Default _implementations_ are provided for types when they conform to -`Attachable` and either `Encodable` or `NSSecureCoding` (or both.) To use these -conformances, Foundation must be imported because `JSONEncoder` and -`PropertyListEncoder` are members of Foundation, not the Swift standard library. - -Some types cannot conform directly to `Attachable` because they require -additional information to encode correctly, or because they are not directly -`Sendable` or `Copyable`. A second protocol, `AttachableContainer`, is provided -that refines `Attachable`: - -```swift -/// A protocol describing a type that can be attached to a test report or -/// written to disk when a test is run and which contains another value that it -/// stands in for. -/// -/// To attach an attachable value to a test report or test run output, use it to -/// initialize a new instance of ``Attachment``, then call -/// ``Attachment/attach(sourceLocation:)``. An attachment can only be attached -/// once. -/// -/// A type can conform to this protocol if it represents another type that -/// cannot directly conform to ``Attachable``, such as a non-final class or a -/// type declared in a third-party module. -public protocol AttachableContainer: Attachable, ~Copyable { - /// The type of the attachable value represented by this type. - associatedtype AttachableValue - - /// The attachable value represented by this instance. - var attachableValue: AttachableValue { get } -} - -extension Attachment where AttachableValue: AttachableContainer & ~Copyable { - /// The value of this attachment. - /// - /// When the attachable value's type conforms to ``AttachableContainer``, the - /// value of this property equals the container's underlying attachable value. - /// To access the attachable value as an instance of `T` (where `T` conforms - /// to ``AttachableContainer``), specify the type explicitly: - /// - /// ```swift - /// let attachableValue = attachment.attachableValue as T - /// ``` - public var attachableValue: AttachableValue.AttachableValue { get } -} -``` - -The cross-import overlay with Foundation also provides the following convenience -interface for attaching the contents of a file or directory on disk: - -```swift -extension Attachment where AttachableValue == _AttachableURLContainer { - /// Initialize an instance of this type with the contents of the given URL. - /// - /// - Parameters: - /// - url: The URL containing the attachment's data. - /// - preferredName: The preferred name of the attachment when writing it to - /// a test report or to disk. If `nil`, the name of the attachment is - /// derived from the last path component of `url`. - /// - sourceLocation: The source location of the call to this initializer. - /// This value is used when recording issues associated with the - /// attachment. - /// - /// - Throws: Any error that occurs attempting to read from `url`. - public init( - contentsOf url: URL, - named preferredName: String? = nil, - sourceLocation: SourceLocation = #_sourceLocation - ) async throws -} -``` - -`_AttachableURLContainer` is a type that conforms to `AttachableContainer` and -encloses the URL and corresponding mapped data. As an implementation detail, it -is omitted from this proposal for brevity. - -## Source compatibility - -This proposal is additive and has no impact on existing code. - -## Integration with supporting tools - -We will add a new command-line argument to the `swift test` command in Swift -Package Manager: - -```sh ---attachments-path Path where attachments should be saved. -``` - -If specified, an attachment will be written to that path when its `attach()` -method is called. If not specified, attachments are not saved to disk. Tools -that indirectly use Swift Testing through `swift test` can specify a path (e.g. -to a directory created inside the system's temporary directory), then move or -delete the created files as needed. - -The JSON event stream ABI will be amended correspondingly: - -```diff ---- a/Documentation/ABI/JSON.md -+++ b/Documentation/ABI/JSON.md - ::= { - "kind": , - "instant": , ; when the event occurred - ["issue": ,] ; the recorded issue (if "kind" is "issueRecorded") -+ ["attachment": ,] ; the attachment (if kind is "valueAttached") - "messages": , - ["testID": ,] - } - - ::= "runStarted" | "testStarted" | "testCaseStarted" | - "issueRecorded" | "testCaseEnded" | "testEnded" | "testSkipped" | -- "runEnded" ; additional event kinds may be added in the future -+ "runEnded" | "valueAttached"; additional event kinds may be added in the future - -+ ::= { -+ "path": , ; the absolute path to the attachment on disk -+} -``` - -As these changes are additive only, the JSON schema version does not need to be -incremented to support them. We are separately planning to increment the JSON -schema version to support other features; these changes will apply to the newer -version too. - -## Future directions - -- Attachment lifetime management: XCTest's attachments allow for specifying a - "lifetime", with two lifetimes currently available: - - ```objc - typedef NS_ENUM(NSInteger, XCTAttachmentLifetime) { - XCTAttachmentLifetimeKeepAlways = 0, - XCTAttachmentLifetimeDeleteOnSuccess = 1 - }; - ``` - - If a test passes, it is probably not necessary to keep its attachments saved - to disk. The exact "shape" this feature should take in Swift Testing is not - yet clear. - -- Image attachments: it is often useful to be able to attach images to tests, - however there is no cross-platform solution for this functionality. An - experimental implementation that allows attaching an instance of `CGImage` (on - Apple platforms) is available in Swift Testing's repository and shows what it - might look like for us to provide this functionality. - -- Additional conformances for types in other modules: in order to keep Swift - Testing's dependency graph as small as possible, we cannot link it to - arbitrary packages such as (for example) swift-collections even if it would be - useful to do so. That means we can't directly provide conformances to - `Attachable` for types in those modules. Adding additional cross-import - overlays would allow us to provide those conformances when both Swift Testing - and those packages are imported at the same time. - - This functionality may require changes in Swift Package Manager that are - beyond the scope of this proposal. - -- Adopting `RawSpan` instead of `UnsafeRawBufferPointer`: `RawSpan` represents a - safer alternative to `UnsafeRawBufferPointer`, but it is not yet available - everywhere we'd need it in the standard library, and our minimum deployment - targets on Apple's platforms do not allow us to require the use of `RawSpan` - (as no shipping version of Apple's platforms includes it.) - -- Adding an associated `Metadata` type to `Attachable` allowing for inclusion of - arbitrary out-of-band data to attachments: we see several uses for such a - feature: - - - Fine-grained control of the serialization format used for `Encodable` types; - - Metrics (scaling factor, rotation, etc.) for images; and - - Compression algorithms to use for attached files and directories. - - The exact shape of this interface needs further consideration, but it could be - added in the future without disrupting the interface we are proposing here. - [swiftlang/swift-testing#824](https://github.com/swiftlang/swift-testing/pull/824) - includes an experimental implementation of this feature. - -## Alternatives considered - -- Doing nothing: there's sufficient demand for this feature that we know we want - to address it. - -- Reusing the existing `XCTAttachment` API from XCTest: while this would - _probably_ have saved me a lot of typing, `XCTAttachment` is an Objective-C - class and is only available on Apple's platforms. The open-source - swift-corelibs-xctest package does not include it or an equivalent interface. - As well, this would create a dependency on XCTest in Swift Testing that does - not currently exist. - -- Implementing `Attachment` as a non-generic type and eagerly serializing - non-sendable or move-only attachable values: an earlier implementation did - exactly this, but it forced us to include an existential box in `Attachment` - to store the attachable value, and that would preclude ever supporting - attachments in Embedded Swift. - -- Having `Attachment` take a byte buffer rather than an attachable value, or - having it take a closure that returns a byte buffer: this would just raise the - problem of attaching arbitrary values up to the test author's layer, and that - would no doubt produce a lot of duplicate implementations of "turn this value - into a byte buffer" while also worsening the interface's ergonomics. - -- Adding a `var contentType: UTType { get set }` property to `Attachment` or to - `Attachable`: `XCTAttachment` lets you specify a Uniform Type Identifier that - tells Xcode the type of data. Uniform Type Identifiers are proprietary and not - available on Linux or Windows, and adding that property would force us to also - add a public dependency on the `UniformTypeIdentifiers` framework and, - indirectly, on Foundation, which would prevent Foundation from authoring tests - using Swift Testing in the future due to the resulting circular dependency. - - We considered using a MIME type instead, but there is no portable mechanism - for turning a MIME type into a path extension, which is ultimately what we - need when writing an attachment to persistent storage. - - Instead, `Attachable` includes the function `preferredName(for:basedOn:)` that - allows an implementation (such as that of `Encodable & Attachable`) to add a - path extension to the filename specified by the test author if needed. - -## Acknowledgments - -Thanks to Stuart Montgomery and Brian Croom for goading me into finally writing -this proposal! - -Thanks to Wil Addario-Turner for his feedback, in particular around `UTType` and -MIME type support. - -Thanks to Honza Dvorsky for his earlier work on attachments in XCTest and his -ideas on how to improve Swift Testing's implementation.