Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Represent non-encodable test argument values in Test.Case.ID #1000

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

stmontgomery
Copy link
Contributor

This expands Test.Case.ID to represent combinations of arguments which aren't fully encodable, and uniquely distinguishes test cases associated with the same parameterized test function which have otherwise identical IDs.

Motivation:

It is possible to declare a parameterized test function with a collection of arguments that includes the same element more than once. As a trivial example:

@Test(arguments: [1, 1])
func repeatedArg(value: Int) { ... }

This can happen in more realistic scenarios if you dynamically construct a collection of arguments that accidentally or intentionally includes the same value more than once.

The testing library attempts to form a unique identifier for each argument passed to a parameterized test. It does so by checking whether the value conforms to Encodable, or one of the other protocols mentioned in the documentation (see Run selected test cases).

One problem with the current implementation is that if the value doesn't conform to one of those known protocols, the testing library gives up and doesn't produce a unique identifier for that test case at all. Specifically, in that situation the argumentIDs property of Test.Case.ID will have a value of nil.

Another problem is that if an argument is passed more than once, the derived identifiers for each argument's test case will be the same and the result reporting will be ambiguous at best; or worse, the lack of a unique identifier could cause an integrated tool to misbehave or crash. (Current versions of Xcode 16 experience this issue non-deterministically for projects which have test parallelization enabled.) To solve this, there needs to be a way to deterministically distinguish test cases which, from the testing library's perspective, appear identical—either because they actually are the same value, or because they encode to the same representation.

Modifications:

  • Add a new isStable boolean property to Test.Case.Argument.ID representing whether or not the testing library was able to encode a stable representation of the argument value it identifies.
  • Add a new SPI property named discriminator to Test.Case which distinguishes test cases associated with the same test function whose arguments are identical.
  • Add a corresponding property to Test.Case.ID with the same name, discriminator.
  • Change the Test.Case.ID.argumentIDs property to non-Optional, so that it always has a value even if one or more argument IDs is non-stable.
  • Add a derived boolean property isStable to Test.Case.ID whose value is true iff all of its argument IDs are stable.
  • Add Hashable conformance to Test.Case, since the reason for avoiding such conformance has been resolved.
  • Add new tests.

Checklist:

  • Code and documentation should follow the style of the Style Guide.
  • If public symbols are renamed or modified, DocC references should be updated.

Resolves #995
Resolves rdar://119522099

@stmontgomery stmontgomery added enhancement New feature or request tools integration Integration of swift-testing into tools/IDEs parameterized-testing Related to parameterized testing functionality labels Mar 6, 2025
@stmontgomery stmontgomery added this to the Swift 6.x milestone Mar 6, 2025
@stmontgomery stmontgomery self-assigned this Mar 6, 2025
@stmontgomery
Copy link
Contributor Author

@swift-ci please test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request parameterized-testing Related to parameterized testing functionality tools integration Integration of swift-testing into tools/IDEs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Represent non-encodable test argument values in Test.Case.ID
2 participants