Skip to content

Skipped unit tests should not show as "passed" #6167

Open
@lbussell

Description

@lbussell

We have lots of instances of skipping tests in this manner:

if (imageData.ImageVariant.HasFlag(DotNetImageVariant.AOT))
{
OutputHelper.WriteLine(
$"Test is not applicable to AOT images. See {nameof(VerifyAotWebScenario)} instead.");
return;
}

However, returning early from a test will count as a "passed" test in Xunit, even if we didn't assert anything. This can lead to a false sense of confidence in test outcomes, especially for tests which are intended to be skipped temporarily.

We should find a way to make these tests show as "skipped" instead, like we do with Theories that have no data due to test filtering.

Xunit V3 (which is in preview) supports dynamic skipping of tests natively, but not Xunit V2. There's opportunity to use a library like Xunit.SkippableFact, which is used by several other Microsoft projects: Xunit.SkippableFact > used by ....

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    • Status

      Current Release

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions