Azure Pipelines no longer shows Tests UI with new SDK

[craigbrown]

I have updated my projects to use the new MSTest SDK:

- <Project Sdk="Microsoft.NET.Sdk">
+ <Project Sdk="MSTest.Sdk/3.4.1">
...

and I have the following task in my Azure Pipeline (I know the Azure tasks are a different repo, but bear with me):

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/**.sln'
    arguments: '--configuration Release'

which seems to translate to this command when it's actually run:

"C:\Program Files\dotnet\dotnet.exe" test D:\a\1\s\MySolution.sln --logger trx --results-directory D:\a\_temp --configuration Release

This used to result in a "Tests" tab appearing on the pipeline's run page with a nice UI of all the tests that were run and whether they passed or not. After the change to the MSTest.Sdk, this no longer appears and I also get this warning, which I think is related:

##[warning]No test result files were found.

I have searched the issues in this repo for any clue as to how to fix this. I've seen some mention of the trx part possibly being the culprit, so I followed the suggestion to add this to my csproj:

<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" Version="1.0.0" />

but this doesn't seem to make a difference.

Could you point me in the right direction to get this working? Or is it something that the Azure Pipelines team will need to fix on their end? Many thanks.

[Evangelink]

Hi @craigbrown,

First of all, sorry that the upgrade experience wasn't good for you. Let me give you some details about the change you did and what is missing.

Context

By default, dotnet test is calling vstest.console.exe which was the main platform for running tests until we introduced MSTest runner. The 2 platforms/runner are fundamentally different and uses different command line arguments. We then introduced MSTest SDK with the main goal to help simplify boilerplate configuration for our users and promote the new platform. As part of using MSTest.Sdk you are switching the default platform from vstest.console to MSTest runner (as explained here https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-sdk#select-the-runner) and we are also opting in switching dotnet test from calling vstest.console to be calling MSTest runner (this part is not well documented, I'll work on improving that), it's equivalent to be setting up https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode.

When you have moved from the regular SDK to MSTest.Sdk you are in one of the following 2 scenarios:

1. You were simply using MSTest (either through MSTest metapackage or through MSTest.TestAdapter and MSTest.TestFramework) but you didn't opt-in to the runner (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-runner-intro#enable-mstest-runner-in-a-mstest-project).

2. You were already using MSTest runner but you were not using the runner with dotnet test.

Answer to your question

Since you are using MSTest.Sdk you are already having the required extra packages added through the default extensions profile (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-mstest-sdk#mstest-runner-profile) so you only need to pass the argument required to activate the TRX (see https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-extensions-test-reports#visual-studio-test-reports).

So your yaml should be:

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/**.sln'
    arguments: '--configuration Release -- --report-trx'

Note the -- in front of --report-trx to let the SDK know the arguments after that separator are not for MSBuild but need to be flown down to the platform.

Please let me know if you have any question left.

[craigbrown]

Hi @Evangelink, thank you for your quick and thorough response! I tried your suggestion and unfortunately it doesn't fix the problem for me. I tried running the same command on my machine:

dotnet test --configuration Release -- --report-trx

It seems to run the tests correctly (the command line shows the correct number of passes and fails), but I get the same issue with no trx being output (I am looking in the TestResults folder based on the last link above).

[Evangelink]

Absolutely sorry I got confused with what is currently implemented (many phases to handle got me confused)... As explained in here https://learn.microsoft.com/dotnet/core/testing/unit-testing-platform-integration-dotnet-test#dotnet-test---microsofttestingplatform-mode, you will need to use -p:TestingPlatformCommandLineArguments="--report-trx".

So

dotnet test --configuration Release -p:TestingPlatformCommandLineArguments="--report-trx"

You can also use -p:TestingPlatformCaptureOutput=false in addition to the previous command to get a more detailed output.

[craigbrown]

Thanks @Evangelink! That does now produce the trx file when I run the command on my machine. Unfortunately it doesn't get picked up by Azure DevOps, but I assume this is something on their side, possibly because they're looking for the trx file in the wrong directory. I will open a separate issue in that repository.

[Evangelink]

@craigbrown I'll try to make some tests on AzDO to see whats going-on. Thanks for linking the issue on their side here, it's helping a lot with triaging and syncing between teams <3

If you have some time, would you mind looking at this doc change PR dotnet/docs#41658 and let me know if that seems to be providing enough info? It's sometimes hard as author to find the right level of details for our users.

[craigbrown]

@Evangelink sure! That looks good. I left a comment with the only change I think is needed to your example. Thanks again for the quick response to this.

[miegir]

Same problem for me. My yaml is:

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    arguments: '--no-build --configuration $(BuildConfiguration) -p:TestingPlatformCommandLineArguments="--report-trx --coverage"'

And I get exactly the same warning

##[warning]No test result files were found.

The complete command line ran by ADO is

D:\Agent\_work\_tool\dotnet\dotnet.exe test --logger trx --results-directory D:\Agent\_work\_temp --no-build --configuration Release "-p:TestingPlatformCommandLineArguments=--report-trx --coverage"

The tests are properly discovered after changing yaml to

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    arguments: '--no-build --configuration $(BuildConfiguration) -p:TestingPlatformCommandLineArguments="--report-trx --coverage  --results-directory $(Agent.TempDirectory)"'

I am not sure how to apply escaping for $(Agent.TempDirectory) that is already inside quotes, but if no spaces in the path it works. Surprisingly, ADO is not searching for test results inside $(Common.TestResultsDirectory) by the way. It would be ideal for dotnet test to continue using old arguments though.

[Evangelink]

It would be ideal for dotnet test to continue using old arguments though.

This is technically not possible because current dotnet test code was written with only VSTest in mind and not allowing any evolution. We are working toward a better experience for .NET 10 SDK.

@miegir for the escaping, the easiest is to use the -- syntax, so in your case:
arguments: '--no-build --configuration $(BuildConfiguration) -- --report-trx --coverage --results-directory $(Agent.TempDirectory)'

[nohwnd]

Please try using %22 (escaped double quote) to wrap the path:

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: 'test'
    arguments: '--no-build --configuration $(BuildConfiguration) -p:TestingPlatformCommandLineArguments="--report-trx --coverage  --results-directory %22$(Agent.TempDirectory)%22"'

Even if you write the yaml to wrap the -p:TestingPlatformCommandLineArguments with ' ' then your " " will be eaten by msbuild parsing, so you need to prevent that.