UI Test results should be part of Test report interface in DevOps

[michael-hawker]

Describe the bug

DevOps has a dashboard for seeing the test results of a build. This currently only shows our unit test results as they're MSBuild based. The UI Tests use TAEF to run in the CI. We need to convert those result outputs into a format the DevOps dashboard understands.

This will make it easier to see and look up test failures when they occur.

FYI @azchohfi

@kmahone any tips/pointers on how this is configured for WinUI in your repo?

[kmahone]

We don't run our tests directly on the build machine, rather we run them in Helix. But we face the same issue: we need to convert the TAEF logs to a a different format. In our case we convert from TAEF logs to XUnit logs (which is what Helix supports).

It looks like WindowsAppSDK are already running TAEF tests in their CI Pipeline using VSTest. I assume it is using an adapter of some kind. @jefgen can you share how this works?

[jefgen]

Sure, I can try to share what I know... However, to be honest, I couldn't find much documentation about it -- so I'm not very well versed in how it works.

The "Microsoft.Taef" Nuget package includes a test adapter called "Taef.TestAdapter". This adapter allows for running TAEF tests using VSTest, either in a Azure DevOps build pipeline, on the command line with vstest/mstest, or with the Visual Studio GUI Test Explorer.

Looking at the release notes for TAEF, it looks like it was originally added back in v6.1.161103008.
From https://www.osgwiki.com/wiki/TAEF_Release_Notes

Features

• TAEF now has a supported VSTest adapter so you can run TAEF tests in using Test Explorer, vstest.console.exe, and VSTS.

The only documentation I could find on it is here:
https://www.osgwiki.com/wiki/Automate_unit_test_run_in_the_Visual_Studio_Build
https://www.osgwiki.com/wiki/TAEF_Test_Adapter_Settings

The first link describes how the use the Nuget package in a build pipeline, and has links to an example pipeline for a Calculator project.

According to some older threads I've seen about it, there are no TAEF logs (ie: WTL format) produced when executing the tests this way. Rather, it directly produces Visual Studio Test (TRX) files instead, which are understood natively by the VSTest task and the ADO Publish results task.

I'm not entirely sure if it internally converts from WTL to TRX, or if is able to directly output just the TRX results.
(I don't know of any way to use this to convert existing WTL logs to TRX, for example)

For the WindowsAppSDK, for some of the MRT Core tests, we use the VSTest task in the build pipeline here to run Unit Tests which are TAEF based. These tests use the TAEF nuget here.
(Example logs: https://dev.azure.com/ms/ceef01e9-e5e2-46d8-bbe1-c83450ca8182/_apis/build/builds/200525/logs/122)

When they run, the VSTest uses the "Taef.TestAdapter", passed by the /TestAdapterPath:"D:\a\1\s\BuildOutput\Release\x64\UnitTest" parameter, which invokes the TAEF test and outputs the results as TRX files.