Switch to VSTest in CI

docs/coding-guidelines/adding-api-guidelines.md

@@ -38,7 +38,7 @@ should be added to `net5.0`. [More Information on TargetFrameworks](https://docs
 **Update tests**
   - Add new `TargetFramework` to the ```TargetFrameworks```.
   - Add new test code following [conventions](project-guidelines.md#code-file-naming-conventions) for new files to that are specific to the new target framework.
-  - To run just the new test targetFramework run `dotnet build <Library>.csproj -f <TargetFramework> /t:Test`. TargetFramework should be chosen only from supported TargetFrameworks.
+  - To run just the new test targetFramework run `dotnet test <Library>.csproj -f <TargetFramework>`. TargetFramework should be chosen only from supported TargetFrameworks.
 
 ## Documentation
 

docs/workflow/building/libraries/README.md

@@ -23,7 +23,7 @@ cd tests
 
 :: Then inner loop build / test
 :: (If using Visual Studio, you might run tests inside it instead)
-pushd ..\src & dotnet build & popd & dotnet build /t:test
+pushd ..\src & dotnet build & popd & dotnet test
 ```
 
 The instructions for Linux and macOS are essentially the same:
@@ -43,7 +43,7 @@ cd src/libraries/System.Text.RegularExpressions
 cd tests
 
 # Then inner loop build / test:
-pushd ../src & dotnet build & popd & dotnet build /t:test
+pushd ../src & dotnet build & popd & dotnet test
 ```
 
 The steps above may be all you need to know to make a change. Want more details about what this means? Read on.

docs/workflow/building/libraries/code-coverage.md

@@ -34,17 +34,17 @@ This runs the tests and generates the full code coverage report. The resulting i
 
     artifacts\coverage\index.htm
 
-You can also build and test with code coverage for a particular test project rather than for the whole repo with the `/p:Coverage=true` property:
+You can also build and test with code coverage for a particular test project rather than for the whole repo with the `--collect:"XPlat Code Coverage"` switch:
 
-    dotnet build /t:Test /p:Coverage=true
+    dotnet test --collect:"XPlat Code Coverage"
 
 The results for this one library will then be available in this index.htm file, where $(OutDir) is the directory where the binaries were generated.
 
     $(OutDir)\report\index.htm
 
 For example, to build, test, and get code coverage results for the System.Diagnostics.Debug library, from the root of the repo one can do:
 
-    dotnet build src\System.Diagnostics.Debug\tests /t:Test /p:Coverage=true
+    dotnet test src\System.Diagnostics.Debug\tests --collect:"XPlat Code Coverage"
 
 And then once the run completes:
 

docs/workflow/debugging/libraries/debugging-packages.md

@@ -72,7 +72,7 @@ Build:
 
 To validate the content of the nupkg, change the extension to .zip. As before, use an IL disassembler to verify that the right APIs are present within `ref\<platform>\contractname.dll` and the right implementations within the `lib\<platform>\contractname.dll`.
 
-4. Run the tests from `\tests`: `dotnet build /t:test`
+4. Run the tests from `\tests`: `dotnet test`
 
 Ensure that the test is referencing the correct pkg. For example:
 ```
@@ -96,10 +96,11 @@ ResolvePkgProjReferences:
 To run a test from a single Build Pivot combination, specify all properties and build the `csproj`:
 
 ```
-dotnet build System.Net.ServicePoint.Tests.csproj -f netcoreapp2.0 /t:test /p:OuterLoop=true /p:xunitoptions=-showprogress
+dotnet test System.Net.ServicePoint.Tests.csproj -f net5.0 /p:OuterLoop=true
 ```
 Will run the test using the following pivot values:
 * Architecture: AnyCPU
-* Flavor: Debug
-* OS: Windows_NT
-* Target: netstandard2.0
+* Configuration: Debug
+* TargetOS: Windows_NT
+* TargetFramework: net5.0
+* OuterLoop: true

docs/workflow/debugging/libraries/unix-instructions.md

@@ -5,7 +5,7 @@ CoreFX can be debugged on unix using both lldb and visual studio code
 
 ## Using lldb and SOS
 
-- Run the test using msbuild at least once with `/t:Test`.
+- Run the test using the SDK at least once with `dotnet test`.
 - [Install version 3.9 of lldb](../coreclr/debugging.md#debugging-core-dumps-with-lldb) and launch lldb with dotnet as the process and arguments matching the arguments used when running the test through msbuild.
 - Load the sos plugin using `plugin load libsosplugin.so`.
 - Type `soshelp` to get help. You can now use all sos commands like `bpmd`.

docs/workflow/debugging/libraries/windows-instructions.md

@@ -29,19 +29,10 @@ You may need to do this for both x64 and x86 versions.
 Any application that crashes should now automatically start a WinDBG session.
 
 ## Debugging tests
-To run a single test from command line:
-
-* Locate the test binary folder based on the CSPROJ name.
-
-For example: `src\System.Net.Sockets\tests\Functional\System.Net.Sockets.Tests.csproj` will build and output binaries at  `bin\tests\Windows_NT.AnyCPU.Debug\System.Net.Sockets.Tests\netcoreapp1.0`.
-
-* Execute the test
-
-Assuming that your repo is at `C:\corefx`:
+Assuming that your repo is at `C:\runtime`:
 
 ```
-cd C:\corefx\bin\tests\Windows_NT.AnyCPU.Debug\System.Net.Sockets.Tests\netcoreapp1.0
-C:\corefx\bin\tests\Windows_NT.AnyCPU.Debug\System.Net.Sockets.Tests\netcoreapp1.0\CoreRun.exe xunit.console.dll System.Net.Sockets.Tests.dll -xml testResults.xml -notrait category=nonwindowstests -notrait category=OuterLoop -notrait category=failing
+dotnet test C:\runtime\src\libraries\System.Net.Sockets\tests
 ```
 
 * If the test crashes or encounters a `Debugger.Launch()` method call, WinDBG will automatically start and attach to the `CoreRun.exe` process
@@ -52,7 +43,7 @@ The following commands will properly configure the debugging extension and fix s
 .symfix
 .srcfix
 .reload
-!load C:\corefx\packages\runtime.win7-x64.Microsoft.NETCore.Runtime.CoreCLR\<version>\tools\sos
+!load C:\runtime\coreclr\runtime.win7-x64.Microsoft.NETCore.Runtime.CoreCLR\<version>\tools\sos
 ```
 
 _Important_: Pass in the correct path to your SOS extension discovered during the Prerequisites, step 2.
@@ -168,6 +159,3 @@ Helper scripts are available at https://github.com/dotnet/runtime/tree/master/sr
 * `*System.Threading.Tasks.TplEventSource {2e5dba47-a3d2-4d16-8ee0-6671ffdcd7b5}`: Provides an event source for tracing TPL information.
 * `*System.Threading.Tasks.Parallel.EventSource`: Provides an event source for tracing TPL information.
 * `*System.Threading.Tasks.Dataflow.DataflowEventSource {16F53577-E41D-43D4-B47E-C17025BF4025}`: Provides an event source for tracing Dataflow information.
-
-## Notes
-* You can find the test invocation command-line by looking at the logs generated after the `dotnet build /t:test` within the test folder.

docs/workflow/testing/libraries/filtering-tests.md

@@ -5,13 +5,13 @@ The tests can be filtered based on xunit trait attributes defined in [`Microsoft
 #### OuterLoopAttribute
 
 ```cs
-[OuterLoop()]
+[OuterLoop]
 ```
-Tests marked as `OuterLoop` are for scenarios that don't need to run every build. They may take longer than normal tests, cover seldom hit code paths, or require special setup or resources to execute. These tests are excluded by default when testing through `dotnet build` but can be enabled manually by adding the `-testscope outerloop` switch or `/p:TestScope=outerloop` e.g.
+Tests marked as `OuterLoop` are for scenarios that don't need to run every build. They may take longer than normal tests, cover seldom hit code paths, or require special setup or resources to execute. These tests are excluded by default when testing through `dotnet test` but can be enabled manually by adding the `-testscope outerloop` switch or `/p:TestScope=outerloop` e.g.
 
 ```cmd
 build -test -testscope outerloop
-cd src/System.Text.RegularExpressions/tests && dotnet build /t:Test /p:TestScope=outerloop
+dotnet test src/System.Text.RegularExpressions/tests /p:TestScope=outerloop
 ```
 
 #### PlatformSpecificAttribute
@@ -22,17 +22,13 @@ cd src/System.Text.RegularExpressions/tests && dotnet build /t:Test /p:TestScope
 Use this attribute on test methods to specify that this test may only be run on the specified platforms. This attribute returns the following categories based on platform
 - `nonwindowstests` for tests that don't run on Windows
 - `nonlinuxtests` for tests that don't run on Linux
-- `nonosxtests` for tests that don't run on OS X
+- `nonosxtests` for tests that don't run on macOS
 
 **[Available Test Platforms](https://github.com/dotnet/arcade/blob/master/src/Microsoft.DotNet.XUnitExtensions/src/TestPlatforms.cs)**
 
 When running tests by building a test project, tests that don't apply to the `TargetOS` are not run. For example, to run Linux-specific tests on a Linux box, use the following command line:
 ```sh
-dotnet build <csproj_file> /t:Test /p:TargetOS=Linux
-```
-To run all Linux-compatible tests that are failing:
-```sh
-dotnet build <csproj_file> /t:Test /p:TargetOS=Linux /p:WithCategories=failing
+dotnet test <csproj_file> /p:TargetOS=Linux
 ```
 
 #### ActiveIssueAttribute
@@ -148,9 +144,11 @@ _**A few common examples with the above attributes:**_
 
 - Run all tests acceptable on Windows that are not failing:
 ```cmd
-dotnet build <csproj_file> /t:Test /p:TargetOS=Windows_NT
+dotnet test <csproj_file> /p:TargetOS=Windows_NT
 ```
 - Run all outer loop tests acceptable on OS X that are currently associated with active issues:
 ```sh
-dotnet build <csproj_file> /t:Test /p:TargetOS=OSX /p:WithCategories="OuterLoop;failing""
+dotnet test <csproj_file> /p:TargetOS=OSX /p:WithCategories="OuterLoop;failing""
 ```
+
+Note: The `TargetOS` property can usually be omitted as it defaults to the OS.

docs/workflow/testing/libraries/testing-android.md

@@ -71,7 +71,7 @@ Make sure an emulator is booted (see `AVD Manager`) or a device is plugged in an
 ### Running individual test suites
 The following shows how to run tests for a specific library
 ```
-./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
+./dotnet.sh test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
 ```
 
 ### Test App Design

docs/workflow/testing/libraries/testing-apple.md

@@ -19,7 +19,7 @@ In order to run the tests on a device you need to specify `DevTeamProvisioning`
 ### Running individual test suites
 - The following shows how to run tests for a specific library:
 ```
-./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
+./dotnet.sh test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=iOS /p:TargetArchitecture=x64
 ```
 
 ### Test App Design

docs/workflow/testing/libraries/testing-wasm.md

@@ -35,14 +35,14 @@ and even run tests one by one for each library:
 ### Running individual test suites
 The following shows how to run tests for a specific library
 ```
-./dotnet.sh build /t:Test src/libraries/System.AppContext/tests /p:TargetOS=Browser /p:TargetArchitecture=wasm /p:Configuration=Release
+./dotnet.sh test src/libraries/System.AppContext/tests /p:TargetOS=Browser /p:TargetArchitecture=wasm -c Release
 ```
 
 ### Running tests using different JavaScript engines
 It's possible to set a JavaScript engine explicitly by adding `/p:JSEngine` property:
 
 ```
-./dotnet.sh build /t:Test src/libraries/System.AppContext/tests /p:TargetOS=Browser /p:TargetArchitecture=wasm /p:Configuration=Release /p:JSEngine=SpiderMonkey
+./dotnet.sh test src/libraries/System.AppContext/tests /p:TargetOS=Browser /p:TargetArchitecture=wasm -c Release /p:JSEngine=SpiderMonkey
 ```
 
 At the moment supported values are:

docs/workflow/testing/libraries/testing.md

@@ -7,59 +7,63 @@ To build the tests and run them you can call the libraries build script.
 **Examples**
 - The following shows how to build only the tests but not run them:
 ```
-build.cmd/sh -subset libs.tests
+build.cmd/sh libs.tests
 ```
 
 - The following builds and runs all tests in release configuration:
 ```
-build.cmd/sh -subset libs.tests -test -c Release
+build.cmd/sh libs.tests -test -c Release
 ```
 
 - The following example shows how to pass extra msbuild properties to ignore tests ignored in CI:
 ```
-build.cmd/sh -subset libs.tests -test /p:WithoutCategories=IgnoreForCI
+build.cmd/sh libs.tests -test /p:WithoutCategories=IgnoreForCI
 ```
 
 Unless you specifiy `-testnobuild`, test assemblies are implicitly built when invoking the `Test` action.
 - The following shows how to only test the libraries without building them
 ```
-build.cmd/sh -subset libs.tests -test -testnobuild
+build.cmd/sh -test -testnobuild
 ```
 
 ## Running tests on the command line
 
-To build tests you need to specify the `test` subset when invoking build.cmd/sh: `build.cmd/sh -subset libs.tests`.
+To build tests you need to specify the `libs.tests` subset when invoking build.cmd/sh: `build.cmd/sh libs.tests`.
 
-The easiest (and recommended) way to build and run the tests for a specific library, is to invoke the `Test` target on that library:
+The easiest (and recommended) way to build and run the tests for a specific library, is to invoke `dotnet test` on that library:
 ```cmd
-cd src\libraries\System.Collections.Immutable\tests
-dotnet build /t:Test
+dotnet test src\libraries\System.Collections.Immutable\tests
 ```
 
-It is possible to pass parameters to the underlying xunit runner via the `XUnitOptions` parameter, e.g.:
+It is possible to pass parameters to the underlying VSTest runner, e.g.:
 ```cmd
-dotnet build /t:Test /p:XUnitOptions="-class Test.ClassUnderTests"
+dotnet test --filter Test.ClassUnderTests
 ```
 
+The available arguments for `dotnet test` are documented [here](https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test) and advanced filtering options can be found [here](https://docs.microsoft.com/en-us/dotnet/core/testing/selective-unit-tests?pivots=xunit). 
+
 There may be multiple projects in some directories so you may need to specify the path to a specific test project to get it to build and run the tests.
 
 #### Running a single test on the command line
 
-To quickly run or debug a single test from the command line, set the XunitMethodName property, e.g.:
+To quickly run or debug a single test from the command line, supply a filter argument value, e.g.:
 ```cmd
-dotnet build /t:Test /p:XunitMethodName={FullyQualifiedNamespace}.{ClassName}.{MethodName}
+dotnet test --filter {FullyQualifiedNamespace}.{ClassName}.{MethodName}
 ```
 
 #### Running outer loop tests
 
 To run all tests, including "outer loop" tests (which are typically slower and in some test suites less reliable, but which are more comprehensive):
 ```cmd
-dotnet build /t:Test /p:Outerloop=true
+dotnet test /p:Outerloop=true
 ```
 
-#### Running tests on a different target framework
+#### Running tests on a single target framework
 
-Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. By default we will build and run only the default build target framework which is `net5.0`. The rest of the `TargetFrameworks` will need to be built and ran by specifying the `BuildTargetFramework` option, e.g.:
+Each test project can potentially have multiple target frameworks. There are some tests that might be OS-specific, or might be testing an API that is available only on some target frameworks, so the `TargetFrameworks` property specifies the valid target frameworks. To run against a single target framework you can specify the `TargetFramework` via the framework `-f` switch:
 ```cmd
-dotnet build src\libraries\System.Runtime\tests\System.Runtime.Tests.csproj /p:BuildTargetFramework=net472
+dotnet test -f net48 src\libraries\System.Text.RegularExpressions\tests
 ```
+
+#### Using an alternative test runner (advanced)
+For non mobile configurations, the default test runner is VSTest (locally and in CI). By passing in `/p:TestRunner=xunit` when invoking `dotnet test`, an alternative test runner `xunit.console` is used. Even though that test runner is out-of-support and deprecated, it is still useful when debugging runtime issues or when adding support for new operating systems or devices. Our plan is to eventually get rid of `xunit.console` by adding a lightweight mode to the VSTest runner.

docs/workflow/testing/mono/testing.md

@@ -32,7 +32,7 @@ build.cmd /p:RuntimeFlavor=mono
 3. Run the tests
 
 ```
-dotnet build /t:Test /p:RuntimeFlavor=mono
+dotnet test /p:RuntimeFlavor=mono
 ```
 
 # Patching Local dotnet (.dotnet-mono)

eng/Subsets.props

@@ -243,7 +243,7 @@
   </ItemGroup>
 
   <ItemGroup Condition="$(_subset.Contains('+libs.tests+'))">
-    <ProjectToBuild Include="$(LibrariesProjectRoot)tests.proj" Category="libs" Test="true" />
+    <ProjectToBuild Include="$(LibrariesProjectRoot)tests.proj" Category="libs" Test="true" Publish="true" />
   </ItemGroup>
 
   <!-- Installer sets -->

eng/Version.Details.xml

@@ -87,6 +87,10 @@
       <Sha>f69d7fc09c4fdb9e9427741b9a176e867dab577f</Sha>
     </Dependency>
     <Dependency Name="Microsoft.NET.Test.Sdk" Version="16.8.0-release-20201009-01">
+      <Uri>https://github.com/microsoft/vstest</Uri>
+      <Sha>2418d9ec492e93245e1ce8699d6ae9cd7b86649b</Sha>
+    </Dependency>
+    <Dependency Name="Microsoft.TestPlatform.CLI" Version="16.8.0-release-20201009-01">
       <Uri>https://github.com/microsoft/vstest</Uri>
       <Sha>0b1e2e51743cc083d99b88673fe518672e2af9f0</Sha>
     </Dependency>

eng/Versions.props

@@ -103,6 +103,9 @@
     <MicrosoftBclAsyncInterfacesVersion>1.1.1</MicrosoftBclAsyncInterfacesVersion>
     <MicrosoftWin32PrimitivesVersion>4.3.0</MicrosoftWin32PrimitivesVersion>
     <runtimenativeSystemIOPortsVersion>5.0.0-alpha.1.19563.3</runtimenativeSystemIOPortsVersion>
+    <!-- VSTest dependencies -->
+    <MicrosoftNETTestSdkVersion>16.8.0-release-20201009-01</MicrosoftNETTestSdkVersion>
+    <MicrosoftTestPlatformCLIVersion>16.8.0-release-20201009-01</MicrosoftTestPlatformCLIVersion>
     <!-- Runtime-Assets dependencies -->
     <SystemComponentModelTypeConverterTestDataVersion>5.0.0-beta.20505.2</SystemComponentModelTypeConverterTestDataVersion>
     <SystemDrawingCommonTestDataVersion>5.0.0-beta.20505.2</SystemDrawingCommonTestDataVersion>
@@ -136,7 +139,6 @@
     <RefOnlyNugetProjectModelVersion>4.9.4</RefOnlyNugetProjectModelVersion>
     <RefOnlyNugetPackagingVersion>4.9.4</RefOnlyNugetPackagingVersion>
     <!-- Testing -->
-    <MicrosoftNETTestSdkVersion>16.8.0-release-20201009-01</MicrosoftNETTestSdkVersion>
     <MicrosoftDotNetXHarnessTestRunnersXunitVersion>1.0.0-prerelease.20516.1</MicrosoftDotNetXHarnessTestRunnersXunitVersion>
     <MicrosoftDotNetXHarnessCLIVersion>1.0.0-prerelease.20516.1</MicrosoftDotNetXHarnessCLIVersion>
     <XUnitVersion>2.4.1</XUnitVersion>

eng/build.ps1

@@ -202,6 +202,7 @@ foreach ($argument in $PSBoundParameters.Keys)
     "framework"              { $arguments += " /p:BuildTargetFramework=$($PSBoundParameters[$argument].ToLowerInvariant())" }
     "os"                     { $arguments += " /p:TargetOS=$($PSBoundParameters[$argument])" }
     "allconfigurations"      { $arguments += " /p:BuildAllConfigurations=true" }
+    "coverage"               { $arguments += " /p:VSTestCollect='XPlat Code Coverage'" }
     "properties"             { $arguments += " " + $properties }
     "verbosity"              { $arguments += " -$argument " + $($PSBoundParameters[$argument]) }
     # configuration and arch can be specified multiple times, so they should be no-ops here

eng/build.sh

@@ -303,7 +303,7 @@ while [[ $# > 0 ]]; do
       ;;
 
      -coverage)
-      arguments="$arguments /p:Coverage=true"
+      arguments="$arguments /p:VSTestCollect='XPlat Code Coverage'"
       shift 1
       ;;
 

eng/pipelines/common/templates/innerloop-steps.yml

@@ -0,0 +1,17 @@
+parameters:
+  buildConfig: ''
+  archType: ''
+  testLibrary: ''
+  testRunner: ''
+  condition: always()
+
+steps:
+  # Build and test leaf solution
+  - ${{ if ne(parameters.testLibrary, '') }}:
+    - script: dotnet test
+              $(Build.SourcesDirectory)/src/libraries/${{ parameters.testLibrary }}/
+              -c ${{ parameters.buildConfig }}
+              /p:TargetArchitecture=${{ parameters.archType }}
+              /p:TestRunner=${{ parameters.testRunner }}
+              /bl:$(Build.SourcesDirectory)/artifacts/log/$(_BuildConfig)/innerloop-steps-test.binlog
+      displayName: Build and test leaf solution

eng/pipelines/global-build.yml

@@ -41,13 +41,18 @@ jobs:
   parameters:
     jobTemplate: /eng/pipelines/common/global-build-job.yml
     buildConfig: release
+    timeoutInMinutes: 90
     platforms:
     - Windows_NT_x86
     - OSX_x64
     jobParameters:
       testGroup: innerloop
       nameSuffix: Runtime_Debug
       buildArgs: -c release -runtimeConfiguration debug
+      extraStepsTemplate: /eng/pipelines/common/templates/innerloop-steps.yml
+      extraStepsParameters:
+        testLibrary: System.AppContext
+        testRunner: xunit
 
 #
 # Build with Debug config and Release runtimeConfiguration