Description
The ability to run all library tests locally without failures should be supported, even though it is time-consuming and somewhat rare to want to do this.
I would like to do this before and after making lower-level changes such as for reflection or other runtime-overlapping changes. There are other reasons to want this including verifying that the environment is correct.
Without this, I depend on CI to catch library test failures (and runtime test failures - more on this later).
Expect: add test and CI infrastructure to run all inner loop library tests as a single unit in order to emulate the local results. Perhaps this would run in the outerloop pipeline since it is time consuming. Having CI support is necessary to prevent future breaks. This should also be done in a non-elevated command prompt. In some cases, having multiple test projects run at the same time can result in conflicts -- especially networking tests -- so there may still be flakiness if we don't have a way to prevent that from occurring.
Currently for me, running this on Windows
build -s clr+libs+libs.tests -rc release -test
results in several failures and a hang:
System.DirectoryServices.Tests Total: 255, Errors: 0, Failed: 1, Skipped: 29, Time: 2.878s
System.Net.Quic.Functional.Tests Total: 441, Errors: 0, Failed: 1, Skipped: 1, Time: 189.488s
System.Security.Cryptography.Tests Total: 10125, Errors: 0, Failed: 19, Skipped: 114, Time: 71.050s
System.Net.Security.Tests Total: 5177, Errors: 0, Failed: 27, Skipped: 21, Time: 50.719s
(and then a hang on HttpListener.Tests)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerResponseHeadersTests.StatusDescription_GetWithCustomStatusCode_ReturnsExpected', Elapsed: 00:02:07
(running again I get another hang)
System.Net.HttpListener.Tests: [Long Running Test] 'System.Net.Tests.HttpListenerRequestTests.ProtocolVersion_GetProperty_ReturnsExpected', Elapsed: 00:08:03
Longer-term
The ideal solution would be to easily specify a "smokescreen" set of tests. For starting, this would be projects that catch lower-level runtime issues, which me for would include specific libraries including System.Runtime, System.Text.Json, System.Runtime.Serialization, System.ComponentModel.TypeConverter and Microsoft.Extensions.DependencyInjection. Today, this can be emulated using && to combine multiple commands, but perhaps this could be done in a single command that would allow parallelization.
Even better, the ability to specify a set of runtime tests along with this would be convenient, although today the runtime test infrastructure is different from the library test infrastructure -- e.g. how to build, how to run, and concepts such as test priority. So doing this in an integrated way may not be feasible unless we have a way to share or wrap these two test infrastructures with similar commands.