[CONSIDER]: Using Microsoft.Testing.Platform in templates

[Youssef1313]

Currently, there are templates for MSTest/NUnit that are VSTest. Consider adding an option for MTP.

For xunit.v3, consider also making the template based on MTP v2 via https://www.nuget.org/packages/xunit.v3.mtp-v2.

Note that for MTP templates, you need to have a global.json specifying MTP.

[gasparnagy]

@Youssef1313 I think nothing speaks against this. Could you please make a PR for this?

[Youssef1313]

Unfortunately I don't currently have the capacity to do PRs for this (as well as the other issues I opened). But I'll happily answer any questions you have.

[304NotModified]

For xunit.v3, consider also making the template based on MTP v2 via https://www.nuget.org/packages/xunit.v3.mtp-

Please note this is xunit3 v3 and we are supporting xunit3 v2

[gasparnagy]

Is there any breaking change in xunit 3 v3 that makes Reqnroll not to work?

[clrudolphi]

Is this a suggestion that Reqnroll migrate its templates to MTP or that it support either?

[clrudolphi]

Picking this thread back up again. I have been looking at the Reqnroll templates (both the dotnet add templates and those generated by the VisualStudio Extension). Attempting to support both VSTest and MTP is complicated, as there is more to it than establishing which package references and versions to use. The following table should give some sense of the complexity involved. Depending upon which test framework and version, tfm, and whether the user wants VSTest or MTP, we may need to change the project's SDK, the set of of package references, and/or set properties within the .csproj file.

.NET Test Framework Package Reference Matrix

Framework	Version	Min .NET Framework	Min .NET (Core/5+)	VSTest — Required Package References	MTP — Required Package References	Notes
NUnit	3.x	3.5+	.NET Core 2.0 / .NET Standard 2.0+	NUnit NUnit3TestAdapter (v4.x) Microsoft.NET.Test.Sdk	NUnit NUnit3TestAdapter (v5.x for MTP 1.x; v6.x for MTP 2.0)	Microsoft.NET.Test.Sdk is included for VSTest. NUnit3TestAdapter v4.x handles both NUnit 3.x and 4.x test discovery under VSTest. For MTP, Microsoft.NET.Test.Sdk can be dropped unless VSTest fallback is desired.
NUnit	4.x	4.6.2+	.NET 6.0+	NUnit NUnit3TestAdapter (v4.5+) Microsoft.NET.Test.Sdk	NUnit NUnit3TestAdapter (v5.x for MTP 1.x; v6.x for MTP 2.0)	Adapter v6.x adds MTP 2.0 support and drops .NET Core 3; minimum becomes .NET 8.0.
xUnit	2.x	4.7.2+	.NET Core 1.0 / .NET Standard 1.0+	xunit xunit.runner.visualstudio Microsoft.NET.Test.Sdk	xunit + MTP bridge package. xunit.runner.visualstudio and Microsoft.NET.Test.Sdk can be safely removed if VSTest support is no longer needed.	xUnit v2 MTP support is via an opt-in bridge (requires <OutputType>Exe</OutputType>), not native.
xUnit	3.x	4.7.2+	.NET 8.0+	xunit.v3 xunit.runner.visualstudio Microsoft.NET.Test.Sdk	xunit.v3 only. MTP support is built natively into xUnit.net v3 — xunit.runner.visualstudio and Microsoft.NET.Test.Sdk can be removed if relying solely on MTP. Add <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport> to use MTP via dotnet test.	Both platforms can coexist. Default via dotnet test is still VSTest unless TestingPlatformDotnetTestSupport is set.
MSTest	2.x	4.5+	.NET Core 1.0 / .NET Standard 1.0+	MSTest.TestFramework MSTest.TestAdapter Microsoft.NET.Test.Sdk	❌ Not supported — must upgrade to v3	Microsoft.NET.Test.Sdk is explicitly needed for VSTest integration and test host support in non-SDK-style MSTest projects.
MSTest	3.x (standard packages)	4.6.2+	.NET 6.0+	MSTest.TestFramework MSTest.TestAdapter Microsoft.NET.Test.Sdk	MSTest.TestFramework MSTest.TestAdapter + <EnableMSTestRunner>true</EnableMSTestRunner> <OutputType>Exe</OutputType> <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport>	The MSTest meta-package is the recommended alternative — it bundles Microsoft.NET.Test.Sdk, MSTest.TestAdapter, MSTest.TestFramework, and MSTest.Analyzers in one reference.
MSTest	4.x (MSTest.Sdk project SDK)	4.6.2+	.NET 6.0+	Add <UseVSTest>true</UseVSTest> — no explicit PackageReference needed; SDK manages all	No explicit PackageReference needed. MSTest SDK defaults to MTP with EnableMSTestRunner and TestingPlatformDotnetTestSupport both set to true.	Simplest setup — project declares Sdk="MSTest.Sdk" instead of Sdk="Microsoft.NET.Sdk".
TUnit	1.x	❌ Not supported	.NET 8.0+ only	❌ Not supported	TUnit (meta-package) — no Microsoft.NET.Test.Sdk, no adapter	Microsoft.NET.Test.Sdk is not declared for TUnit because TUnit is exclusively built on top of MTP. Run via dotnet run or dotnet test with MTP mode.

---

Key caveats

Microsoft.NET.Test.Sdk across all frameworks: This package is not needed for MTP as it is linked to VSTest. It is declared for NUnit, xUnit, and transitively for MSTest to ensure older versions of Test Explorer can fall back to VSTest mode, and to fall back to VSTest if TestingPlatformDotnetTestSupport is disabled. For TUnit it is not declared because TUnit is only built on top of MTP.

dotnet test in MTP mode: For all frameworks except TUnit and MSTest.Sdk, MTP is opt-in when using dotnet test. You must add <TestingPlatformDotnetTestSupport>true</TestingPlatformDotnetTestSupport> to the project file; otherwise dotnet test defaults to VSTest even if MTP packages are present.

xUnit v2 MTP support: This is not native — it requires a separate third-party or community bridge package and <OutputType>Exe</OutputType>. For any greenfield project, prefer v3 if MTP is a goal.

As a product, do we want/need Reqnroll to emit bulletproof project templates or do we expect users to have the responsibility of crafting a project file that fits their need (and Reqnroll templates can be 'close enough')?

[gasparnagy]

Thanks for the detailed analysis.

I have found this thread from the MTP team, that is fresh (2 months old). It is worth reading through the comments, they are well showing how people feel about MTP now. https://www.reddit.com/r/dotnet/comments/1qazgd9/testing_in_net_survey_about_test_platforms/

My key findings from the answers:

• 56% of the responders did not even hear about MTP, only 28% moved or considered to move
• The key blockers of moving to MTP:
  • unclear what the benefits are
  • complicated and unclear configuration (e.g. global.json), it is not enabled by default
  • downstream tooling issues (test runners, CI/CD servers, coverage tools)
  • version confusions MTPv1, MTPv2, different configuration path for the different runners

My personal feeling was similar on my own existing projects: the RoI of migration was unclear or low, but I would use MTP for any new projects.

As a product, do we want/need Reqnroll to emit bulletproof project templates or do we expect users to have the responsibility of crafting a project file that fits their need (and Reqnroll templates can be 'close enough')?

• I think the new Reqnroll project templates are used for new projects, so fundamentally it would be possible to generate them with MTP.
• I don't think it is our responsibility for be able to provide a template for all combinations. I.e., if we generate MTP projects with the template, and the user needs non-MTP, they need to figure out themselves, how to do it.
• Whatever we choose, we should generate projects that work out of the box without any additional tweak, configuration, etc. I don't have a full overview now, but for example if the user still needs to add/modify global.json (last time I tried I could not avoid it), than it is a blocker in my opinion. Our goal with the templates to enable users to try out Reqnroll easily. If they need to make global, solution level decisions (e.g. adding a global.json that influences all other projects as well), people would rather not use our templates or would not try Reqnroll.
• My feeling is that the configuration/enabling workflow for MTP is a bit unclear still. So if we move our templates to MTP, we might need to keep updating them as the configuration changes.

Does this answer your questions?

And maybe a totally independent idea: Reqnroll is basically a simple extension to the test projects: you just need to add a NuGet package and that's all. The best would be if our templates could just build upon the existing templates of the test execution frameworks: e.g. create an nunit project with the nunit template and add the Reqnroll.NUnit package and some sample content files to it. I'm not sure this is possible with the existing template capabilities... But conceptually we could check how the current mstest/nunit/etc templates work and just do the same.

[clrudolphi]

That clarifies that we should not be aiming to cover all situations and combinations. For those configurations we do choose to support with templates (or other onboarding mechanisms) we should aim for a 'it just works' outcome.

Given the market confusion over MTP, perhaps we pause on that? While we might pause on making MTP the default for all, we should proceed with ensuring our TUnit template is correct since it requires MTP.

The latter remarks about simplifying the installation process is interesting and we should discuss further. Are you willing to remove the "Reqnroll Project" from the VisualStudio New Project Wizard? (If we wish to deploy as simply as adding a reference to an existing (_x_Unit) project, then it follows that there is no such thing as a Reqnroll project.) Likewise, we could drop the command-line templates as well.

[gasparnagy]

Given the market confusion over MTP, perhaps we pause on that? While we might pause on making MTP the default for all, we should proceed with ensuring our TUnit template is correct since it requires MTP.

Agree.

The latter remarks about simplifying the installation process is interesting and we should discuss further. Are you willing to remove the "Reqnroll Project" from the VisualStudio New Project Wizard? (If we wish to deploy as simply as adding a reference to an existing (_x_Unit) project, then it follows that there is no such thing as a Reqnroll project.) Likewise, we could drop the command-line templates as well.

Good questions, but I would not remove it from the VS extension. :) It sounds illogical, but the concept is that the VS extension basically promotes the first steps with the project templates. So makes them more discoverable.

For the NuGet based dotnet new templates, you anyway need quite some knowledge to know that you can install the template package, so if you have found that and installed it, you anyway know probably how Reqnroll works. The 3.3.3 version of the template package was downloaded only 1900 times and that includes bots (same version of Reqnroll is downloaded 477000).

But maybe it's just me. I use the VS project wizards often for trying out things, but never or very rarely used dotnet new.