MSTest v4 Breaking Changes

[Evangelink]

Description

The goal of this ticket is to announce the upcoming breaking changes in MSTest v4.

Planned

• Convert MSTest to async pattern (Convert MSTest to async pattern #1284)
  • Youssef note: So far, nothing I see we can improve further.
• TestMethodAttribute.Execute should become TestMethodAttribute.ExecuteAsync #5310
  • User impact: Custom TestMethodAttribute implementations need to change signature to be Async. There is not heavy usage of custom TestMethodAttribute in user codebases, so it's too easy and straightforward to fix (and we can provide automation for it, if needed). This change fixes deadlocks that can happen in some cases due to blocking via GetAwaiter().GetResult()
• Disable appdomain usage by default (Disable appdomain usage by default #1297)
  • Most users will only notice very big performance improvements with this change, without affecting anything for them. In very very niche cases where it affected users, they can add enable it back via runsettings.
• Change ClassCleanup default behavior to be "EndOfClass" #1316 / Remove ClassCleanupBehavior.EndOfAssembly #1574 (PR Remove ClassCleanupBehavior #5827)
  • We already have an analyzer for that with zero user complains in the past. The "default" behavior is almost always incorrect and not what the users want.
• Drop TestTimeout enum and related overload on TimeoutAttribute (Drop TestTimeout enum and related overload on TimeoutAttribute #1317)
  • API that no one uses. Replacement is simple, just replace TestTimeout.Infinite with int.MaxValue.
• Update TestContext.Properties return type to IDictionary<string, object> (Update TestContext.Properties return type to IDictionary<string, object> #1377)
  • Small impact on users.
• [Breaking][v4] Assembly version shouldn't be fixed to 14.0.0.0 #5690 (kinda linked to Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices errors due to path length not handled well on UWP #1607) - also let's change executor uri.
  • In most cases this will be transparent for users, but has high impact on reliability of MSTest.
• [v4] Broader changes for the story around attributes and their inheritance #4292
• [Breaking][v4] Remove [DataTestMethod] #4166
• [MSTest v4][breaking] Use CallerFilePath and CallerLineNumber in TestMethodAttribute #5500
• Return T from Assert.IsInstanceOfType<T>() #1573
• [Breaking] Remove ITestMethod interface in platform services in v4 #4724
• Reading runsettings shouldn't use ReadToFollowing #5800
• [MSTest v4] Consider removing CssIterationAttribute, CssProjectStructureAttribute, and DescriptionAttribute #5787
• DataRow: Nulls and empty string are displayed as no data in display name (DataRow: Nulls and empty string are displayed as no data in display name #1647) ([Breaking][v4] Remove TestIdGenerationStrategy #5757)
• Remove deprecated TestContext properties that were deprecated as part of Deprecate TestContext properties kept for compatibility #1453 - [Breaking][v4] Remove deprecated TestContext properties #5759
• MSTest v4 TestContext properties should not be nullable #1403
• Add relaxed versions of Assert.ThrowsException and Assert.ThrowsExceptionAsync #4257 (comment)
• Deprecate ExpectedExceptionAttribute (Deprecate ExpectedException attribute. #135) - Bump analyzer in 3.10 to be warning, and drop in v4
• [v4][Breaking] Review analyzer serverities #5748
• Use a better serialization/deserialization mechanism for data discovery (Use a better serialization/deserialization mechanism for data discovery #1462) (Related PR: Adjust TestCase.Id to account for randomized tests (include test case index instead of serialized data) #5871)
• [MSTest][v4] STAThreadAttribute should be sealed #5693
• Make UseCooperativeCancellationForTimeoutAnalyzer warning #5879
• Drop support for < net8.0 in MSTest and MTP #5614
• Move unfolding strategy from data sources to test method #5876
• [Breaking][v4] MSTest.TestAdapter (and consequently MSTest metapackage) should drop netstandard2.0 #5645
• Caller argument expression (Openness for Caller Argument Expression in Assert failure messages #1154)
  • Part of the work is being done in Add CallerArgumentExpression overloads to Assert.IsTrue/IsFalse/IsNull/IsNotNull #5851
  • Open question Openness for Caller Argument Expression in Assert failure messages #1154 (comment)
• [MSTest v4] Fail when discovery fails on a class #5327
• [Breaking][v4] Don't ignore ReflectionTypeLoadExceptions #5680
• Include parameter types in test id #5899

Follow-ups

• Analyzers for the caller info changes of TestMethodAttribute
  • Warn when a single string argument is explicitly passed to TestMethodAttribute. This likely used to mean "display name".
  • Warn when the attribute isn't syntactically declared just above the method. Having the attribute declared just above the method helps caller info be as close as possible to the method declaration.
  • When inheriting the attribute, ensure that there is a constructor that propagates caller info.
• Cleanup resource files (resx).
• Warning for no longer supported ClassCleanupLifecycle in runsettings/testconfig.json
• Add analyzer for Console.Write**, Trace.Write**, and Debug.Write** usage in tests to suggest using TestContext instead.
• Consolidate MicrosoftTestingTargetFrameworks and SupportedNetFrameworks

Delayed

• Consider enforcing MapInconclusiveToFailed in MTP #5584
• Timeout should produce dangling activities by default #2433
• Disable CaptureTraceOutput by default (linked issue TraceListenerManager is not thread safe #2616) ([Breaking][v4] Disable CaptureTraceOutput by default #5756)

Canceled

• Better DisplayName for data tests (Better DisplayNames for DataSource tests #713)
• Reintroduce tree structure (InnerResults) for parameterized tests (Data Driven tests - Individual results of data rows are not in tree structure, missing parentID in trx file #1024)
• [MSTest v4]: Discuss if MSTest.Sdk should set MSTestAnalysisMode to Recommended #4974
• [MSTest v4]: Discuss if the default of TestingPlatformDotnetTestSupport should match EnableMSTestRunner #4973 (we want to break the old dotnet test exp for MTP altogether, so this will become irrelevant)
• Remove Assert.Equals (Obsolete Assert.Equals, and add an obsolete ReferenceEquals #5758)
• Separate GetDisplayName out of ITestDataSource #5307
• Discuss whether TestMethodAttribute.Execute should return a single TestResult object #5309
• Align naming used for attributes/types allowing to control some behavior #1327

Done in v3 already

• Remove deprecated BeginTimer and EndTimer methods (Remove deprecated BeginTimer and EndTimer methods #1454)
• Deprecate TestContext properties kept for compatibility (Deprecate TestContext properties kept for compatibility #1453)
• Unify SetOutcome behavior across all target frameworks (TestContext: Unify SetOutcome behavior across TFMs #1426)

[Nirmal4G]

I also propose a new Shared Framework of testing tools and frameworks in .NET SDK along with a new Test SDK for the Project (not just a simple rename as requested in #2693). May be not for v4 but for v5 maybe? .NET 10? Basically, a built-in Testing support from the .NET SDK without any external dependencies.

See #3254 for the proposal and discussion.