Skip to content

[BUG]: VSTest@2 is passing while there are reported test failures when using batching #20788

Open
@Youssef1313

Description

@Youssef1313

New issue checklist

Task name

No response

Task version

No response

Issue Description

https://dev.azure.com/uno-platform/Uno%20Platform/_build/results?buildId=152367&view=logs&j=c814f4c4-6968-579a-5adb-420b4397838c&t=fad2c059-e56b-516e-11ce-011393560ef1

YML:

  - task: VSTest@2
    inputs:
      testAssemblyVer2: |
        **\*test*.dll
        !**\obj\**
        !**\*Wasm.Test*.dll
        !**\*UITests.dll
        !**\*.RuntimeTests.dll
        !**\Uno.UI.Tests.ViewLibraryProps.dll
        !**\Uno.UI.Tests.ViewLibrary.dll
        !**\testhost*.*
        !**\Microsoft*.dll
      vsTestVersion: toolsInstaller
      testRunTitle: $(Agent.JobName)
      testSelector: testAssemblies
      batchingBasedOnAgentsOption: customBatchSize
      rerunFailedTests: 'true'
      customBatchSizeValue: 200 # test count / 10 (https://developercommunity.visualstudio.com/content/problem/891803/vstestconsoleadapter-fails-with-outofmemory.html?childToView=896206#comment-896206)

Logs show:

Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,NetCore,XR:False,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,MonoVM,XR:False,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,NetCore,XR:True,)
Failed    Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService.HR(System.String,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Scenario,Uno.UI.SourceGenerators.Tests.MetadataUpdateTests.Given_HotReloadService+Project[])    HR (When_Simple_Xaml_Add_xBind_Function_Property_Add,Debug,MonoVM,XR:True,)

Looking at the run via https://dev.azure.com/uno-platform/Uno%20Platform/_TestManagement/Runs#_a=resultQuery&runId=3201282&queryPath=Recent+Run%2FRun3201282 doesn't show the failure. However, downloaded the relevant trx from the attachments on the same link shows the failure. The end result is that the job is passing when there are test failures.

The job will fail correctly if batching is not used (i.e, delete batchingBasedOnAgentsOption and customBatchSizeValue from YML)

FYI @jeromelaban @MartinZikmund. Though you are no longer impacted as Uno is now using Microsoft.Testing.Platform with the proper dotnet test support.

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Relevant log output

-

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION] 

Repro steps

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions