Test failure: profiler/transitions/transitions/transitions.cmd

[SakeTao]

Failed in: runtime-coreclr pgostress 20250411.1

Failed tests:

coreclr windows x64 Checked fullpgo_methodprofiling_always_optimized @ Windows.10.Amd64.Open
    - profiler/transitions/transitions/transitions.cmd

Error message:

 
Return code:      1
Raw output file:      C:hwC2CE0A3AwB7850954uploads	ransitions	ransitionsoutput.txt
Raw output:
BEGIN EXECUTION
 "C:hwC2CE0A3Apcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true"  transitions.dll 
Profiler path: C:hwC2CE0A3AwB7850954eprofiler	ransitions	ransitionsProfiler.dll
Profilee STDOUT: Profiler.dll!DllGetClassObject
Profilee STDOUT: Profiler.dll!Profiler::Initialize
Profilee STDOUT: PInvoke received i=13
Profilee STDOUT: Profiler.dll!Profiler::Shutdown
Profilee STDOUT: PROFILER TEST PASSES
Profiler path: C:hwC2CE0A3AwB7850954eprofiler	ransitions	ransitionsProfiler.dll
Profilee STDOUT: Profiler.dll!DllGetClassObject
Profilee STDOUT: Profiler.dll!Profiler::Initialize
Profilee STDOUT: PInvoke received i=13
Profilee STDOUT: Profiler.dll!Profiler::Shutdown
Profilee STDOUT: PROFILER TEST PASSES
Profiler path: C:hwC2CE0A3AwB7850954eprofiler	ransitions	ransitionsProfiler.dll
Profilee STDOUT: Profiler.dll!DllGetClassObject
Profilee STDOUT: Profiler.dll!Profiler::Initialize
Profilee STDOUT: PInvoke received i=1
Profilee STDOUT: Profiler.dll!Profiler::Shutdown
Test failed: Profiler tests are expected to contain the text 'PROFILER TEST PASSES' in the console output of the profilee app to indicate a passing test. Usually it is printed from the Shutdown() method of the profiler implementation. This text was not found in the output above. Profilee returned exit code 100.
System.Exception: Profiler tests are expected to contain the text 'PROFILER TEST PASSES' in the console output of the profilee app to indicate a passing test. Usually it is printed from the Shutdown() method of the profiler implementation. This text was not found in the output above. Profilee returned exit code 100.
   at Profiler.Tests.ProfilerTestRunner.FailFastWithMessage(String error)
   at Profiler.Tests.ProfilerTestRunner.Run(String profileePath, String testName, Guid profilerClsid, String profileeArguments, ProfileeOptions profileeOptions, Dictionary`2 envVars, String reverseServerName, Boolean loadAsNotification, Int32 notificationCopies)
   at Profiler.Tests.Transitions.RunProfilerTest(String testName, String pInvokeExpectedName, String reversePInvokeExpectedName)
Expected: 100
Actual: 103
END EXECUTION - FAILED
FAILED
Test failed. Trying to see if dump file was created in C:cores since 4/12/2025 7:35:28 AM
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:hwC2CE0A3AwB7850954eprofilerprofiler../transitions/transitions/transitions.cmd

Stack trace:

   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
   at Program.<<Main>$>g__TestExecutor19|0_20(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

[dotnet-policy-service]

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

[tommcdon]

@mdh1418 @BruceForstall

[mdh1418]

Given that we didn't see either of the results messages between Profiler.dll!Profiler::Shutdown and the profiler test process output validation then it seems the process exited/crashed during Profiler Shutdown.

Given that this only failed once, compared to #114594 and #114595 which were failing across the board, I think this is more of a test flake. And in the manual run of that pipeline, profiler/transitions/transitions/transitions.cmd passed.

Do we care about this scenario running with TieredPGO=1? We could set DOTNET_TieredPGO=0 to avoid a possible flake altogether.

[BruceForstall]

Do we care about this scenario running with TieredPGO=1? We could set DOTNET_TieredPGO=0 to avoid a possible flake altogether.

Note that DOTNET_TieredPGO=1 is the default.

I guess you have to decide why the test failed, and if it can be made resilient to this set of configuration variables, namely:

set DOTNET_TieredCompilation=1
set DOTNET_ReadyToRun=0
set DOTNET_TC_QuickJitForLoops=1
set DOTNET_TieredPGO=1
set DOTNET_JitDelegateProfiling=1
set DOTNET_JitVTableProfiling=1
set DOTNET_JitGuardedDevirtualizationMaxTypeChecks=3
set DOTNET_JitProfileCasts=1
set DOTNET_TieredPGO_InstrumentedTierAlwaysOptimized=1

[mdh1418]

@BruceForstall Could we skip the profiler tests outright on the pgostress pipeline? The failure does not seem related to the profiler.

Looking over the previous runs, this is the only failure over the past 90 runs in the past 30 days for this Pipeline. Moreover, it has not failed in the main pipeline over the past 30 days.

[BruceForstall]

@BruceForstall Could we skip the profiler tests outright on the pgostress pipeline? The failure does not seem related to the profiler.

Looking over the previous runs, this is the only failure over the past 90 runs in the past 30 days for this Pipeline. Moreover, it has not failed in the main pipeline over the past 30 days.

The only mechanism we have to disable tests related to JIT is JitOptimizationSensitive -- we can't disable for a particular pipeline -- but it doesn't know anything about the PGO environment variables listed above. They would have to be added to

runtime/src/tests/Common/CLRTest.Execute.Batch.targets

Line 100 in 9fd6a57

IF "%DOTNET_JitStress%"=="" IF "%DOTNET_JitStressRegs%"=="" IF "%DOTNET_JITMinOpts%"=="" IF "%DOTNET_TailcallStress%"=="" goto :Compatible

and

runtime/src/tests/Common/CLRTest.Execute.Bash.targets

Line 95 in 9fd6a57

if [[ ( ! -z "$DOTNET_JitStress" ) || ( ! -z "$DOTNET_JitStressRegs" ) || ( ! -z "$DOTNET_JITMinOpts" ) || ( ! -z "$DOTNET_TailcallStress" ) ]]

However, it's not clear to me that disabling for PGO makes sense; I'm not convinced we know the problem is due to PGO stress, or that it is due to an incompatibility, as opposed to a real (reliability) problem that should be fixed.

If the test isn't failing regularly, and the failure can't be reproduced manually, I would simply remove the "blocking" label on this issue and leave it open for a while, and see if the test fails again (at which point the new failures would be added to this issue).

[tommcdon]

However, it's not clear to me that disabling for PGO makes sense; I'm not convinced we know the problem is due to PGO stress, or that it is due to an incompatibility, as opposed to a real (reliability) problem that should be fixed.

@BruceForstall I'm moving this to the codegen area path as the diagnostics team does not plan on investigating unless this issue reproduces outside of the PGO Stress pipeline. Feel free to close/disable the test in PGO Stress.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.