SB Scenario tests are intermittently failing to execute

[ellahathaway]

[FAIL] Microsoft.DotNet.ScenarioTests.SdkTemplateTests.SdkTemplateTests.VerifyMSTestTemplate(language: VB)
System.InvalidOperationException : Failed to execute /__w/1/s/artifacts/obj/extracted-dotnet-sdk/dotnet new globaljson --sdk-version 10.0.100-preview.3.25167.1
    Exit code: 139
    The template "global.json file" was created successfully.

       at Microsoft.DotNet.ScenarioTests.SdkTemplateTests.DotNetSdkHelper.ExecuteCmd(String args, String workingDirectory, Action`1 additionalProcessConfigCallback, Int32 expectedExitCode, Int32 millisecondTimeout) in /_/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/DotNetSdkHelper.cs:line 38
       at Microsoft.DotNet.ScenarioTests.SdkTemplateTests.DotNetSdkHelper.ExecuteNew(String projectType, String projectName, String projectDirectory, String language, String customArgs) in /_/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/DotNetSdkHelper.cs:line 113
       at Microsoft.DotNet.ScenarioTests.SdkTemplateTests.SdkTemplateTest.Execute(DotNetSdkHelper dotNetHelper, String testRoot, String[] frameworks, String PreMadeSolution, Int32 retryCounter) in /_/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/SdkTemplateTest.cs:line 42
       at Microsoft.DotNet.ScenarioTests.SdkTemplateTests.SdkTemplateTests.VerifyMSTestTemplate(DotNetLanguage language) in /_/src/scenario-tests/src/Microsoft.DotNet.ScenarioTests.SdkTemplateTests/SdkTemplateTests.cs:line 99
       at InvokeStub_SdkTemplateTests.VerifyMSTestTemplate(Object, Span`1)
       at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

https://dev.azure.com/dnceng/internal/_build/results?buildId=2665635&view=logs&j=609589e2-4f74-5576-cdb7-914bcaea778b&t=85a3ad82-e833-5143-3e25-ec1affd2c703&l=251

[ellahathaway]

@ViktorHofer - have you seen this failure before? It's affecting PRs in sdk too.

[ViktorHofer]

Yes. The assumption is that this gets fixed with dotnet/runtime#113598

This got introduced with dotnet/sdk#47580 (comment)

[ViktorHofer]

It looks like dotnet/sdk@f981d32 didn't help. I still see the crash in jobs with that commit in HEAD.

[akoeplinger]

@ViktorHofer it makes sense that dotnet/runtime#113598 wouldn't have fixed this since the VMR uses it's own versions of the container images in https://github.com/dotnet/sdk/blob/f255b7461bef7db14620199c3aa00626752edb52/eng/pipelines/templates/variables/vmr-build.yml#L118-L121. We'd need to do the pinning there.

[ViktorHofer]

Makes sense. This is failing in the source-build leg most of the time, which uses this image: https://github.com/dotnet/sdk/blob/f255b7461bef7db14620199c3aa00626752edb52/eng/pipelines/templates/variables/vmr-build.yml#L98-L101

Do we know the list of images that need to be pinned? cc @richlander @MichalStrehovsky

[akoeplinger]

@ViktorHofer hm that's actually an interesting point, clang wasn't updated in the centos-stream9 from what I can see so that one doesn't seem related.