The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2707872	dotnet-aspire	Build	Log	#49988
2707842	dotnet-aspire	Run helix tests	Log	#49988
1040001	dotnet/runtime	Send to Helix	Log
2707549	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2707545	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
1039726	dotnet/runtime	Send job to Helix (Unix)	Log
2707566	dotnet-aspnetcore	Component Detection (auto-injected by policy)	Log
1039622	dotnet/runtime	Send to Helix	Log
2707312	dotnet-runtime	Send UTF8Fuzzer to OneFuzz	Log
1039348	dotnet/runtime	Send to Helix	Log
1039298	dotnet/runtime	AOT compile CoreCLR tests	Log
2707219	dotnet-dotnet	Build (centOSStreamContainer)	Log
2707007	dotnet-runtime	Send job to Helix (Windows)	Log
1038977	dotnet/runtime	AOT compile CoreCLR tests	Log
2706892	dotnet-runtime	Send job to Helix (Unix)	Log
1038871	dotnet/runtime	Send to Helix	Log
1038862	dotnet/runtime	AOT compile CoreCLR tests	Log
2706613	dotnet-install-scripts	Run Tests	Log
1038736	dotnet/runtime	Send to Helix	Log
1038725	dotnet/runtime	Send tests to Helix (Windows)	Log
1038719	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#115408
1038726	dotnet/runtime	AOT compile CoreCLR tests	Log
2706523	dotnet-runtime	Send job to Helix (Windows)	Log
2706498	dotnet-runtime	Send job to Helix (Unix)	Log
2706477	dotnet-dotnet	Build (centOSStreamContainer)	Log
1038651	dotnet/roslyn	Run Integration Tests	Log
2706356	dotnet-runtime	Send TypeNameFuzzer to OneFuzz	Log
2706392	dotnet-performance	Send job to Helix (Unix)	Log
2706371	dotnet-runtime	Send job to Helix (Windows)	Log
2706363	dotnet-dotnet	Build (centOSStreamContainer)	Log
1038542	dotnet/runtime	Send to Helix	Log
2706325	dotnet-dotnet	Build (centOSStreamContainer)	Log
1038525	dotnet/runtime	AOT compile CoreCLR tests	Log
2706314	dotnet-dotnet	Build (azurelinuxArm64CrossContainer)	Log
2706262	dotnet-runtime	Send tests to Helix (Unix)	Log
2706281	dotnet-extensions	Publish AOT Test	Log	#49960
2706278	dotnet-runtime	Send job to Helix (Unix)	Log
2706247	dotnet-aspire	Run helix tests	Log	#49958
2706227	dotnet-runtime	Send job to Helix (Windows)	Log
2706146	dotnet-dotnet	Build (centOSStreamContainer)	Log
2706123	dotnet-runtime	Send job to Helix (Unix)	Log
2706087	dotnet-dotnet	Build (centOSStreamContainer)	Log
2706070	dotnet-performance	Send job to Helix (Unix)	Log
2706016	dotnet-runtime	Send job to Helix (Windows)	Log
1038236	dotnet/roslyn	Run Integration Tests	Log
1038152	dotnet/runtime	Send to Helix	Log
2705981	dotnet-runtime	Send job to Helix (Unix)	Log
1038139	dotnet/roslyn	Run Integration Tests	Log
1038079	dotnet/runtime	AOT compile CoreCLR tests	Log
1038073	dotnet/runtime	Send to Helix	Log
1037918	dotnet/roslyn	Run Integration Tests	Log
2705809	dotnet-runtime	Send job to Helix (Unix)	Log
2705784	dotnet-arcade-services	Run E2E tests	Log
2705769	dotnet-runtime	Send job to Helix (Windows)	Log
1037815	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#115426
1037529	dotnet/aspnetcore	Run build.cmd helix target	Log
2705730	dotnet-arcade-services	Auto-baselining mode and SDL pre-checks (1ES PT)	Log
2705706	dotnet-aspire	Run helix tests	Log	#49912
2705691	dotnet-arcade-services	Run E2E tests	Log
2705687	dotnet-runtime	Send job to Helix (Unix)	Log
2705649	dotnet-performance	Send job to Helix (Unix)	Log
2705585	dotnet-runtime	Send job to Helix (Windows)	Log
2705580	dotnet-arcade-services	Synchronize Secrets	Log
2705573	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Source)	Log
1037613	dotnet/runtime	AOT compile CoreCLR tests	Log
2705496	dotnet-runtime	Send job to Helix (Unix)	Log
2704631	dotnet-dotnet	Build (centOSStreamContainer)	Log
2705459	dotnet-aspire	Run helix tests	Log	#49907
2705394	dotnet-runtime	Send job to Helix (Unix)	Log
2705215	dotnet-sdk	Run AoT Tests	Log	#49771
1037417	dotnet/runtime	Send job to Helix (Unix)	Log
2705277	dotnet-roslyn	Test	Log
2705261	dotnet-performance	Send job to Helix (Unix)	Log
1037346	dotnet/aspnetcore	Run build.cmd helix target	Log
1037237	dotnet/runtime	Send to Helix	Log
2705228	dotnet-roslyn	Test	Log
1037339	dotnet/roslyn	Test	Log
1037310	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#61840
1037222	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115395
1037213	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#115345
1037201	dotnet/runtime	Send tests to Helix (Unix)	Log
2705193	dotnet-roslyn	Test	Log
1037253	dotnet/sdk	🟣 Run AoT Tests	Log	dotnet/sdk#48815
1037196	dotnet/runtime	Send tests to Helix (Unix)	Log
2705076	dotnet-sdk	Run FullFramework Tests	Log	#49771
1037299	dotnet/roslyn	Test	Log
2705184	dotnet-roslyn	Test	Log
2705196	dotnet-aspnetcore	Validate Hosted Pool Information (1ES PT)	Log
1037271	dotnet/aspnetcore	Run build.cmd helix target	Log	dotnet/aspnetcore#61813
1037194	dotnet/runtime	Send to Helix	Log
2705071	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2705177	dotnet-runtime	Send job to Helix (Unix)	Log
1037274	dotnet/roslyn	Test	Log
2705155	dotnet-roslyn	Test	Log
2705124	dotnet-runtime	Send job to Helix (Windows)	Log
1037199	dotnet/runtime	Send to Helix	Log
2705153	dotnet-roslyn	Test	Log
1037255	dotnet/roslyn	Test	Log
2705129	dotnet-dotnet	🛡 Guardian: CredScan (Binary)	Log
1037243	dotnet/roslyn	Test	Log
Displaying 100 of 1046 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
12	221	1046

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)