The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1223773	dotnet/runtime	Publish	Log	dotnet/runtime#122285
1223796	dotnet/runtime	Publish native test components	Log	dotnet/runtime#122286
1223608	dotnet/runtime	Send to Helix	Log
1223611	dotnet/runtime	Build product	Log
2855941	dotnet-dotnet	Run Tests	Log
2855943	dotnet-dotnet	Run Tests	Log
2855958	dotnet-dotnet-monitor	Publish Using Darc	Log
2854091	dotnet-dotnet	Build	Log
2856007	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
2855859	dotnet-arcade	Publish packages, blobs and symbols	Log
2855850	dotnet-arcade	Publish packages, blobs and symbols	Log
2854916	dotnet-roslyn	Publish Using Darc	Log
2855807	dotnet-source-indexer	🟣Build source index	Log
2855762	dotnet-runtime	Send tests to Helix (Unix)	Log
1223058	dotnet/runtime	Send tests to Helix (Windows)	Log
1223054	dotnet/runtime	Send to Helix	Log
2855539	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1222932	dotnet/runtime	Publish Build Assets	Log	dotnet/runtime#122224
1222988	dotnet/runtime	AOT compile CoreCLR tests	Log
1222967	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#81387
2855413	dotnet-runtime	Send tests to Helix (Unix)	Log
2855408	dotnet-arcade	Publish packages, blobs and symbols	Log
1222853	dotnet/runtime	Send to Helix	Log
1222788	dotnet/runtime	Zip	Log	dotnet/runtime#122162
1222804	dotnet/runtime	AOT compile CoreCLR tests	Log
2855284	dotnet-dotnet	Build (centOSStreamContainer)	Log
2855183	dotnet-runtime	Send to Helix	Log
1222664	dotnet/runtime	Publish Build Assets	Log
2855074	dotnet-docker-tools	🟪 Generate EOL Data for all images in msdotnetcontainersstaging.azurecr.io	Log
1222616	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121853
1222586	dotnet/runtime	Send tests to Helix (Windows)	Log
1222592	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122257
1222598	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121694
1222595	dotnet/runtime	Send to Helix	Log	dotnet/runtime#121656
1222459	dotnet/runtime	Run Native AOT Test App Tests	Log	dotnet/runtime#121614
2855036	dotnet-dotnet-monitor	Build	Log
1222517	dotnet/runtime	Send to Helix	Log
1222350	dotnet/runtime	Run Trimming Tests	Log	dotnet/runtime#122249
1222414	dotnet/runtime	Send to Helix	Log
1222430	dotnet/runtime	Send to Helix	Log	dotnet/runtime#119432
1222374	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122012
1222401	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122085
2854861	dotnet-dotnet	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2854910	dotnet-dotnet	Build (centOSStreamContainer)	Log
1222300	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122227
1222368	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122247
1222219	dotnet/runtime	Send to Helix	Log
1222410	dotnet/aspnetcore	Run build.cmd helix target	Log
2854934	dotnet-aspnetcore	Run build.cmd helix target	Log
1222411	dotnet/performance	Send job to Helix (Unix)	Log
1222383	dotnet/roslyn	Component Detection (auto-injected by policy)	Log
1222399	dotnet/roslyn	Test	Log	dotnet/roslyn#81567
1222362	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122250
1220952	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122202
2854925	dotnet-performance	Send job to Helix (Unix)	Log
2854917	dotnet-roslyn	Test	Log
1222387	dotnet/roslyn	Test	Log	dotnet/roslyn#81562
1222266	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122245
1222331	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122195
1222164	dotnet/runtime	Send to Helix	Log
1222165	dotnet/runtime	Send to Helix	Log
1222239	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1222312	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#122248
1222272	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122243
1222217	dotnet/runtime	Send to Helix	Log
1222186	dotnet/runtime	Send to Helix (Unix)	Log	dotnet/runtime#122239
1222213	dotnet/runtime	Send tests to Helix (Windows)	Log
1222206	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122167
1222071	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1222142	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#122089
1222255	dotnet/performance	Send job to Helix (Unix)	Log	dotnet/performance#5063
1222166	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122242
2854836	dotnet-runtime	Send job to Helix (Unix)	Log
2854793	dotnet-dotnet-monitor	Build	Log
1222097	dotnet/runtime	Send tests to Helix (Unix)	Log
2854690	dotnet-installer	Build	Log	#55871
2854686	dotnet-dotnet-monitor	Build	Log
2854666	dotnet-sdk	Run Tests in Helix	Log	#55868
2854667	dotnet-installer	Checkout dotnet-installer@why-no-container to s	Log
2854652	dotnet-sdk	Checkout dotnet-sdk@dev/mthalman/8.0-sb-container to s	Log
2854624	dotnet-dotnet	🛡 Generate SBoM Manifest	Log
2854643	dotnet-installer	Checkout dotnet-installer@why-no-container to s	Log
2854637	dotnet-installer	🛡 Guardian: ARMory	Log
2854635	dotnet-installer	Secure Supply Chain Analysis (auto-injected by policy)	Log
2854628	dotnet-installer	Build (ubuntu2204DebPkg)	Log
2854626	dotnet-installer	Generate delegation SAS Token for dotnetbuilds/internal	Log
2854528	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
2854427	dotnet-maui	Run Integration Tests - Build macOS	Log
1221563	dotnet/xharness	Run tests in Helix	Log
1221490	dotnet/runtime	Send tests to Helix (Unix)	Log
1221435	dotnet/runtime	Zip Build Assets	Log	dotnet/runtime#121305
1221425	dotnet/runtime	Send to Helix	Log
1221103	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122167
1221426	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122036
1221415	dotnet/runtime	Send to Helix	Log	dotnet/runtime#122206
1221481	dotnet/runtime	Send job to Helix (Unix)	Log
1221457	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51826
1221423	dotnet/aspnetcore	Run build.cmd helix target	Log
2854274	dotnet-dotnet	Build	Log
2854265	dotnet-dotnet-monitor	Build	Log
Displaying 100 of 1823 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
14	387	1823

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)

[CarnaViire]

For the "known issues" folks: the timeout here is reported twice, making BA red despite the tracking issue. E.g.:

1️⃣ dotnet/runtime#117255

2️⃣ dotnet/runtime#117800

cc @AlitzelMendez