Builds legs getting abandoned due to agent notification issues

[jaredpar]

Full message

##[error]The request: 5641504 was abandoned due to an infrastructure failure. Notification of assignment to an agent was never received.

Runfo Tracking Issue: Runtime jobs being abandoned due to infra

Definition	Build	Kind	Job Name
runtime	1072738	PR 50450	CoreCLR Product Build windows arm checked
runtime	1072738	PR 50450	Libraries Build windows arm64 Release
runtime	1072738	PR 50450	Build windows x64 Release SingleFile
runtime	1072738	PR 50450	CoreCLR Product Build windows arm64 checked
runtime	1072738	PR 50450	CoreCLR Product Build windows x64 checked
runtime	1072738	PR 50450	CoreCLR Product Build windows arm64 release
runtime	1072738	PR 50450	Libraries Build windows x64 Debug
runtime	1072738	PR 50450	Libraries Build windows arm Release
runtime	1072738	PR 50450	CoreCLR Product Build windows x64 release
runtime	1072738	PR 50450	CoreCLR Product Build windows x86 release
runtime	1072738	PR 50450	Libraries Build windows x86 Release
runtime	1072738	PR 50450	CoreCLR Product Build windows x64 release PGO
runtime	1072738	PR 50450	CoreCLR Product Build windows x86 checked
runtime	1072738	PR 50450	CoreCLR Product Build windows arm release
runtime	1072731	PR 49906	Mono Product Build windows x64 debug
runtime	1072731	PR 49906	CoreCLR Product Build windows arm checked
runtime	1072731	PR 49906	CoreCLR Product Build windows x86 checked
runtime	1072731	PR 49906	Libraries Build windows allConfigurations x64 Debug
runtime	1072731	PR 49906	CoreCLR Product Build windows x64 release PGO
runtime	1072731	PR 49906	Libraries Build windows x86 Release
runtime	1072731	PR 49906	CoreCLR Product Build windows x86 release
runtime	1072731	PR 49906	Libraries Build windows x86 Debug
runtime	1072731	PR 49906	CoreCLR Product Build windows x64 release
runtime	1072731	PR 49906	CoreCLR Product Build windows arm release
runtime	1072731	PR 49906	Libraries Build windows net48 x86 Release
runtime	1072731	PR 49906	Mono Product Build windows x64 release
runtime	1072731	PR 49906	Libraries Build windows arm Release
runtime	1072731	PR 49906	Libraries Build windows x64 Debug
runtime	1072731	PR 49906	Mono Product Build windows x86 release
runtime	1072731	PR 49906	CoreCLR Product Build windows arm64 release
runtime	1072731	PR 49906	CoreCLR Product Build windows x64 checked
runtime	1072731	PR 49906	CoreCLR Product Build windows arm64 checked
runtime	1072731	PR 49906	Mono Product Build windows x86 debug
runtime	1072731	PR 49906	Libraries Build windows arm64 Release
runtime	1072731	PR 49906	Build windows x64 Release SingleFile
runtime	1072700	PR 50622	Mono Product Build windows x64 debug
runtime	1072700	PR 50622	Libraries Build windows arm64 Release
runtime	1072700	PR 50622	Libraries Build windows allConfigurations x64 Debug
runtime	1072700	PR 50622	CoreCLR Product Build windows x64 release PGO
runtime	1072700	PR 50622	Libraries Build windows x86 Release
runtime	1072700	PR 50622	CoreCLR Product Build windows x86 release
runtime	1072700	PR 50622	Libraries Build windows x86 Debug
runtime	1072700	PR 50622	CoreCLR Product Build windows x64 release
runtime	1072700	PR 50622	Libraries Build windows net48 x86 Release
runtime	1072700	PR 50622	Mono Product Build windows x64 release
runtime	1072700	PR 50622	Libraries Build windows arm Release
runtime	1072700	PR 50622	Libraries Build windows x64 Debug
runtime	1072700	PR 50622	Mono Product Build windows x86 release
runtime	1072700	PR 50622	CoreCLR Product Build windows arm64 release
runtime	1072700	PR 50622	Mono Product Build windows x86 debug
runtime	1072700	PR 50622	CoreCLR Product Build windows arm release
runtime	1072700	PR 50622	Build windows x64 Release SingleFile
runtime	1072687	PR 50732	Libraries Build windows arm64 Release
runtime	1072687	PR 50732	Build windows x64 Release SingleFile
runtime	1072687	PR 50732	CoreCLR Product Build windows x86 checked
runtime	1072687	PR 50732	Libraries Build windows allConfigurations x64 Debug
runtime	1072687	PR 50732	CoreCLR Product Build windows x64 release PGO
runtime	1072687	PR 50732	Libraries Build windows x86 Release
runtime	1072687	PR 50732	CoreCLR Product Build windows x86 release
runtime	1072687	PR 50732	Libraries Build windows x86 Debug
runtime	1072687	PR 50732	CoreCLR Product Build windows x64 release
runtime	1072687	PR 50732	CoreCLR Product Build windows arm release
runtime	1072687	PR 50732	Mono Product Build windows x64 debug
runtime	1072687	PR 50732	Mono crossaot Product Build windows x64 release
runtime	1072687	PR 50732	CoreCLR Product Build windows arm checked
runtime	1072687	PR 50732	Libraries Build windows net48 x86 Release
runtime	1072687	PR 50732	Mono Product Build windows x64 release
runtime	1072687	PR 50732	Libraries Build windows arm Release
runtime	1072687	PR 50732	Libraries Build windows x64 Debug
runtime	1072687	PR 50732	Mono Product Build windows x86 release
runtime	1072687	PR 50732	CoreCLR Product Build windows arm64 release
runtime	1072687	PR 50732	CoreCLR Product Build windows x64 checked
runtime	1072687	PR 50732	CoreCLR Product Build windows arm64 checked
runtime	1072687	PR 50732	Mono Product Build windows x86 debug
runtime	1072679	PR 50644	CoreCLR Product Build windows arm64 checked
runtime	1072679	PR 50644	CoreCLR Product Build windows x64 checked
runtime	1072679	PR 50644	CoreCLR Product Build windows arm64 release
runtime	1072679	PR 50644	Libraries Build windows x64 Debug
runtime	1072679	PR 50644	Libraries Build windows arm Release
runtime	1072679	PR 50644	CoreCLR Product Build windows arm checked
runtime	1072679	PR 50644	CoreCLR Product Build windows arm release
runtime	1072679	PR 50644	CoreCLR Product Build windows x64 release
runtime	1072679	PR 50644	CoreCLR Product Build windows x86 release
runtime	1072679	PR 50644	Libraries Build windows x86 Release
runtime	1072679	PR 50644	CoreCLR Product Build windows x64 release PGO
runtime	1072679	PR 50644	CoreCLR Product Build windows x86 checked
runtime	1072679	PR 50644	Build windows x64 Release SingleFile
runtime	1072679	PR 50644	Libraries Build windows arm64 Release
runtime	1072670	PR 50612	Mono Product Build windows x64 debug
runtime	1072670	PR 50612	CoreCLR Product Build windows arm checked
runtime	1072670	PR 50612	CoreCLR Product Build windows x86 checked
runtime	1072670	PR 50612	Libraries Build windows allConfigurations x64 Debug
runtime	1072670	PR 50612	CoreCLR Product Build windows x64 release PGO
runtime	1072670	PR 50612	Libraries Build windows x86 Release
runtime	1072670	PR 50612	CoreCLR Product Build windows x86 release
runtime	1072670	PR 50612	Libraries Build windows x86 Debug
runtime	1072670	PR 50612	CoreCLR Product Build windows x64 release
runtime	1072670	PR 50612	CoreCLR Product Build windows arm release
runtime	1072670	PR 50612	Mono crossaot Product Build windows x64 release
runtime	1072670	PR 50612	Libraries Build windows arm64 Release

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
6	6	6

[ilyas1974]

Using runfo, in the last 200 builds, I'm seeing 2 instances of this - from April 14 and April17.

C:\Users\ilyas>runfo search-timeline -d runtime -c 200 -pr -v "abandoned due to an infrastructure failure"
https://dev.azure.com/dnceng/public/_build/results?buildId=606419
https://dev.azure.com/dnceng/public/_build/results?buildId=600517

Evaluated 200 builds
Impacted 2 builds
Impacted 2 jobs

Unfortunately, the logs from these builds are gone and there is nothing to investigate. Do you have additional examples or should we consider this issue closed (until we see new instances)?

/cc @ViktorHofer

[ViktorHofer]

Using runfo, in the last 200 builds, I'm seeing 2 instances of this - from April 14 and April17.

Are you sure about that? Because the top post says: > Most recent failure 5/7/2020 1:12:00 AM

[jaredpar]

@ilyas1974 try taking out the -d runtime portion of the query. This particular failure is happening across a broader range of build definitions.

[ilyas1974]

Similar results -

C:\Users\ilyas>runfo search-timeline -c 200 -pr -v "abandoned due to an infrastructure failure"

Evaluated 200 builds
Impacted 0 bulids
Impacted 0 jobs

C:\Users\ilyas>runfo search-timeline -c 200 -project internal -v "abandoned due to an infrastructure failure"

Evaluated 200 builds
Impacted 0 bulids
Impacted 0 jobs

Do you have any additional suggestions?

[jaredpar]

Something isn't right here. I'm taking a look.

[jaredpar]

Okay figured this out. There are two sources of confusion here:

1. Searching back 200 builds for the entire project isn't far enough. This problem doesn't hit often enough to show up in that data. You either need to go back much further or restrict to a particular definition. My recommendation is to use -d 104 (machine learning) as that has the most recent attempts
2. The machine learning team was pretty aggressively retrying their failed jobs that hit this error. My tooling which auto-triages builds does so immediately upon build completion and hence triaged builds before they could retry. By the time @ilyas1974 ran his query though the build had been retried and the failure was gone from the latest attempt.

To work around the second issue I added a new argument to search-timeline and timeline: -attempt. This lets you specify the build attempt that you want to get timeline information for. The first attempt is always labeled as 1 in AzDO. Once you have an updated runfo (at least 0.5.2) you can do the following search.

P:\devops-util\runfo>runfo search-timeline -c 50 -d 104 -pr  -v "abandoned due to an infrastructure failure" -a 1
https://dev.azure.com/dnceng/public/_build/results?buildId=634214
https://dev.azure.com/dnceng/public/_build/results?buildId=634214
https://dev.azure.com/dnceng/public/_build/results?buildId=634291
https://dev.azure.com/dnceng/public/_build/results?buildId=634291
https://dev.azure.com/dnceng/public/_build/results?buildId=634291
https://dev.azure.com/dnceng/public/_build/results?buildId=634291
https://dev.azure.com/dnceng/public/_build/results?buildId=634291

[ilyas1974]

Thank you @jaredpar. I have created https://github.com/dotnet/core-eng/issues/9801 for our FR team to take a look and see what we can determine about these failures.

[jaredpar]

Did some more searching and I found pretty much every variant of this. Essentially builds that hit this on first attempt, latest attempt and for some attempt in the middle. That's not surprising on one hand but it does really mean that probably need expanded search capabilities for certain issues.

Have a tweak available locally where I can say -attempt -1 which means "search all attempts". Need to work on the presentation though. Should have some builds out tonight / tomorrow.