Helix Android queues repeatedly assign unhealthy devices to MAUI R2R work items

[simonrozsival]

Summary

The dotnet/android public PR pipeline is seeing frequent infrastructure failures in MAUI ReadyToRun smoke tests submitted to the Helix Android device queues.

The submissions succeed and the work items are scheduled, but device setup fails before the APK is launched. This is affecting unrelated non-fork PRs and same-commit reruns pass when assigned to healthy devices.

Full dotnet/android tracking issue: dotnet/android#12658

Impact

Sample window: the latest 100 completed dotnet-android builds from 2026-08-31 through 2026-09-03.

Of 92 builds containing the MAUI R2R Helix task, 72 passed, 19 failed, and 1 was skipped. Excluding eight separately understood fork-token failures, 11 direct-PR builds contained 12 failed Helix work items:

• 8 Android package-manager/UID exhaustion failures
• 3 ADB authorization timeouts
• 1 work item with no Android device detected

All 12 failed before APK launch and therefore do not demonstrate a MAUI or ReadyToRun product regression.

Affected work items

Build / PR	Configuration	Helix queue	Helix machine	Android device	Cause
1579911 / dotnet/android#12650	android-arm/Default	windows.11.amd64.android-armel.open	DNCENGWIN-141	27091JEGR21946 (unauthorized)	ADB authorization timeout
1579675 / dotnet/android#12644	android-arm/Full R2R	windows.11.amd64.android-armel.open	DNCENGWIN-141	27091JEGR21946 (unauthorized)	ADB authorization timeout
1578760 / dotnet/android#12597	android-arm64/Full R2R	windows.11.amd64.android.open	DNCENGWIN-135	26081FDF60041K (Pixel 6)	Package manager / UID exhaustion
1578755 / dotnet/android#12641	android-arm64/Default	windows.11.amd64.android.open	DNCENGWIN-036	01XAYV0UJJ (Pixel 3a)	Package manager / UID exhaustion
1578133 / dotnet/android#12631	android-arm64/Default	windows.11.amd64.android.open	DNCENGWIN-006	9CJAY1PSW4 (Pixel 3a)	Package manager / UID exhaustion
1578128 / dotnet/android#12638	android-arm64/Full R2R	windows.11.amd64.android.open	DNCENGWIN-135	26081FDF60041K (Pixel 6)	Package manager / UID exhaustion
1577702 / dotnet/android#12629	android-arm64/Default	windows.11.amd64.android.open	DNCENGWIN-137	26091FDF6001Y5 (Pixel 6)	Package manager / UID exhaustion
1577702 / dotnet/android#12629	android-arm/Full R2R	windows.11.amd64.android-armel.open	DNCENGWIN-141	27091JEGR21946 (unauthorized)	ADB authorization timeout
1576771 / dotnet/android#12585	android-arm64/Default	windows.11.amd64.android.open	DNCENGWIN-036	01XAYV0UJJ (Pixel 3a)	Package manager / UID exhaustion
1576613 / dotnet/android#12614	android-arm64/Full R2R	windows.11.amd64.android.open	DNCENGWIN-034	None detected	No-device timeout
1576382 / dotnet/android#12541	android-arm64/Default	windows.11.amd64.android.open	DNCENGWIN-020	9CBAY1NP1W (Pixel 3a)	Package manager / UID exhaustion
1575529 / dotnet/android#12604	android-arm64/Full R2R	windows.11.amd64.android.open	DNCENGWIN-010	94SAY0SM1M (Pixel 3a)	Package manager / UID exhaustion

Failure signatures

The ARM32 failures are concentrated on one host/device pair:

Helix machine: DNCENGWIN-141
Android device: 27091JEGR21946
Queue: windows.11.amd64.android-armel.open

List of devices attached
27091JEGR21946         unauthorized transport_id:1
adb.exe wait-for-device
END OF WORK ITEM LOG: Command timed out, and was killed

The ARM64 package-manager failures occur across seven Helix machines and six Android devices:

adb.exe uninstall com.xamarin.mauir2r.arm64.fullr2r
Failure [DELETE_FAILED_INTERNAL_ERROR]

adb.exe install -r MauiR2R-android-arm64-fullr2r-Signed.apk
Failure [INSTALL_FAILED_INSUFFICIENT_STORAGE: Scanning Failed.: Package ... could not be assigned a valid UID]

One work item on DNCENGWIN-034 had no device listed at all before adb wait-for-device timed out.

Same-commit rerun evidence

• [typemap] Diagnose unsupported export signatures android#12597: failed in build 1578760; MAUI task passed in 1579125
• [r8-obfuscation] Add PE metadata rebuild substrate android#12629: failed in build 1577702; passed in 1578074
• [ci] Fix MAUI R2R Helix submissions for fork PRs android#12641: failed in build 1578755; MAUI task passed in 1579094 and 1579507

Action requested for Engineering Services / Helix

• Quarantine or re-authorize Android device 27091JEGR21946 attached to DNCENGWIN-141.
• Investigate and clean/reprovision the affected ARM64 devices' package-manager state.
• Investigate why DNCENGWIN-034 accepted work while no Android device was visible.
• Consider queue health checks that reject hosts with missing/unauthorized devices or exhausted Android package-manager state before assigning work.

[akoeplinger]

For INSTALL_FAILED_INSUFFICIENT_STORAGE we've seen this in the past and a device reboot usually fixes it, xharness is supposed to do that: https://github.com/dotnet/xharness/blob/1fc9369e393e63fa8760afb30d2fcc596bb799bf/src/Microsoft.DotNet.XHarness.Android/AdbRunner.cs#L660-L674
Can you check whether the reboot happened in that case?