Silent workload updates break builds

[svaldetero-envoc]

Description

Another update Tuesday and another broken Azure Pipelines CI build. One build works and then an hour late the build breaks.

I have a step to use a dotnet version 8.0.x. It downloads 8.0.203. Specifically: https://download.visualstudio.microsoft.com/download/pr/87f31249-1415-4edb-87d5-7f16e63414e9/a1ad58c05a131c618ad458a1564784e4/dotnet-sdk-8.0.203-osx-x64.tar.gz.

Next step is to:
dotnet workload install maui-ios maui-android --source https://api.nuget.org/v3/index.json.

The first time it ran, it installed successfully:

Installing workload manifest microsoft.net.sdk.android version 34.0.85...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8043...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8043...
Installing workload manifest microsoft.net.sdk.maui version 8.0.7...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12...
Installing pack Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7...
Writing workload pack installation record for Microsoft.AspNetCore.Components.WebView.Maui version 8.0.7...
Installing pack Microsoft.Maui.Sdk version 8.0.7...

The second time it ran (an hour later), it failed:

Installing workload manifest microsoft.net.sdk.android version 34.0.95...
Installing workload manifest microsoft.net.sdk.ios version 17.2.8043...
Installing workload manifest microsoft.net.sdk.maccatalyst version 17.2.8043...
Installing workload manifest microsoft.net.sdk.macos version 14.2.8043...
Installing workload manifest microsoft.net.sdk.maui version 8.0.7...
Installing workload manifest microsoft.net.sdk.tvos version 17.2.8043...
Installing workload manifest microsoft.net.workload.mono.toolchain.current version 8.0.4...
Installing workload manifest microsoft.net.sdk.aspire version 8.0.0-preview.5.24201.12...
Workload installation failed. Rolling back installed packs...
Installation rollback failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]
Workload installation failed: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.3' is lower than version '8.0.4' required by manifest 'microsoft.net.workload.mono.toolchain.current' [/Users/runner/hostedtoolcache/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.4/WorkloadManifest.json]

Looking at the nugets:
https://www.nuget.org/packages/Microsoft.NET.Workload.Mono.ToolChain.Current.Manifest-8.0.100/8.0.4 was published between the builds.

HOWEVER, the nuget package that it claims is required:
https://www.nuget.org/packages/Microsoft.NET.Workload.Emscripten.Current.Manifest-8.0.100/8.0.4
DOES NOT EXIST.

I understand these may be separate teams, however these nugets should have been deployed at the same time.

Steps to Reproduce

No response

Link to public reproduction project repository

No response

Version with bug

8.0.7 SR2

Is this a regression from previous behavior?

No, this is something new

Last version that worked well

8.0.7 SR2

Affected platforms

iOS, Android

Affected platform versions

No response

Did you find any workaround?

I changed my workload restore to include :
--from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Relevant log output

No response

[svaldetero-envoc]

Update: After submitting this, the dependent nuget package showed up (~3-4 hours later).

The issue still remains, we need a way to pin a version of the workload (without using the undocumented and hard to find rollback file). Because pinning a dotnet version doesn't stop the problem.

[svaldetero-envoc]

Same type of issue, but different package: dotnet/maui#19934

[Redth]

@svaldetero please try again. It seems there was a delay in publishing of some packages for this update, which should now be available.

[Redth]

Related: #23820

[svaldetero-envoc]

@Redth Yes it works now after I removed the rollback option.

As a developer, how do I prevent this in the future? How can I pin a workload version without having to find a hidden rollback file? In that issue you linked, it doesn't happen every time, but it happens frequently enough. Today it just happened to be ~3 hours before all the nugets were available. That suddenly blocks and backlogs ALL of my builds for close to half the day. Realistically, I can't just wait for the delay to catch up because I don't know how long it will be.

[kgouraw]

I experienced the same with "dotnet workload install maui" in my GitHub workflow today. I changed it to "dotnet workload install maui-mobile" and it works for me for now.

[Ghostbird]

For us, the way we managed to "pin" the MAUI version for Android CI builds is by building in docker, which we need to do anyway for CI. For an example see our repository. Note: We do not use the docker for builds on the developer's own machine, but updating MAUI there takes only a minute when it breaks.
Alas, our iOS builds keep automatically breaking, but that happens from Apple's side too. Their updates often break things too, so in that sense MAUI is not doing any worse.

[MatthewRamsey]

Problem:
We were experiencing similar issues this week which were the result of an issue with the Android sdk version bump. Specifically, Microsoft.Android.Sdk.Windows.Msi.x64 (34.0.95) which caused malformed content in in our Android http responses.

Solution:
As others have mentioned, we found a workaround by pinning a rollback file for the specific version of the workloads we need. The rollback file below reverted the Android dependencies to 34.0.79 which solved our problem.

dotnet workload install maui --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

dotnet workload install android ios --source https://api.nuget.org/v3/index.json --from-rollback-file https://maui.blob.core.windows.net/metadata/rollbacks/8.0.7.json

Suggestion:
It would be great to have a page with a list of available rollback files. We're content to pin a version while new releases come out so we can control stability of our applications.

[marcpopMSFT]

Without functionality in nuget to batch these, the best we have is pushing all of the packs first and then the manifest. Looking forward, we'll have workload versions coming soon that will make this easier as we can ensure all of the packs and manifests are published and then publish a single workload set version package.