Inconsistent NuGet push order in .NET release publishing

[filipnavara]

Describe the bug

When new .NET service release is pushed a lot of packages are pushed to the NuGet feed. Among those packages are the workload manifests and the workload packages referenced from those manifests. There is currently no enforced order in which these assets are uploaded which can cause the manifest feed to refer to non-existent packages. This will cause temporary failures on dotnet workload install command.

Example:

Run dotnet workload install macos ios
  dotnet workload install macos ios
  shell: /bin/bash -e {0}
  env:
    DOTNET_ROOT: /Users/teamcity/.dotnet

Skip NuGet package signing validation. NuGet signing validation is not available on Linux or macOS https://aka.ms/workloadskippackagevalidation .
Updated advertising manifest microsoft.net.sdk.tvos.
Updated advertising manifest microsoft.net.sdk.maui.
Updated advertising manifest microsoft.net.sdk.maccatalyst.
Updated advertising manifest microsoft.net.sdk.ios.
Updated advertising manifest microsoft.net.workload.emscripten.
Updated advertising manifest microsoft.net.sdk.android.
Updated advertising manifest microsoft.net.sdk.macos.
Updated advertising manifest microsoft.net.workload.mono.toolchain.
Installing pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2...
Installing pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2...
Writing workload pack installation record for Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2...
Installing pack Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm version 6.0.2...
Workload installation failed. Rolling back installed packs...
Rolling back pack Microsoft.NET.Runtime.MonoAOTCompiler.Task installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoAOTCompiler.Task version 6.0.2…
Rolling back pack Microsoft.NET.Runtime.MonoTargets.Sdk installation...
Uninstalling workload pack Microsoft.NET.Runtime.MonoTargets.Sdk version 6.0.2…
Rolling back pack Microsoft.NETCore.App.Runtime.AOT.Cross.ios-arm installation...
Workload installation failed: microsoft.netcore.app.runtime.aot.osx-x64.cross.ios-arm::6.0.2 is not found in NuGet feeds https://nuget.pkg.github.com/emclient/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json;https://api.nuget.org/v3/index.json;https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json".
Error: Process completed with exit code 1.

[baronfel]

Hi @filipnavara, thanks for raising this issue.

We don't see this as a workloads-specific issue, instead we see this as an issue with the way dotnet as an organization publishes packages on NuGet. We push a huge number of packages with each servicing release between workloads, various .NET Runtimes, and library releases, and several of these have similar timing issues as you describe here. Even if we had some ordering where we uploaded all workload packs prior to uploading the manifests, there's nothing enforcing on NuGet's end that indexing/CDN distribution/etc. keep that same ordering. The best advice I can give you is to delay a bit of time and try again in the general case.

[filipnavara]

The problem for individual NuGets is different since we are in control of the versions (which we update through explicit Dependabot PR, dotnet/arcade or other mechanism). However, with workloads there is no control unless we enforce a usage of a full custom manifest. So the moment the new manifest is pushed to NuGet it breaks our CI pipelines. During the publishing of .NET 6.0.2 the breakage lasted approximately 2 hours. I am rising the issue in an attempt to find a way how to minimize the disruption in future. While I can easily identify what is happening and what was the cause of the errors others may not.

[akoeplinger]

I agree that workloads are a special case here because of the way advertising manifests are automatically updated during restore.

Maybe we can skip a manifest during that automatic update until it is at least X hours old?

[filipnavara]

FWIW we hit it again when 6.0.200 SDK was being published.

We install the SDK as part of the CI pipeline. So in addition to the order of NuGets being pushed the SDK acquisition process also gets the newer SDK first [before the NuGets are uploaded]. That also doesn't seem quite right.

[mattjohnsonpint]

Hitting this presently with the current rollout for the ios workload.

sudo dotnet workload install ios

...
Workload(s) 'ios' are already installed.
Skipping NuGet package signature verification.
Installing workload manifest microsoft.net.sdk.ios version 16.2.1024…
Installing workload manifest microsoft.net.sdk.maccatalyst version 16.2.1024…
Installing workload manifest microsoft.net.sdk.macos version 13.1.1024…
Installing workload manifest microsoft.net.sdk.tvos version 16.1.1521…
Installing pack Microsoft.iOS.Sdk version 16.2.1024...
Writing workload pack installation record for Microsoft.iOS.Sdk.net7 version 16.2.1024...
Installing pack Microsoft.iOS.Sdk version 16.2.19...
Workload installation failed. Rolling back installed packs...
Rolling back pack Microsoft.iOS.Sdk installation...
Rolling back pack Microsoft.iOS.Sdk installation...
Uninstalling workload pack Microsoft.iOS.Sdk.net7 version 16.2.1024…
Workload installation failed: microsoft.ios.sdk::16.2.19 is not found in NuGet feeds https://api.nuget.org/v3/index.json".

On closer inspection, it appears that Microsoft.NET.Sdk.iOS.Manifest-7.0.100 version 16.2.1024 contains:

"packs": {
    "Microsoft.iOS.Sdk.net7": {
        "kind": "sdk",
        "version": "16.2.1024",
        "alias-to": {
            "any": "Microsoft.iOS.Sdk"
        }
    },
    "Microsoft.iOS.Sdk.net6": {
        "kind": "sdk",
        "version": "16.2.19",
        "alias-to": {
            "any": "Microsoft.iOS.Sdk"
        }
    },

And it appears that Microsoft.iOS.Sdk version 16.2.1024 has been published, but version 16.2.19 has not - yet.

I suggest that there be some validation added such that a manifest can't be published until all of its dependent packages have been published first.

[mattjohnsonpint]

Also note that this breaks our CI/CD builds, since the current workloads are installed with each build. We can't build until the workload install succeeds.

[mattjohnsonpint]

Note also, even after it appeared on Nuget a while later, it's still not available until fully indexed.

Thus, the validation would need to really make sure it was available on nuget, not just sent there.

[mattjohnsonpint]

FYI, it's working again now that the indexing has completed. Hope the validation can be added so we don't encounter it again next time.