Workload errors with nightly SDK

[sebastienros]

Issue only on Windows. SDK installed in a dedicated folder using dotnet-install.ps1

dotnet publish BenchmarkServer.csproj -c Release -o \14mt5zti.uek\src\BenchmarksApps\SignalR\published /p:MicrosoftNETCoreAppPackageVersion=9.0.0-rc.1.24403.1 /p:MicrosoftAspNetCoreAppPackageVersion=9.0.0-rc.1.24405.8 /p:GenerateErrorForMissingTargetingPacks=false /p:RestoreNoCache=true /p:MicrosoftWindowsDesktopAppPackageVersion=9.0.0-rc.1.24404.1 /p:MicrosoftNETPlatformLibrary=Microsoft.NETCore.App --framework net9.0 --self-contained -r win-x64 
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242: SDK Resolver Failure: "The SDK resolver "Microsoft.DotNet.MSBuildWorkloadSdkResolver" failed while attempting to resolve the SDK "Microsoft.NET.SDK.WorkloadAutoImportPropsLocator". Exception: "Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadManifestCompositionException: Workload manifest dependency 'Microsoft.NET.Workload.Emscripten.Current' version '8.0.7' is lower than version '9.0.0-rc.1.24379.5' required by manifest 'microsoft.net.workload.mono.toolchain.current' [\fbrsqhju.aat\sdk-manifests\9.0.100-rc.1\microsoft.net.workload.mono.toolchain.current\9.0.0-rc.1.24403.1\WorkloadManifest.json] [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.ComposeWorkloadManifests() [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.InitializeManifests() [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadManifestReader.WorkloadResolver.GetInstalledWorkloadPacksOfKind(WorkloadPackKind kind)+MoveNext() [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, IWorkloadManifestProvider manifestProvider, IWorkloadResolver workloadResolver) [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.CachingWorkloadResolver.Resolve(String sdkReferenceName, String dotnetRootPath, String sdkVersion, String userProfileDir, String globalJsonPath) [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.NET.Sdk.WorkloadMSBuildSdkResolver.WorkloadSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory) [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]
\sdk\9.0.100-rc.1.24406.1\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.ImportWorkloads.props(14,38): error MSB4242:    at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.TryResolveSdkUsingSpecifiedResolvers(IReadOnlyList`1 resolvers, Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio, SdkResult& sdkResult, IEnumerable`1& errors, IEnumerable`1& warnings)"" [\14mt5zti.uek\src\BenchmarksApps\SignalR\BenchmarkServer.csproj]

[sebastienros]

Clearing the SDKs folder which contained other versions seems to fix the issue. But it still looks like a bug that older version can trigger this issue.

[sebastienros]

@joperezr

[joperezr]

cc @marcpopMSFT @baronfel as FYI.

[marcpopMSFT]

@sebastienros mind sharing the folder structure you have in sdk-manifests? The logic we have should prefer a matching band workload to a downlevel but it will prefer a downlevel if it thinks the matching band isn't actually matching.

[sebastienros]

Current state (typical day). Next time it happens I'll copy the whole folder it that helps

08/13/2024  03:25 AM    <DIR>          7.0.100
08/13/2024  12:30 AM    <DIR>          8.0.100
08/13/2024  12:30 AM    <DIR>          9.0.100-preview.6
08/13/2024  12:30 AM    <DIR>          9.0.100-rc.1

Also, sdk folder:

08/13/2024  03:26 AM    <DIR>          7.0.410
08/13/2024  03:27 AM    <DIR>          8.0.303
08/13/2024  12:30 AM    <DIR>          9.0.100-rc.1.24412.21
08/13/2024  02:06 AM    <DIR>          9.0.100-rc.1.24413.1

[sebastienros]

It happened again today and I copied all the files to create a local repro. Ping on Teams if you want it.

[Forgind]

sebastienros was able to share a local repro. After investigating for a while, I came up with a hypothesis as to what might have gone wrong that led to the errors he mentioned. After looking further, I was able to validate my hypothesis.

The problem at its core is actually pretty simple, just masked very well. We do garbage collection for workloads. We also permit pinning specific versions of workloads via workload versions (workload sets) and the install state file. (Rollback files don't enforce anything; it's just a way to specify which you want.)

We have logic to look for an install state file, and we have logic to look for workload versions. Workload versions can be (and often are) installed in random file locations, as in this example where the SDK itself was installed to a random folder. We assume that if anything is pinned, it must have been pinned via global.json, and whenever we update to a workload version from a global.json, we list it in a global file so we can find it from other SDK installs and not uninstall needed workloads.

In this case, however, it was pinned via a baseline workload set. We had no way of tracking those in the original work to support garbage collecting workload sets in #42475. On the other hand, we no longer ship baseline workload sets, so this problem was unintentionally resolved in that way.