Cannot update workload manifests from authenticated AzDO feeds on macOS

[DamianEdwards]

I'm not able to get any dotnet workload related commands to work when one of the workload manifests has updates on an authenticated AzDO artifacts feed on macOS. I've installed the nuget credential provider and dotnet restore from that feed works just fine.

I've tried relying on on-demand token acquisition (via device flow and passing --interactive), configuring a PAT in the NuGet.config file, passing the source directly, etc. Every time, sudo dotnet workload update fails with the following error message:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.aspire: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.ios: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maccatalyst: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.macos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.maui: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.sdk.tvos: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.current: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.emscripten.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net6: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json..
Failed to update the advertising manifest microsoft.net.workload.mono.toolchain.net7: Unable to load the service index for source https://pkgs.dev.azure.com<feed-name>nuget/v3/index.json.

Output of dotnet --info:

.NET SDK:
 Version:   8.0.100-rc.2.23472.8
 Commit:    e13fd5c19e

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  13.4
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.100-rc.2.23472.8/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      8.0.0-rc.2.23466.4
  Architecture: arm64
  Commit:       287c10d253

.NET SDKs installed:
  6.0.402 [/usr/local/share/dotnet/sdk]
  8.0.100-rc.2.23472.8 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.0-rtm.23468.24 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.10 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.7 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.0-rc.2.23466.4 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  /Users/damian/src/MyProject/global.json

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[marcpopMSFT]

Chet believes the code in question might be here but it is correctly passing the interactive flag through:

sdk/src/Cli/dotnet/commands/dotnet-workload/WorkloadCommandNuGetRestoreActionConfigOptions.cs

Lines 55 to 58 in 6483a69

	return new RestoreActionConfig(DisableParallel: parseResult.GetValue(DisableParallelOption),
	NoCache: parseResult.GetValue(NoCacheOption),
	IgnoreFailedSources: parseResult.GetValue(IgnoreFailedSourcesOption),
	Interactive: parseResult.GetValue(InteractiveRestoreOption));

[DamianEdwards]

I didn't have this issue on Ubuntu FWIW, and others have hit this on macOS too, but I appreciate it might be hard to repro without a workload manifest update ready to go on a private feed.

[rmarinho]

@DamianEdwards did you tried --configfile pointing to the global Nuget.config with the PAT ?

[dsplaisted]

Thanks a bunch to @rmarinho and @mattleibow for helping investigate this.

The problem is caused by running under su/sudo. When running a workload command under sudo, we override the HOME directory to point to a temporary directory. This is so that we don't end up writing files owned by root (for example first run flags or other things stored in the <HOME>\.dotnet folder) which would then not be accessible to the .NET CLI when running under normal permissions.

This means that when run under sudo, the credential provider isn't found by NuGet (since the path to search for it is in the HOME directory). This is also probably why putting the PATs in the NuGet.config file didn't work. (We do try to copy the NuGet.config to the new temporary home directory, but maybe this is failing).

We will discuss to see what we can do to improve this. For now, putting PATs in a NuGet.config file and specifying that file via the --configfile option to workload commands should work as a workaround.

[DamianEdwards]

That will explain why it always runs the first-run experience too I guess.

I'm still unable to get it to work with the --configfile option using the nuget.config in the repo, modified to contain a PAT in plaintext. It does report it's trying to access the correct feed though, so it must be reading at least that from the nuget.config, but not the PAT for some reason.

[DamianEdwards]

That said, I'd expect an authorization failure to be logged exactly as that, but what I'm seeing is "NotFound" results. I'll share the repo details and logs internally to see if that can aid in diagnosing what's going on.

[dsplaisted]

The output is pretty verbose, and it's checking for packages on all the different feeds that are set up. So you get a lot of NotFound log messages before you get the message related to the authorization failure, which looks something like:

Failed to update the advertising manifest microsoft.net.sdk.android: Unable to load the service index for source https://pkgs.dev.azure.com/<feed-name>/nuget/v3/index.json..

We might be able to detect the authorization error specifically and generate a more specific error message.

It might also be a good idea to generate a warning if --interactive is used under sudo, since we now know that doesn't work.

[DamianEdwards]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

[dsplaisted]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

[DamianEdwards]

Any idea why I'm unable to make it work with a PAT and the --configfile option?

Not really. Are you sure the syntax in the config file for the PAT is correct?

Not sure. I'm just working from documentation. The fact it's trying the right feed suggests it is picking up the nuget.config, and there are no parsing errors. I'm not sure what else to try.

[Eilon]

BTW I think I'm hitting the same issue on Windows (@DamianEdwards had it on macOS). I have Azure Cred Provider, but it doesn't seem to get used when I restore workloads. If I pass in --interactive then I get the devicelogin workflow and everything works.