NuGetCommand defaulting to very old version of NuGet on windows-2022

[MattJeanes]

Required Information

Question, Bug, or Feature?
Type: Bug

Enter Task Name: NuGetCommand@2

Environment

• Server: Azure Pipelines
  • Account: exclaimerltd
  • Team Project Id: b677b16d-bceb-42e2-b5b8-c198664edbd8
  • Build Definition Id: 2386
  • Build Id: 219818
• Agent: Hosted Agent / windows-latest

Issue Description

We have been using windows-latest to run our builds which has all been good up until very recently when we understand that the switchover from windows-2019 to windows-2022 has happened, at which point several of our builds started failing due to NuGet issues. We raised this issue (actions/runner-images#5303) but it was closed saying that both 2019 and 2022 have the same NuGet version installed and to raise an issue here instead.

We noted that our previous builds running against windows-2019 were using NuGet 5.8.0 which we can see at the top of the NuGetCommand task:

Caching tool: NuGet 5.8.0 x64
Found tool in cache: NuGet 5.8.0 x64
Resolved from tool cache: 5.8.0
Using version: 5.8.0
Found tool in cache: NuGet 5.8.0 x64
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
C:\Windows\system32\chcp.com 65001
Active code page: 65001
Detected NuGet version 5.8.0.6930 / 5.8.0+830c8be45dbbccd411ecf6080abff0c2c98079cf.830c8be45dbbccd411ecf6080abff0c2c98079cf

Now on windows-2022 we are seeing this:

Caching tool: NuGet 4.1.0 x64
Found tool in cache: NuGet 4.1.0 x64
Resolved from tool cache: 4.1.0
Using version: 4.1.0
Found tool in cache: NuGet 4.1.0 x64
SYSTEMVSSCONNECTION exists true
SYSTEMVSSCONNECTION exists true
C:\Windows\system32\chcp.com 65001
Active code page: 65001
Detected NuGet version 4.1.0.2450 / 4.1.0

The older version of NuGet then causes issues in our pipelines as it does not understand for example .NET 6 properly and fails with errors such as One or more packages are incompatible with .NETFramework,Version=v6.0., Package X is not compatible with net60 (.NETFramework,Version=v6.0). or Package X is not compatible with net472 (.NETFramework,Version=v4.7.2). Package X supports: netstandard2.0 (.NETStandard,Version=v2.0)

We have been working around this issue by using the NuGetToolInstaller@1 task to install a newer version of NuGet and this works but we would not expect certainly a lower version of NuGet being used by default on windows-2022 vs windows-2019.

Task logs

Please contact me if you need logs, I cannot share them publically due to potentially sensitive information in them.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

As described above, we quickly worked out that it seems caused by the switch over to windows-2022 from windows-2019 and that we can workaround the issue using the NuGetToolInstaller@1 task so it's not critical, but we feel that this should not be the expected behaviour especially when NuGet 6.1.0 is installed on the hosted agent by default.

Error logs

##[error]The nuget command failed with exit code(1) and error(Errors in D:\a\1\s\test\<redacted>\<redacted>.csproj
    Project <redacted> is not compatible with net60 (.NETFramework,Version=v6.0). Project <redacted> supports: netstandard2.0 (.NETStandard,Version=v2.0)
    Package Azure.Identity 1.3.0 is not compatible with net60 (.NETFramework,Version=v6.0). Package Azure.Identity 1.3.0 supports: netstandard2.0 (.NETStandard,Version=v2.0)
    One or more projects are incompatible with .NETFramework,Version=v6.0.
    One or more packages are incompatible with .NETFramework,Version=v6.0.

Note for that one our project TFM is correctly set to net6.0 but the old NuGet version is interpreting it incorrectly.

From another failing build:

Package Azure.Data.Tables 12.4.0 is not compatible with net472 (.NETFramework,Version=v4.7.2). Package Azure.Data.Tables 12.4.0 supports: netstandard2.0 (.NETStandard,Version=v2.0)
    One or more packages are incompatible with .NETFramework,Version=v4.7.2.

[maxim-lobanov]

Some additional context:

• All Hosted images have NuGet 6.1.0.106 installed.
• If you add NuGet task to your pipeline and enable debug logging, you will see that task downloads and install NuGet 4.1 on-flight and place it to "C:\hostedtoolcache\windows\NuGet\4.1.0\x64" folder. This folder doesn't exist on clean images.
• I have also done additional investigation and found that the NuGet task uses some hard coded logic to determine the version of NuGet to use:
  • If Visual Studio 2019 is installed, task uses NuGet 5.8.0.
  • Otherwise, fallback to NuGet 4.1.0.

Obviously, Windows Server 2022 doesn't have Visual Studio 2019 installed and have Visual Studio 2022 instead.
Related implementation can be found here.
I think this logic should be extended to support Visual Studio 2022 properly. Otherwise, NuGet tasks won't be usable with Windows Server 2022 image.

As a workaround, the task NuGet Tool Installer can be added before other NuGet tasks. Since all NuGet tasks are from one collection. Next steps will be able to consume NuGet version installed by this task.

[MattJeanes]

Super useful insights @maxim-lobanov thank you! That explains perfectly why the switchover from windows-2019 to windows-2022 broke some of our builds as VS2019 would have no longer been available.

I wonder why it is that we don't use the latest available version of NuGet on the machine by default here, the hosted agents ship with NuGet 6.1.0 at the moment but if you're using these tasks it will never actually get used which is a confusing user experience. I would expect as a user that it would use whatever is installed on the agent by default, and allowing me to override or install a specific version with the installer task.

[maxim-lobanov]

I think initially tasks were implemented the way to not depend on image somehow and avoid breakage when something goes wrong with image. But I am not really sure why task doesn't install "latest" version of NuGet by default. It would allow to avoid such maintenance problems.

Actually, you can drop the task at all and use pre-installed version of NuGet manually.
Replace NuGetCommand task with CommandLine task and write

nuget restore

This way you will invoke pre-installed version of NuGet. AFAIK some users prefer this way if they don't have difficult auth and don't use private NuGet feeds.

[KenticoDevTrev]

Running into this same issue, thanks for identifying it, i was able to swap out the NuGetCommand with NuGetToolInstaller for now, and notified our team. I really hope microsoft fixes this asap.

[fmad]

Same as above but on Windows 2019 (May 10/2022) and VS 2022. Funny enough, only started happening today (22/09/2022) and the last time we've updated our build agent images was on 12/06/2022, about 3.5 months ago...

Microsoft Windows [Version 10.0.17763.2928]
(c) 2018 Microsoft Corporation. All rights reserved.

>"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"
installationVersion: 17.2.32526.322
displayName: Visual Studio Enterprise 2022

>nuget
NuGet Version: 6.2.0.146

The above workaround (adding the NuGetToolInstaller task) gets around the issue but we're now hardcoding a specific version.

[tafs7]

@fmad we started seeing this very same issue today as well (September 22, 2022).

Not sure what changed on the windows-latest build agents from yesterday to today, but they seem to only install Nuget v4.1, instead of Nuget v5.8

I also noticed that the NugetCommand task was upgraded from version 2.208.1 to 2.210.0 between yesterday and today, so perhaps a regression bug?

Yesterday:

Today:

[simon-letch]

Our issue was definitely related to the newer version of the NuGetCommand task.

Failed:

Task         : NuGet
Version      : 2.210.0
##[debug]FORCE_NUGET_4_0_0=undefined
##[debug]isExplicit: 4.1.0
##[debug]explicit? true
##[debug]Agent.Version=2.210.1

Working:

Task         : NuGet
Version      : 2.208.1
##[debug]FORCE_NUGET_4_0_0=undefined
##[debug]isExplicit: 5.8.0
##[debug]explicit? true
##[debug]Agent.Version=2.210.1

Docs for the task state that NuGet 4.1.0 is the default. So the PR for v2.210.0 of the NuGetCommand appears to have 'fixed' this. Following the guidance in the docs and using the NugetToolInstaller task to specify a higher version allowed us to successfully build again.

[samcarswell]

Starting seeing this issue today. Explicitly installing a Nuget version with NugetToolInstaller before running NuGetCommand solved the problem.

[swenkeratmicrosoft]

In case it helps anyone: We started seeing the same issue but resolved it in a different way as follows.

• Remove NuGetCommand@2
• Add NuGetAuthenticate@1
• Add the following to msbuildArguments when invoking MSBuild@1: /t:clean;restore;build

In other words, if you're using msbuild, you can let it do the nuget restore for you.

That way NuGetCommand@2 for restoring packages can't possibly cause us future issues.

[WTobor]

We are facing the same issue in our pipelines. We can create a workaround with installing a specific version of NuGet, but this is a temporary solution. When this issue will be fixed?