Dotnet nuget push to GPR fails randomly

[RehanSaeed]

Details about Problem

dotnet.exe --version: 3.1.301
OS version: GitHub Actions windows-latest - Microsoft Windows Server 2019 10.0.17763

Detailed repro steps so we can see the same problem

I have two repos Schema.NET and FastestNuGet where I have the exact same build.yml file for GitHub Actions. However, dotnet nuget push fails on one and succeeds on the other.

The only difference I can understand is that one project has a . in the name and the other does not.

Failing CLI Output

Run dotnet nuget push .\windows-latest\*.nupkg --source GitHub --skip-duplicate
warn : No API Key was provided and no API Key could be found for 'https://nuget.pkg.github.com/RehanSaeed'. To save an API Key for a source use the 'setApiKey' command.
Pushing Schema.NET.7.1.1-preview.0.33.nupkg to 'https://nuget.pkg.github.com/RehanSaeed'...
  PUT https://nuget.pkg.github.com/RehanSaeed/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/RehanSaeed/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
  An existing connection was forcibly closed by the remote host.
  PUT https://nuget.pkg.github.com/RehanSaeed/
An error was encountered when fetching 'PUT https://nuget.pkg.github.com/RehanSaeed/'. The request will now be retried.
Error while copying content to a stream.
  Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
  An existing connection was forcibly closed by the remote host.
  PUT https://nuget.pkg.github.com/RehanSaeed/
error: Error while copying content to a stream.
error:   Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
error:   An existing connection was forcibly closed by the remote host.

build.yml

name: Build

on:
  push:
  pull_request:
  release:
    types:
      - published

env:
  # Set the DOTNET_SKIP_FIRST_TIME_EXPERIENCE environment variable to stop wasting time caching packages
  DOTNET_SKIP_FIRST_TIME_EXPERIENCE: true
  # Disable sending usage data to Microsoft
  DOTNET_CLI_TELEMETRY_OPTOUT: true
  # Set the build number in MinVer
  MINVERBUILDMETADATA: build.${{github.run_number}}

jobs:
  build:
    name: Build-${{matrix.os}}
    runs-on: ${{matrix.os}}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
    steps:
    - name: 'Checkout'
      uses: actions/checkout@v2
      with:
        lfs: true
        fetch-depth: 0
    - name: 'Git Fetch Tags'
      run: git fetch --tags
      shell: pwsh
    - name: 'Install .NET Core SDK'
      uses: actions/setup-dotnet@v1
      with:
        dotnet-version: 3.1.301
    - name: 'Dotnet Tool Restore'
      run: dotnet tool restore
      shell: pwsh
    - name: 'Dotnet Cake Build'
      run: dotnet cake --target=Build
      shell: pwsh
    - name: 'Dotnet Cake Test'
      run: dotnet cake --target=Test
      shell: pwsh
    - name: 'Dotnet Cake Pack'
      run: dotnet cake --target=Pack
      shell: pwsh
    - name: 'Publish Artefacts'
      uses: actions/upload-artifact@v1.0.0
      with:
        name: ${{matrix.os}}
        path: './Artefacts'

  push-github-packages:
    name: 'Push GitHub Packages'
    needs: build
    if: github.ref == 'refs/heads/master' || github.event_name == 'release'
    runs-on: windows-latest
    steps:
      - name: 'Download Artefact'
        uses: actions/download-artifact@v1
        with:
          name: 'windows-latest'
      - name: 'Dotnet NuGet Add Source'
        run: dotnet nuget add source https://nuget.pkg.github.com/RehanSaeed/index.json --name GitHub --username RehanSaeed --password ${{secrets.GITHUB_TOKEN}}
        shell: pwsh
      - name: 'Dotnet NuGet Push'
        run: dotnet nuget push .\windows-latest\*.nupkg --source GitHub --skip-duplicate
        shell: pwsh

[RehanSaeed]

There seem to be multiple issues describing this bug but they all seem to have been closed with unsatisfactory workarounds. The core issue is that the dotnet nuget push or dotnet nuget add source commands (I'm not sure which) seem to be broken.

• Running dotnet nuget push on GPR randomly fails #9493
• dotnet nuget push to GPR -- Invalid or unrecognized response #8580

[RehanSaeed]

I've tried re-running the build and it fails randomly 4/5 times for the Schema.NET build but is rock solid for FastestNuGet.

[RehanSaeed]

I've resorted to retrying to push a NuGet package to GitHub Actions.

$count = 0;
do { 
    ++$count;
    dotnet nuget push .\*.nupkg --source GitHub --skip-duplicate;
} 
while ($count -lt 30 -and $LastExitCode -ne 0)

[PeterKnealeBizCover]

Schema.NET contains a '.' in the name, while 'FastestNuGet' doesn't. I can't find where I read it but I recall someone noticing unreliable nuget publishing when the package name contained a period. Would it be difficult to try removing it?

[RehanSaeed]

Removing it is not an option.

[RehanSaeed]

Someone has written an entire dotnet global tool to workaround this bug:

https://github.com/jcansdale/gpr

There seem to be a lot of others experiencing the same issue:

https://twitter.com/RehanSaeedUK/status/1281705870917881856?s=20

[jcansdale]

Try passing the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER=0 env var to dotnet nuget push.

This seems to make it work most of the time.

Unfortunately I then see the odd error like this:

Pushing nuget-symbols.1.0.0-g6f7e8c13efc9a5ad6e2d28195a975d1b4b1a8d7c-33.nupkg to 'https://nuget.pkg.github.com/jcansdale-test'...
  PUT https://nuget.pkg.github.com/jcansdale-test/
warn : Missing nuget package in request
  BadRequest https://nuget.pkg.github.com/jcansdale-test/ 32ms
error: Response status code does not indicate success: 400 (Bad Request).

You can see what happens in the matrix run here:
https://github.com/jcansdale-test/nuget-symbols/actions/runs/178328401

[chadly]

Someone has written an entire dotnet global tool to workaround this bug:

I second @jcansdale's gpr tool. It solved all my problems pushing to GH packages and I'm not worried about this issue anymore.

I mean...the issue still needs to be dealt with, but I can at least move on with my life now.

[jcansdale]

I'm dropping some notes here about the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLE property, incase they might be useful. I grabbed them from an internal GitHub PR that repros this issue. It is still a mystery what is going on!

Description

Starting with .NET Core 2.1, the SocketsHttpHandler class provides the implementation used by higher-level HTTP networking classes such as HttpClient. The use of SocketsHttpHandler offers a number of advantages:

• A significant performance improvement when compared with the previous implementation.
• The elimination of platform dependencies, which simplifies deployment and servicing. For example, libcurl is no longer a dependency on .NET Core for macOS and .NET Core for Linux.
• Consistent behavior across all .NET platforms.

By defining an environment variable named DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLE and setting it to false, we can force dotnet nuget push to use the legacy HttpClientHandler instead of SocketsHttpHandler.

When SocketsHttpHandler is used, publishing sometimes fails on the client side like this:

info : Pushing Chicken.1.0.0-gb945a02962fd3753b51f67efb3b98c631f2f5189-macos-latest-2020-07-24-080210.nupkg to 'https://nuget.pkg.github.com/sweethaven-village'...
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
info : An error was encountered when fetching 'PUT https://nuget.pkg.github.com/sweethaven-village/'. The request will now be retried.
info : An error occurred while sending the request.
info :   The response ended prematurely.
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
info : An error was encountered when fetching 'PUT https://nuget.pkg.github.com/sweethaven-village/'. The request will now be retried.
info : An error occurred while sending the request.
info :   The response ended prematurely.
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
error: An error occurred while sending the request.
error:   The response ended prematurely.

When HttpClientHandler is used, publishing sometimes fails on the server side like this:

info : Pushing Chicken.1.0.0-g5e6a3076b547632358aac5994f5b6e0fcff14bd6-ubuntu-latest-2020-07-24-004434.nupkg to 'https://nuget.pkg.github.com/sweethaven-village'...
info :   PUT https://nuget.pkg.github.com/sweethaven-village/
warn : Missing nuget package in request
info :   BadRequest https://nuget.pkg.github.com/sweethaven-village/ 71ms
error: Response status code does not indicate success: 400 (Bad Request).

For this particular package, the first error seems to happen on macos-latest and the second on ubuntu-latest. I've seen the first error happen on all platforms for different packages.

[DamienDennehy]

After 9 failures in a row using DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER seems to have worked. Thank you @jcansdale. If it happens again I'll post any of the 400 warnings so this can be investigated more.