[Feature] GitHub API rate limiting: Download release assets using API and add GitHub token to it

[o-l-a-v]

Feature Request

Is your feature request related to a problem? Please describe.

It seems that Scoop does not add GitHub token when downloading GitHub release assets, only for API calls.

Scoop/lib/download.ps1

Lines 89 to 197 in 0d0334c

	function Invoke-Download ($url, $to, $cookies, $progress) {
	# download with filesize and progress indicator
	$reqUrl = ($url -split '#')[0]
	$wreq = [Net.WebRequest]::Create($reqUrl)
	if ($wreq -is [Net.HttpWebRequest]) {
	$wreq.UserAgent = Get-UserAgent
	if (-not ($url -match 'sourceforge\.net' -or $url -match 'portableapps\.com')) {
	$wreq.Referer = strip_filename $url
	}
	if ($url -match 'api\.github\.com/repos') {
	$wreq.Accept = 'application/octet-stream'
	$wreq.Headers['Authorization'] = "Bearer $(Get-GitHubToken)"
	$wreq.Headers['X-GitHub-Api-Version'] = '2022-11-28'
	}
	if ($cookies) {
	$wreq.Headers.Add('Cookie', (cookie_header $cookies))
	}
	get_config PRIVATE_HOSTS | Where-Object { $_ -ne $null -and $url -match $_.match } | ForEach-Object {
	(ConvertFrom-StringData -StringData $_.Headers).GetEnumerator() | ForEach-Object {
	$wreq.Headers[$_.Key] = $_.Value
	}
	}
	}
	try {
	$wres = $wreq.GetResponse()
	} catch [System.Net.WebException] {
	$exc = $_.Exception
	$handledCodes = @(
	[System.Net.HttpStatusCode]::MovedPermanently, # HTTP 301
	[System.Net.HttpStatusCode]::Found, # HTTP 302
	[System.Net.HttpStatusCode]::SeeOther, # HTTP 303
	[System.Net.HttpStatusCode]::TemporaryRedirect # HTTP 307
	)
	# Only handle redirection codes
	$redirectRes = $exc.Response
	if ($handledCodes -notcontains $redirectRes.StatusCode) {
	throw $exc
	}
	# Get the new location of the file
	if ((-not $redirectRes.Headers) -or ($redirectRes.Headers -notcontains 'Location')) {
	throw $exc
	}
	$newUrl = $redirectRes.Headers['Location']
	info "Following redirect to $newUrl..."
	# Handle manual file rename
	if ($url -like '*#/*') {
	$null, $postfix = $url -split '#/'
	$newUrl = "$newUrl`#/$postfix"
	}
	Invoke-Download $newUrl $to $cookies $progress
	return
	}
	$total = $wres.ContentLength
	if ($total -eq -1 -and $wreq -is [net.ftpwebrequest]) {
	$total = ftp_file_size($url)
	}
	if ($progress -and ($total -gt 0)) {
	[console]::CursorVisible = $false
	function Trace-DownloadProgress ($read) {
	Write-DownloadProgress $read $total $url
	}
	} else {
	Write-Host "Downloading $url ($(filesize $total))..."
	function Trace-DownloadProgress {
	#no op
	}
	}
	try {
	$s = $wres.getresponsestream()
	$fs = [io.file]::openwrite($to)
	$buffer = New-Object byte[] 2048
	$totalRead = 0
	$sw = [diagnostics.stopwatch]::StartNew()
	Trace-DownloadProgress $totalRead
	while (($read = $s.read($buffer, 0, $buffer.length)) -gt 0) {
	$fs.write($buffer, 0, $read)
	$totalRead += $read
	if ($sw.elapsedmilliseconds -gt 100) {
	$sw.restart()
	Trace-DownloadProgress $totalRead
	}
	}
	$sw.stop()
	Trace-DownloadProgress $totalRead
	} finally {
	if ($progress) {
	[console]::CursorVisible = $true
	Write-Host
	}
	if ($fs) {
	$fs.close()
	}
	if ($s) {
	$s.close()
	}
	$wres.close()
	}
	}

Related:

• Excavator - Handle GitHub API rate limiting GithubActions#64
• [Feature] GitHub API rate limiting: Cache response from checkver and use it with autoupdate #6608

Describe the solution you'd like

Add GitHub token when downloading release assets from GitHub releases too.

• Use asset.url instead of asset.browser_download_url.
• Add header Accept: application/octet-stream, else it won't download the file.
• Add GitHub token if present.

Looks like this requires Scoop to download using the API, instead of the browser download URL?

Example: AzCopy v10.32.1:

• Latest version API: https://api.github.com/repos/Azure/azure-storage-azcopy/releases
• Example browser download URL: https://github.com/Azure/azure-storage-azcopy/releases/download/v10.32.1/azcopy_windows_amd64_10.32.1.zip
  • Redirects to: https://release-assets.githubusercontent.com/github-production-release-asset/*
• Download using the API: https://api.github.com/repos/Azure/azure-storage-azcopy/releases/assets/{asset_id}
  • Example: https://api.github.com/repos/Azure/azure-storage-azcopy/releases/assets/358222966
    • Invoke-RestMethod -Method 'Get' -Uri 'https://api.github.com/repos/Azure/azure-storage-azcopy/releases/assets/358222966' -Headers @{Accept = 'application/octet-stream'} -OutFile ('{0}/file.zip' -f [System.Environment]::GetFolderPath('Desktop'))

Minimal example for downloading a GitHub release asset using the API instead:

# Get available versions
$Versions = Invoke-RestMethod -Method 'Get' -Uri 'https://api.github.com/repos/Azure/azure-storage-azcopy/releases'

# Get latest stable version
$Latest = $Versions.Where({-not $_.prerelease}, 'First')[0]

# View available assets and download URL if using API
$Latest.assets.ForEach{
    [PSCustomObject]@{
        'name' = $_.'name'
        'url'  = $_.'url'
        'hash' = $_.'digest'
    }
}

# Download
$Asset = $Latest.assets.Where({$_.'name' -eq 'azcopy_windows_amd64_10.32.1.zip'}, 'First')[0]
$OutFile = [System.IO.Path]::Combine($env:TEMP, $Asset.'name')
$Headers = [ordered]@{
    'Accept' = [string] 'application/octet-stream'
    # Here we could add GitHub token too
}
Remove-Item -ErrorAction 'Ignore' -Path $OutFile -Force
Invoke-RestMethod -Method 'Get' -Uri $Asset.'url' -OutFile $OutFile -Headers $Headers
(Get-FileHash -Path $OutFile -Algorithm 'SHA256').'Hash' -eq $Asset.'digest'.Split(':')[-1]

Describe alternatives you've considered