Add curl retry with backoff and download concurrency limit - #115
Closed
angerman wants to merge 2 commits into
Closed
Add curl retry with backoff and download concurrency limit#115angerman wants to merge 2 commits into
angerman wants to merge 2 commits into
Conversation
Fix transient HTTP 502 failures when downloading source tarballs from GitHub under high parallelism (-j 0). Two changes: 1. Add --retry 3 --retry-connrefused to curl invocations. Curl retries transient HTTP errors (408, 429, 500, 502, 503, 504) with built-in exponential backoff (1s, 2s, 4s). The existing actionRetry 5 is kept as a second layer for non-HTTP failures. 2. Cap concurrent downloads at 20 using Shake's Resource mechanism. This prevents hundreds of simultaneous curl processes from overwhelming GitHub when running with -j 0. Non-network tasks (patching, sdist creation, index building) still run at full parallelism.
angerman
added a commit
to angerman/cardano-haskell-packages-1
that referenced
this pull request
Feb 11, 2026
Point foliage input at angerman/foliage#fix/retry-backoff-download-concurrency (input-output-hk/foliage#115) to test the fix for transient HTTP 502 failures under high parallelism (-j 0).
- DeterminateSystems/nix-installer-action: v9 → v21 - DeterminateSystems/magic-nix-cache-action: v2 → v13 - cachix/cachix-action: v14 → v16 Fixes CI failures caused by TypeError: Invalid URL in the old nix-installer-action when fetching from install.determinate.systems.
Contributor
Author
|
Reopening as a non-fork PR from the same repo. |
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes transient HTTP 502 failures when downloading source tarballs from GitHub under high parallelism (
-j 0), as seen in cardano-haskell-packages CI run #21892846719 (PR #1248) which failed 4 times before succeeding on the 5th manual re-run.Root cause
Three compounding factors in
app/Foliage/FetchURL.hs:actionRetry 5retries with zero delay — five instantaneous retries against an overloaded server all fail within milliseconds-j 0means hundreds of concurrent curl processes hammer GitHub simultaneouslyChanges
--retry 3 --retry-connrefusedto curl invocations — curl retries transient HTTP errors (408, 429, 500, 502, 503, 504) with built-in exponential backoff (1s, 2s, 4s). The existingactionRetry 5is kept as a second layer for non-HTTP failures (DNS, disk errors).Resourcemechanism — prevents overwhelming GitHub when running with-j 0. Non-network tasks (patching, sdist creation, index building) still run at full parallelism.Test plan
cabal buildsucceedscabal test)-j 0