Skip to content

utils: back off longer and correctly on retry_on_exception (#311) - #328

Open
robwoolley wants to merge 1 commit into
ros-infrastructure:masterfrom
robwoolley:retry-backoff
Open

utils: back off longer and correctly on retry_on_exception (#311)#328
robwoolley wants to merge 1 commit into
ros-infrastructure:masterfrom
robwoolley:retry-backoff

Conversation

@robwoolley

Copy link
Copy Markdown
Contributor

Fixes #311. Under concurrent CI load (multiple distro/platform combinations running at once), GitHub rate limits package.xml fetches with HTTP 429. retry_on_exception's default backoff totaled under 2 seconds (5 retries, starting at 0.125s) -- nowhere near enough for the rate limit to clear -- so fetches were giving up and packages silently failed to generate.

Two changes:

  • Raise the default retry budget so the backoff can actually ride out a real rate-limit window (10 retries, starting at 1s, doubling up to a 60s cap).
  • Fix the cap logic itself: it previously reset the interval back down to 0.125s after 6 retries instead of staying capped high, which meant retries sped back up right when they should have kept backing off the most under sustained rate limiting.

…structure#311)

Fixes ros-infrastructure#311. Under concurrent CI load
(multiple distro/platform combinations running at once), GitHub rate
limits package.xml fetches with HTTP 429. retry_on_exception's
default backoff totaled under 2 seconds (5 retries, starting at
0.125s) -- nowhere near enough for the rate limit to clear -- so
fetches were giving up and packages silently failed to generate.

Two changes:
- Raise the default retry budget so the backoff can actually ride out
  a real rate-limit window (10 retries, starting at 1s, doubling up
  to a 60s cap).
- Fix the cap logic itself: it previously reset the interval back
  down to 0.125s after 6 retries instead of staying capped high,
  which meant retries sped back up right when they should have kept
  backing off the most under sustained rate limiting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Rob Woolley <rob.woolley@windriver.com>
@robwoolley
robwoolley requested a review from cottsay August 20, 2026 10:29
@robwoolley

Copy link
Copy Markdown
Contributor Author

@cottsay Please may I get your feedback on this change to superflore? I am adjusting the backoff from 0.125, 0.25, 0.5, 1, 2 to 1,2,4,8,16 to give GitHub a chance to recover and stop issuing HTTP error 429.

This typically happens when I try running multiple superflore scripts in different GHA at the same time and flood GitHub with requests from the same source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Uncaught HTTP Error causes problems with generated recipes

1 participant