[resty] [retry] Retry-After w/ Jitter Support and Slower Status Checks #111
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: hfuss <[email protected]>
Signed-off-by: hfuss <[email protected]>
Signed-off-by: hfuss <[email protected]>
Signed-off-by: hfuss <[email protected]>
onelapahead
changed the title
onelapahead
commented
Jul 15, 2025
Comment on lines
+126
to
+135
| SetRetryCount(5). | ||
| SetRetryAfter(func(c *resty.Client, r *resty.Response) (time.Duration, error) { | ||
| tflog.Debug(logCtx, fmt.Sprintf("retryAfter: %s", r.Header().Get("Retry-After"))) | ||
| retryAfter, err := strconv.ParseFloat(r.Header().Get("Retry-After"), 64) // decimal seconds | ||
| if err != nil { | ||
| retryAfter = 1.0 | ||
| } | ||
| jitter := time.Duration(rand.NormFloat64() * float64(5*time.Second)) // up to 5 second random jitter to account for 5 concurrent connections all retrying up to 5 times | ||
| return time.Duration(retryAfter*float64(time.Second)) + jitter, nil | ||
| }). |
Contributor
Author
There was a problem hiding this comment.
We should also consider adding throttling, so no matter what the client doesn't go faster than 1 req every 250ms ?
onelapahead
commented
Jul 22, 2025
| if err != nil { | ||
| retryAfter = 1.0 | ||
| } | ||
| jitter := time.Duration(rand.NormFloat64() * float64(5*time.Second)) // up to 5 second random jitter to account for 5 concurrent connections all retrying up to 5 times |
Contributor
Author
There was a problem hiding this comment.
This is wrong - should rand.Float64() otherwise we could have negative jitter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For ExtraSmall runtimes with conservative rate limits - we need to retry much more slowly.
For general HTTP requests we use the
Retry-Afterheader if present, or just 1s, with random jitter up to 5s, to retry 429s.For resources that use
Retryto check the status of a resource before marking it completed (contract builds/actions, FireFly registrations, etc.) - we slow down the retry factor significantly, and add code to try to treat explicit 429 errors as retries.