Skip to content

Handle rate limit waiting on Celery level #2927

@nforro

Description

@nforro

Rate limiting headers such as Retry-After or X-Ratelimit-Reset that provide an amount of waiting time needed before retrying the request are currently handled by blocking wait on urllib level. This makes the Celery tasks that hit the limits block for the specified time, congesting workers and potentially getting timeout-killed.
When a task is rate-limited, the waiting time should be propagated and the task scheduled for retry after that time on Celery level.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions