Worker should follow instructions in HTTP 429 responses

[MichaelRoeder]

Description

As a website owner, I expect crawlers to behave. If they are crawling too fast and my server responds with HTTP 429 (RFC 6585), the crawler should follow the instructions and slow down.

[MichaelRoeder]

Note: If Web API returns status code 429, it means that you have sent too many requests. When this happens, check the Retry-After header, where you will see a number displayed. This is the number of seconds that you need to wait, before you try your request again.