Use a dedicated crate to manage call retries

[jparklev]

Based on this note #1 (comment) in the review done in PR #1.

Consider either https://docs.rs/retry/1.3.0/retry/ or https://docs.rs/backoff/0.3.0/backoff/

[jparklev]

Update: production was struggling with ethscan a bit, so I went ahead and added the second option here (backoff, the first didn't have futures support (link)) in for the ethscan requests: 0799e26

Should still update the run_selector call that uses manual retry logic, so that's To Do

cc @abylaw

[abylaw]

Awesome, thanks for catching that. Was it struggling with the rate-limit? We might also want to check whether ethscan implements this header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 and retry accordingly. @jparklev

[jparklev]

Was it struggling with the rate-limit?

Yep, though the change I made actually doesn't address it (b/c the call succeeds, it just has a failure message)

We might also want to check whether ethscan implements this header: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 and retry accordingly

For sure, can't tell from their docs if they do, but do see see that "result":"Max rate limit reached, please use API Key for higher rate limit" will be in the response. Definitely think we should do retries on that call if we get a response like that (also potentially this check should be in the retry policy). Would you mind making a quick pass at this if you have any spare capacity in the coming days? @abylaw