add backoff-retry to shards downloading, 429 handling

[wolfv]

Description

I noticed a couple 429 issues when @hunger was grabbing shards from 20 different jobs with * as a package name spec.

Arguably, we should use the big repodata.json in this case.

Nevertheless, it might be good to add retries & backoff. I need to double check though whether we use the retry middleware already.

How Has This Been Tested?

Only with the mock tests.

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR.

Tools: Claude

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added sufficient tests to cover my changes.