Problems with CoinGecko request rate limiter

[obrusvit]

For the Release 25.11, we weren't able to prepare new definitions. We should fix it for the next release. New coins are not properly listed. See also: slack.

We tried:

git submodule update --init --recursive --force && poetry run ./do_update.sh

Result:

(cmd)(definitions-py3.12) obrusvit@obrusvit-X1:~/repos/definitions$ ./do_update.sh
Submodule path 'coins_details/trezor_common': checked out '1f486f59ca9b1607bab4ebdcc3b3c59025370fb6'
Submodule path 'ethereum-lists/chains': checked out '784207b9f10a8140ac87a3034fdfb7f769105119'
Submodule path 'ethereum-lists/tokens': checked out 'e0794eefac7af977ee7913434bacbde1e1be4e02'
Fetching data from https://api.coingecko.com/api/v3/asset_platforms
Starting new HTTPS connection (1): api.coingecko.com:443
[https://api.coingecko.com:443](https://api.coingecko.com/) "GET /api/v3/asset_platforms HTTP/1.1" 200 None
Fetching data from https://api.llama.fi/chains
Starting new HTTPS connection (1): api.llama.fi:443
[https://api.llama.fi:443](https://api.llama.fi/) "GET /chains HTTP/1.1" 200 None
Fetching data from https://tokens.coingecko.com/ethereum/all.json
Starting new HTTPS connection (1): tokens.coingecko.com:443
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /ethereum/all.json HTTP/1.1" 200 329903
Fetching data from https://tokens.coingecko.com/ubiq/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /ubiq/all.json HTTP/1.1" 403 None
Fetching data from https://tokens.coingecko.com/quai-network/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /quai-network/all.json HTTP/1.1" 200 172
Fetching data from https://tokens.coingecko.com/optimistic-ethereum/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /optimistic-ethereum/all.json HTTP/1.1" 200 16454
Fetching data from https://tokens.coingecko.com/flare-network/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /flare-network/all.json HTTP/1.1" 200 1338
Fetching data from https://tokens.coingecko.com/songbird/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /songbird/all.json HTTP/1.1" 200 619
Fetching data from https://tokens.coingecko.com/elastos/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /elastos/all.json HTTP/1.1" 200 822
Fetching data from https://tokens.coingecko.com/kardiachain/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /kardiachain/all.json HTTP/1.1" 200 752
Fetching data from https://tokens.coingecko.com/cronos/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /cronos/all.json HTTP/1.1" 200 8336
Fetching data from https://tokens.coingecko.com/rootstock/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /rootstock/all.json HTTP/1.1" 200 1063
Fetching data from https://tokens.coingecko.com/telos/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /telos/all.json HTTP/1.1" 200 980
Fetching data from https://tokens.coingecko.com/lukso/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /lukso/all.json HTTP/1.1" 200 484
Fetching data from https://tokens.coingecko.com/darwinia-crab-network/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /darwinia-crab-network/all.json HTTP/1.1" 403 None
Fetching data from https://tokens.coingecko.com/xdc-network/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /xdc-network/all.json HTTP/1.1" 200 1871
Fetching data from https://tokens.coingecko.com/coinex-smart-chain/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /coinex-smart-chain/all.json HTTP/1.1" 200 366
Fetching data from https://tokens.coingecko.com/zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Incremented Retry for (url='/zyx/all.json'): Retry(total=4, connect=None, read=None, redirect=None, status=None)
Retry: /zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Incremented Retry for (url='/zyx/all.json'): Retry(total=3, connect=None, read=None, redirect=None, status=None)
Retry: /zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Incremented Retry for (url='/zyx/all.json'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
Retry: /zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Incremented Retry for (url='/zyx/all.json'): Retry(total=1, connect=None, read=None, redirect=None, status=None)
Retry: /zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Incremented Retry for (url='/zyx/all.json'): Retry(total=0, connect=None, read=None, redirect=None, status=None)
Retry: /zyx/all.json
[https://tokens.coingecko.com:443](https://tokens.coingecko.com/) "GET /zyx/all.json HTTP/1.1" 429 None
Traceback (most recent call last):
  File "/home/obrusvit/repos/definitions/cli.py", line 34, in <module>
    cli()
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/click/core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/click/core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/click/core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/click/core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/click/core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/repos/definitions/eth_definitions/download.py", line 474, in download
    cg_tokens = _load_erc20_tokens_from_coingecko(downloader, networks)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/repos/definitions/eth_definitions/download.py", line 265, in _load_erc20_tokens_from_coingecko
    all_tokens = downloader.get_coingecko_tokens_for_network(network_id)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/repos/definitions/eth_definitions/download.py", line 167, in get_coingecko_tokens_for_network
    raise err
  File "/home/obrusvit/repos/definitions/eth_definitions/download.py", line 161, in get_coingecko_tokens_for_network
    data = self._download_json(url)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obrusvit/repos/definitions/eth_definitions/download.py", line 138, in _download_json
    r.raise_for_status()
  File "/home/obrusvit/.cache/pypoetry/virtualenvs/definitions-VdydbjpS-py3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://tokens.coingecko.com/zyx/all.json