When TRD fails, the next run attempts to run the failed payments again. But this results in a lot of calls to tzkt api. Recently, it seems that this endpoint has been rate-limited, causing failure like the example below (taken on ghostnet TRD).
A workaround is to delete the failed payment folder (assuming it's a solid failure and not a partial payment). But it would be good to look into why it's querying every
│ 2023-06-22 21:39:11,754 - producer - INFO - Summary 4 paid, 0 done, 0 injected, 9 fail, 1 avoided │
│ Exception in thread producer: │
│ Traceback (most recent call last): │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn │
│ conn = connection.create_connection( │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 72, in create_connection │
│ for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): │
│ File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo │
│ for res in _socket.getaddrinfo(host, port, family, type, proto, flags): │
│ socket.gaierror: [Errno -3] Try again │
│ │
│ During handling of the above exception, another exception occurred: │
│ │
│ Traceback (most recent call last): │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen │
│ httplib_response = self._make_request( │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request │
│ self._validate_conn(conn) │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn │
│ conn.connect() │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect │
│ self.sock = conn = self._new_conn() │
│ File "/app/.local/lib/python3.10/site-packages/urllib3/connection.py", line 186, in _new_conn │
│ raise NewConnectionError( │
│ urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7ff7a2064460>: Failed to establish a new connection: [Errno -3] Try again │
│ │
│ During handling of the above exception, another exception occurred:
This was described on slack and I've seen it as well:
When TRD fails, the next run attempts to run the failed payments again. But this results in a lot of calls to tzkt api. Recently, it seems that this endpoint has been rate-limited, causing failure like the example below (taken on ghostnet TRD).
A workaround is to delete the failed payment folder (assuming it's a solid failure and not a partial payment). But it would be good to look into why it's querying every
Link to baker slack discussion: https://tezos-baking.slack.com/archives/CQ35AM8KE/p1685978794781209