MinerConnectionError when "helium-miner" container is down

[posterzh]

Follow up to:

• MinerConnectionError: Unable to connect to miner http://helium-miner:4467 #350

Thoughts:

• For any miner, if helium-miner container is down, then one of the following actions makes MinerConnectionError.
  • open the diagnostics page.
  • call diagnostics/json endpoint
  • upload the diagnostics to the Google big query

For example, Balena device https://dashboard.balena-cloud.com/devices/cae984a6b29efe5198bc582744310fe9

• stop the helium-miner container intentionally
  

• open the diagnostics web page
  

• MinerConnectionError is observed from the Balena logs for diagnostics container
  

INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
DEBUG:flask_caching.backends.simplecache:set key 'view//json'
DEBUG:urllib3.connectionpool:https://o571444.ingest.sentry.io:443 "POST /api/5730184/envelope/ HTTP/1.1" 200 2
DEBUG:flask_caching.backends.simplecache:get key 'view//json' -> miss (expired)
INFO:root:Running periodic hardware diagnostics
2022-07-05 05:26:25,786 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(36) - gateway_mfr response stdout: b'{\n  "key": "112Jp9VrwUBQaVd2f2fhKJvxFm2uf2dVQSiQFpaLjrDJtMAqbb7c",\n  "name": "cool-brick-goblin"\n}\n'
INFO:hm_pyhelper.miner_param:gateway_mfr response stdout: b'{\n  "key": "112Jp9VrwUBQaVd2f2fhKJvxFm2uf2dVQSiQFpaLjrDJtMAqbb7c",\n  "name": "cool-brick-goblin"\n}\n'
2022-07-05 05:26:25,787 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(38) - gateway_mfr response stderr: b''
INFO:hm_pyhelper.miner_param:gateway_mfr response stderr: b''
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): helium-miner:4467
2022-07-05 05:26:25,823 - [ERROR] - hm_pyhelper.miner_param - (miner.py).fetch_miner_data -- /opt/python-dependencies/hw_diag/utilities/miner.py:(21) - Unable to connect to miner http://helium-miner:4467
Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/python-dependencies/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python-dependencies/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1276, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1322, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1271, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1031, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 969, in send
    self.connect()
  File "/opt/python-dependencies/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/python-dependencies/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/python-dependencies/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 27, in __fetch_data
    response = requests.post(self.url, json=req_body)
  File "/opt/python-dependencies/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/python-dependencies/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python-dependencies/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/utilities/miner.py", line 10, in fetch_miner_data
    peerbook = client.get_peer_book()[0]
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 62, in get_peer_book
    return self.__fetch_data('peer_book', addr='self')
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 29, in __fetch_data
    raise MinerConnectionError(
hm_pyhelper.miner_json_rpc.exceptions.MinerConnectionError: Unable to connect to miner http://helium-miner:4467
ERROR:hm_pyhelper.miner_param:Unable to connect to miner http://helium-miner:4467
Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/opt/python-dependencies/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python-dependencies/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1276, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1322, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1271, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1031, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 969, in send
    self.connect()
  File "/opt/python-dependencies/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/opt/python-dependencies/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/opt/python-dependencies/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/opt/python-dependencies/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 27, in __fetch_data
    response = requests.post(self.url, json=req_body)
  File "/opt/python-dependencies/requests/api.py", line 117, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/opt/python-dependencies/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/python-dependencies/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/opt/python-dependencies/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='helium-miner', port=4467): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xf3c758b0>: Failed to establish a new connection: [Errno -5] No address associated with hostname'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/utilities/miner.py", line 10, in fetch_miner_data
    peerbook = client.get_peer_book()[0]
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 62, in get_peer_book
    return self.__fetch_data('peer_book', addr='self')
  File "/opt/python-dependencies/hm_pyhelper/miner_json_rpc/client.py", line 29, in __fetch_data
    raise MinerConnectionError(
hm_pyhelper.miner_json_rpc.exceptions.MinerConnectionError: Unable to connect to miner http://helium-miner:4467
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): fuzzy-marmalade-warlock.skittles.stakejoy.com:443
DEBUG:urllib3.connectionpool:https://fuzzy-marmalade-warlock.skittles.stakejoy.com:443 "GET /v1/blocks/height HTTP/1.1" 200 27
2022-07-05 05:26:26,271 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(79) - Retrieving list of BLE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of BLE device(s)
2022-07-05 05:26:26,286 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(99) - Found the following BLE Devices: [{'Address': '00:E9:3A:F9:D0:C2', 'Name': 'cae984a', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
INFO:hw_diag.utilities.hardware:Found the following BLE Devices: [{'Address': '00:E9:3A:F9:D0:C2', 'Name': 'cae984a', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
2022-07-05 05:26:26,287 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(145) - Retrieving list of LTE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of LTE device(s)
2022-07-05 05:26:26,297 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(172) - Found the following LTE Devices: []
INFO:hw_diag.utilities.hardware:Found the following LTE Devices: []
INFO:root:Frequency: EU868

[posterzh]

The reason of this issue is most likely the corrupted EMMC and the following actions can fix:

• Purge the device
• Reflash the EMMC
• Replace the EMMC