Possible Bug #35
Description
I see the following issue frequently when I start/restart the service in production.
INFO: Started server process [7204]
2022-01-12 12:28:51.305 | INFO | asyncio.events:_run:80 - Started server process [7204]
INFO: Waiting for application startup.
2022-01-12 12:28:51.305 | INFO | uvicorn.server:startup:97 - Waiting for application startup.
INFO: Application startup complete.
2022-01-12 12:28:51.305 | INFO | uvicorn.server:startup:97 - Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:4010 (Press CTRL+C to quit)
2022-01-12 12:28:51.306 | INFO | uvicorn.server:startup:176 - Uvicorn running on http://127.0.0.1:4010 (Press CTRL+C to quit)
2022-01-12 12:29:15.795 | DEBUG | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:16.008 | DEBUG | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/a73ff62e4b648541e5fa75ce0a5edae4/raw/example_sensitivity_input.json HTTP/1.1" 200 33754
2022-01-12 12:29:23.690 | DEBUG | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:23.906 | DEBUG | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/52a5836735eff0d33692cfd06d049197/raw/uncertainty_inventory.json HTTP/1.1" 200 80
2022-01-12 12:29:23.913 | DEBUG | urllib3.connectionpool:_get_conn:281 - Starting new HTTPS connection (1): gist.githubusercontent.com:443
2022-01-12 12:29:24.119 | DEBUG | urllib3.connectionpool:urlopen:703 - https://gist.githubusercontent.com:443 "GET /rampvisdevelopment/acb6a1e6e33d0358553d7de09d6232e0/raw/sensitivity_inventory.json HTTP/1.1" 200 557
2022-01-12 12:29:59.205 | ERROR | app.services.download_service:download_urls:183 - IncompleteRead(209715200 bytes read, 43243700 more expected)
Traceback (most recent call last):
File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 912, in _bootstrap
self._bootstrap_inner()
│ └ <function Thread._bootstrap_inner at 0x7f877599b040>
└ <Thread(Thread-1, started 140219028420352)>
File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
│ └ <function Thread.run at 0x7f8775999d30>
└ <Thread(Thread-1, started 140219028420352)>
File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/threading.py", line 892, in run
self._target(*self._args, **self._kwargs)
│ │ │ │ │ └ {}
│ │ │ │ └ <Thread(Thread-1, started 140219028420352)>
│ │ │ └ ()
│ │ └ <Thread(Thread-1, started 140219028420352)>
│ └ <function run_agents at 0x7f8749ec0a60>
└ <Thread(Thread-1, started 140219028420352)>
File "./app/controllers/agents/__init__.py", line 52, in run_agents
download_data()
└ <function download_data at 0x7f874d017820>
File "./app/controllers/agents/data_downloader_agent.py", line 35, in download_data
download_urls(urls, DATA_PATH_LIVE)
│ │ └ '/home/scrc/rampvis-api/data-api/app/core/../../../data/live'
│ └ [{'name': 'sobol', 'url': 'https://gist.githubusercontent.com/rampvisdevelopment/a73ff62e4b648541e5fa75ce0a5edae4/raw/example...
└ <function download_urls at 0x7f8749eb9790>
> File "./app/services/download_service.py", line 159, in download_urls
df = pd.read_csv(url['url'], encoding='iso-8859-1')
│ │ └ {'name': 'phe', 'url': 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=c...
│ └ <function read_csv at 0x7f87658790d0>
└ <module 'pandas' from '/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/__init__.py'>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
│ │ └ {'encoding': 'iso-8859-1'}
│ └ ('https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv',)
└ <function read_csv at 0x7f8765876f70>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
│ │ └ {'delimiter': ',', 'header': 'infer', 'names': None, 'index_col': None, 'usecols': None, 'squeeze': False, 'prefix': None, 'm...
│ └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
└ <function _read at 0x7f8765876e50>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 482, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
│ │ └ {'delimiter': ',', 'header': 'infer', 'names': None, 'index_col': None, 'usecols': None, 'squeeze': False, 'prefix': None, 'm...
│ └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
└ <class 'pandas.io.parsers.readers.TextFileReader'>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 811, in __init__
self._engine = self._make_engine(self.engine)
│ │ │ │ └ 'c'
│ │ │ └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
│ │ └ <function TextFileReader._make_engine at 0x7f8765879700>
│ └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
└ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1040, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
│ │ │ │ │ └ {'delimiter': ',', 'escapechar': None, 'quotechar': '"', 'quoting': 0, 'doublequote': True, 'skipinitialspace': False, 'linet...
│ │ │ │ └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
│ │ │ └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
│ │ └ <pandas.io.parsers.readers.TextFileReader object at 0x7f8748100fa0>
│ └ 'c'
└ {'c': <class 'pandas.io.parsers.c_parser_wrapper.CParserWrapper'>, 'python': <class 'pandas.io.parsers.python_parser.PythonPa...
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/c_parser_wrapper.py", line 51, in __init__
self._open_handles(src, kwds)
│ │ │ └ {'delimiter': ',', 'escapechar': None, 'quotechar': '"', 'quoting': 0, 'doublequote': True, 'skipinitialspace': False, 'linet...
│ │ └ 'https://api.coronavirus.data.gov.uk/v2/data?areaType=ltla&metric=vaccinationsAgeDemographics&format=csv'
│ └ <function ParserBase._open_handles at 0x7f8765946f70>
└ <pandas.io.parsers.c_parser_wrapper.CParserWrapper object at 0x7f8748100fd0>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/parsers/base_parser.py", line 222, in _open_handles
self.handles = get_handle(
│ │ └ <function get_handle at 0x7f8765d49280>
│ └ None
└ <pandas.io.parsers.c_parser_wrapper.CParserWrapper object at 0x7f8748100fd0>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/common.py", line 609, in get_handle
ioargs = _get_filepath_or_buffer(
└ <function _get_filepath_or_buffer at 0x7f8765d49040>
File "/home/scrc/rampvis-api/data-api/venv/lib/python3.9/site-packages/pandas/io/common.py", line 317, in _get_filepath_or_buffer
reader = BytesIO(req.read())
│ │ └ <function HTTPResponse.read at 0x7f87741655e0>
│ └ <http.client.HTTPResponse object at 0x7f873be42a60>
└ <class '_io.BytesIO'>
File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/http/client.py", line 471, in read
s = self._safe_read(self.length)
│ │ │ └ 252958900
│ │ └ <http.client.HTTPResponse object at 0x7f873be42a60>
│ └ <function HTTPResponse._safe_read at 0x7f87741659d0>
└ <http.client.HTTPResponse object at 0x7f873be42a60>
File "/home/scrc/miniconda3/envs/py-3.9.2/lib/python3.9/http/client.py", line 614, in _safe_read
raise IncompleteRead(data, amt-len(data))
│ │ │ └ b'areaCode,areaName,areaType,date,age,VaccineRegisterPopulationByVaccinationDate,cumPeopleVaccinatedCompleteByVaccinationDate...
│ │ └ 252958900
│ └ b'areaCode,areaName,areaType,date,age,VaccineRegisterPopulationByVaccinationDate,cumPeopleVaccinatedCompleteByVaccinationDate...
└ <class 'http.client.IncompleteRead'>
http.client.IncompleteRead: IncompleteRead(209715200 bytes read, 43243700 more expected)
2022-01-12 12:30:38.394 | INFO | app.algorithms.sim_search.cube_functions:cube_master:91 - cube_master will take a few minutes...