This repository was archived by the owner on Mar 4, 2025. It is now read-only.
This repository was archived by the owner on Mar 4, 2025. It is now read-only.
API download function seems unreliable :( #375
Open
Description
As part of post-migration testing of things in the new data centre, I'm running some sanity tests to verify things are working ok.
However, it appears the API download() function doesn't work properly with larger sized database files:
$ curl -F apikey="MY_API_KEY" -F dbowner="justinclift" -F dbname="National Statistics Postcode Lookup UK.sqlite" -o stuff.db https://api.dbhub.io/v1/download
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
43 771M 43 337M 100 442 33.3M 43 0:00:23 0:00:10 0:00:13 34.1M
curl: (18) transfer closed with 454725632 bytes remaining to read
That's happening multiple times in a row with databases of non-trivial size.
The backend isn't showing the problem with any useful detail either:
Jun 18 01:53:16 api systemd[1]: Started dbhub-api.service - DBHub.io API daemon.
Jun 18 01:53:16 api api[535]: 2024/06/18 01:53:16 API server: request log opened: /var/log/dbhub/api.log
[...]
Jun 18 01:54:02 api api[535]: 2024/06/18 01:54:02 Error returning DB file: write tcp 103.216.220.135:443->203.12.2.188:41960: i/o timeout
Jun 18 01:54:19 api api[535]: 2024/06/18 01:54:19 Error returning DB file: write tcp 103.216.220.135:443->203.12.2.188:37868: i/o timeout
Jun 18 01:55:08 api api[535]: 2024/06/18 01:55:08 Error returning DB file: write tcp 103.216.220.135:443->203.12.2.188:54474: i/o timeout
Jun 18 01:56:16 api api[535]: 2024/06/18 01:56:16 Error returning DB file: write tcp 103.216.220.135:443->203.12.2.188:60412: i/o timeout
It's happening on both the old and new API servers(!), and none of MinIO, Memcached, nor PostgreSQL are displaying any kind of error either.
After the migration is completed we'd better take a look at this.