Caching/saving downloaded OHLCV data

[Random-Investments]

Hi Oleg and community. I hope you are doing well.

Playing around with the package, I have encountered myself with this issue: when downloading large sets of data with vectorbt.data.custom (multiple symbols OHLCV 1 minute data for several months for instance), sometimes I stumble upon some timeout error, I lose connection momentarily or something else happens, and the code throws an error and all the downloaded data so far is lost. Sometimes is really frustrating to, specially when a long time has passed and it's about to finish.

So I'm trying to come out with a solution to fix this. A way to save a temp-file that can be accessed if the code blows out and can be used as a starting point to resume the downloading. So far I have tried to pickle the result once in a while, but it doesn't seems to be very efficient and sometimes an error is shown when (I presume) the pickle is corrupted or contains incomplete data that can't be matched.

I tried, then, to use the cache decorators provided in vectorbt.utils.decorators but honestly I have no idea how to use them appropriately. I am about to start implementing some sort of SQL database or similar, but I remembered what is often said about this awesome package: there is some easier workaround lying somewhere.

So, there is some tool I can use to resume the downloads seamlessly without complicating my life?

As always, thanks a lot for your invaluable time, effort and help.

[polakowo]

Hi @GogeiD, this is a good idea, I’ll see how to add an option to persist data on the disk. Right now you can try to download some data, then call data.update with end value set to some further point in time, and repeat the process until you get all data.

[Random-Investments]

Great Oleg. I am glad you find it useful. One additional thing it may be a nice feature: having some sort of track of the remaining symbols to be downloaded. For instance, in the progress bar, something that says [downloading] 4 out of 10 symbols… it's just cosmetic and nothing really important, but may be something nice to have when downloading several symbols and you want to see how many symbols are left.

Anyway, thanks a lot and see you next time :)

[polakowo]

@GogeiD You mean apart from showing the progress bar for one symbol, as it's done right now? This is doable, just depends whether tqdm can play along with showing two progress bars at once.

[Random-Investments]

Two bars aren't necessary. What I am thinking, it's that at the beginning of the progress bar it can be shown something like this:

vbt.BinanceData.download(["BTCUSDT", "ETHUSDT", "BNBUSDT"], start='2 hours ago UTC', end='now UTC', interval='1m')
(1 out of 3) 2021-10-06 11:52:10.522000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:01,  1.68s/it]
(2 out of 3) 2021-10-06 11:52:12.975000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:01,  1.58s/it]
(3 out of 3) 2021-10-06 11:52:15.265000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:03,  3.84s/it]

or alternatively:

vbt.BinanceData.download(["BTCUSDT", "ETHUSDT", "BNBUSDT"], start='2 hours ago UTC', end='now UTC', interval='1m')
(BTCUSDT) 2021-10-06 11:52:10.522000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:01,  1.68s/it]
(ETHUSDT) 2021-10-06 11:52:12.975000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:01,  1.58s/it]
(BNBUSDT) 2021-10-06 11:52:15.265000+00:00 - 2021-10-06 13:52:00+00:00: : 1it [00:03,  3.84s/it]

It's just a little something extra that show where we at. I know it's simple and I can even try to implement it myself, but I am so new at python that I am afraid to break the code somehow :)

[polakowo]

I'm not a big fan of printing because vectorbt may be used in production systems where logging make more sense. I'll see how to do it.