Slow when reading data

[waynelapierre]

I have a large data in R and I save it in fst format using the fst package with compression rate of 100. When I try to read it into R, it takes a lot of time and crashes my laptop. The fst file data is 873 MB, is it normal or it could be something wrong with my laptop?

PS: The original data is about 10 GB, so after compression with a rate of 100, it should be 100 MB, but the resulted fst file is 873 MB, and I don't understand that.

[MarcusKlik]

Hi @waynelapierre, thanks for submitting! At a compression setting of 100, fst will use the maximum amount of compression that it has available (for most column types, that means ZSTD compression at the maximum setting). But (unfortunately), that will usually not mean that the resulting file will be a factor 100 smaller (I wish :-)).

You are getting to a compression factor of ~11, which is already pretty good. If you're interested in the speed vs. compression ratio benchmarks of ZSTD, there is some information at the ZSTD project site on that. Note however, that fst has specific compression settings for each column type and most reported benchmarks use some piece of large text to measure performance (and a large body of text is not very representative of the data used in common data science tasks).

Can you provide some more information on the crash you are experiencing? Are you sure that you have enough memory available for reading in the complete dataset (see also #170) ?

thanks!

[MarcusKlik]

Hi @waynelapierre, do you still have crashes when loading your compressed fst file?

If you have a reproducible example to share, that would be great!