"Embedded nul in string" error in large files

[hyokangs]

I tried to load the following (zipped) tsv file with fread and then saved using write_fst (specifying uniform_encoding=F).

http://www.patentsview.org/data/20170808/uspatentcitation.zip
or
http://www.patentsview.org/data/20170808/rawinventor.zip

Tried to load fst file, but get the following error message:

Error in fstretrieve(fileName, columns, from, to, old_format) :
embedded nul in string: 'à«z\0\0\0\0ðÇ\005'

It seems that the specific wording of the error changes every time I try. Sometimes it successfully load the full data without error.

[MarcusKlik]

Hi @hyokangs, thanks for reporting!

I will try to reproduce your error, what OS and R version are you using?
Those are very interesting public data-sets by the way!

[hyokangs]

Hi @MarcusKlik,

Thank you for your response. My working environment yesterday was: Windows Server 2012R2 & R 3.3.3.
I tried to replicate the error with my laptop, Windows 10 Pro and R 3.3.4., but failed to do so -- meaning, everything worked fine with my laptop.

I will post an update once I encounter the same error again. Thanks!

[MarcusKlik]

Hi @hyokangs,

I managed to download your files and used the following code to test:

library(data.table)
library(fst)

# read and rewrite as fst file
x <- fread("rawinventor.tsv")
write_fst(x, "rawinventor.fst", uniform_encoding = FALSE)

# read from fst file
x_read <- read_fst("rawinventor.fst", as.data.table = TRUE)

# verify equality of both tables
testthat::expect_equal(x, x_read)

# contents
metadata_fst("rawinventor.fst")
#> <fst file>
#> 14959652 rows, 8 columns (rawinventor.fst)
#> 
#> * 'uuid'          : character
#> * 'patent_id'     : character
#> * 'inventor_id'   : character
#> * 'rawlocation_id': character
#> * 'name_first'    : character
#> * 'name_last'     : character
#> * 'sequence'      : integer
#> * 'rule_47'       : integer

Everything seems to work as expected (using the latest CRAN version of fst). Your error looks like a problem with reading the format correctly. Sometimes that happens when you try to read from a fst file that was created with a dev version (< 0.8.0) of the package, does that apply to your case?

Is there a special reason why you are using uniform_encoding = FALSE, you're file doesn't seem
to contain rows with different encoding than the first row?

Thanks for testing!

[hyokangs]

Thank you, @MarcusKlik.

I added "uniform_encoding=FALSE" after getting an error message ("Embedded nul in string"). This did not solve the problem, though.

After multiple tests, I think it is a problem of a network drive. Whenever I write to and read from my local drive, no problem occurs. When I try to write to and read from a network drive, I frequently encounter this problem. I believe this is due to slow AND unstable I/O.

Will update you if any problem occurs with my local drive.

[MarcusKlik]

Hi @hyokangs, thanks for testing! Yes, it looks like the byte stream is interrupted somehow when you load from the network drive. Still, normally loading from a network drive should just work (but slower obviously).

During a read or write, fst is firing a lot of short system API calls to the file system. Local (SSD) drives know how to handle (and combine) those effectively, but perhaps your network drive has problems with the high rate of calls. It would be interesting to learn whether the problem occurs during a write or read operation. When your file file is stored, you say that can you read the data correctly some of the times (in that case the error probably occurs in writing) ? Or do you get the same error message for the same file every time you load that file (error probably during read) ?

[Giqles]

I'm having what seems to be the same issue -- it seems to be write related for me. Writing to a network file system sometimes seems to fail silently -- I've found if I write then read until I get one successful read, it seems to be fine from then on.

I can't share my data or environment -- I am using fst v 0.8.2.

Some observations though:

1. it happened for me with a set of tables that contained quite a few character columns -- approx 300k lines and 30 character columns caused all of my problems so far.

2. other tables that were much larger had no issues -- 200m rows and 60 numeric columns didn't present any problems so far

If this is related to the byte stream getting interrupted would increasing the amount of compression help reduce the number of errors? (by making the amount to actually write to the file system smaller?)

[MarcusKlik]

Hi @Giqles, thanks for reporting. Yes, apparently the stream gets interrupted by the large number of relatively small writes to the network drive. One of the features that is planned to be released soon is a hash calculation of the actual data blocks that are stored. That would give the user a 'clean' error when incorrect data is read from disk (and could pinpoint the problem to the responsible column and approximate row number). Perhaps that will offer some information as to which column types are lost most often.

Also, writing much larger blocks for 'slow' drives would solve the problem I hope, because network drives usually have larger cache size than local drives.

Perhaps offering the user a 'network' mode with strong compression (as you say) and bigger chunks for writing would be a good option, although I would like to understand what's causing the problem so that we can fix it for any setting.

[Giqles]

I had what might be a related issue -- a file I've worked with a lot got a small subset of dates set (incorrectly) to 1970-01-01 in the stored file on one write. This didn't happen consistently; running exactly the same code again when I spotted the issue didn't result in any noticeable problems with the data.

[shrektan]

@MarcusKlik (I'm using fst vesion 0.8.8.) I was bite by the same error at times... Sorry that I can't reproduce it stably but the error occurs every a few weeks... What weird is, if I delete the file and re-write again, it works fine...

The target table should be around 1000 KBs. However, when the error happens, I will find that the size of the file increases to 3GBs...

The table I write contains some non-ASCII characters. I doubt that relates to the bug although I have other tables contain non-ASCII strings which never had a problem.

The error message is something like below:

   [2018-12-22 03:05:52.219][INFO] synchoronizing tbl `mf_chargeratenew`...
    Error in fstretrieve(fileName, columns, from, to, old_format) : 
      embedded nul in string: '\0\0\0\0'
    Calls: <Anonymous> ... <Anonymous> -> <Anonymous> -> <Anonymous> -> fstretrieve -> .Call
    Execution halted

[MarcusKlik]

Hi @shrektan, thanks for reporting!

Do you also see these occasionally errors when reading from a network drive ? It's difficult to pinpoint the exact problem with the errors reported by @Giqles and @hyokangs. I have been trying to reproduce it by reading large fst files with character columns using various encodings from a network drive but didn't get any errors so far.

But your problem seems to be different with the file blowing up to a large size. Do I understand you correctly that you don't get an error while writing but you see a result file that's much larger than can be expected? And when you read that (large) file, you get the error that you reported above?

thanks for testing, if you would have a (generated) sample file with which you can trigger the (occasional) error, that would be great for debugging!