Creating character dataset with LZF filter crashes R

[lazappi]

We were having a problem with R crashing when trying to write a file with LZF compression. I've tracked it down to trying to create a character dataset with the LZF filter:

h5f <- rhdf5::H5Fcreate(tempfile())
rhdf5::h5createDataset(
    h5f,
    dataset = "test",
    dims = 1,
    storage.mode = "character",
    filter = "LZF"
)

Running this crashes my R session with this output (on Linux and Mac):

 *** caught segfault ***
address 0x1, cause 'memory not mapped'

Traceback:
 1: H5Pset_lzf(dcpl, h5tid = dtype)
 2: .createDCPL(chunk, dims, level, fillValue, dtype = tid, filter = filter,     shuffle = shuffle)
 3: rhdf5::h5createDataset(h5f, dataset = "test", dims = 1, storage.mode = "character",     filter = "LZF")

[Bisaloo]

The issue is that we pass a buffer of size NULL here:

rhdf5/R/H5P_extras.R

Line 86 in 919a739

res <- .Call("_H5Pset_lzf", h5plist@ID, as.integer(bufsize), PACKAGE='rhdf5')

Would it make sense at all for your use case to use a simpler array with fixed-length strings rather than disabling compression? You can see relevant examples & discussion here:

rhdf5/R/h5create.R

Lines 371 to 385 in 919a739

	#' ## create two string datasets
	#' ## the first is variable length strings, the second fixed at the length of our longest word
	#' h5createDataset(f1, "strings", dims = length(words), storage.mode = "character",
	#' size = NULL, chunk = 25)
	#' h5createDataset(f2, "strings", dims = length(words), storage.mode = "character",
	#' size = max(nchar(words)), chunk = 25)
	#'
	#' ## Write the data
	#' h5write(words, f1, "strings")
	#' h5write(words, f2, "strings")
	#'
	#' ## Check file sizes.
	#' ## In this example the fixed length string dataset is normally much smaller
	#' file.size(f1)
	#' file.size(f2)

[lazappi]

I think in this case it needs to be variable length but I could check. Skipping the compression is not so bad but the crashes were a problem 😸

[Bisaloo]

Note that even if your strings don't have constant length, you don't necessarily have to use variable length. As described in the docs, we would pad the strings to set them all the the same length.

Note that in previous conversations, the benefits of variable length didn't seem obvious: #88 and

rhdf5/R/h5create.R

Lines 246 to 257 in 919a739

	#' The `size` argument is only used when `storage.mode = 'character'`. When
	#' storing strings HDF5 can use either a fixed or variable length datatype.
	#' Setting `size` to a positive integer will use fixed length strings where
	#' `size` defines the length. \pkg{rhdf5} writes null padded strings by default
	#' and so to avoid data loss the value provided here should be the length of the
	#' longest string. Setting `size = NULL` will use variable length strings. The
	#' choice is probably dependent on the nature of the strings you're writing. The
	#' principle difference is that a dataset of variable length strings will not be
	#' compressed by HDF5 but each individual string only uses the space it
	#' requires, whereas in a fixed length dataset each string is of length uses
	#' `size`, but the whole dataset can be compressed. This explored more in the
	#' examples below.

If you have a use case where you observe something different, I would be interested to hear about it.

The crash is indeed definitely a bug. Thanks for the report!

[lazappi]

I've never looked into the details but the specification we are working with says strings should be stored as variable length so we've stuck with that https://anndata.readthedocs.io/en/stable/fileformat-prose.html#string-arrays