Feature Request: Parametrization with compression_opts

[kmuehlbauer]

I'm trying to make hdf5plugin usable within h5netcdf.

It works already nicely using the advertised approach using either compression=hdf5plugin.LZ4() or via **hdf5plugin.Blosc().

Now, for conciseness, I want to be able to directly use it like this:

compression = 320001  # blosc
compression_opts = (0, 0, 0, 0, 4, 1, 1)  # should work, since this is what will be provided by dict-unpacking

This does return without flaws:

import numpy
import h5py
import hdf5plugin
# Compression
with h5py.File('test.h5', 'w') as f:
    f.create_dataset(
        'data', 
        data=numpy.arange(100), 
        chunks=(50,), 
        compression=32001, 
        compression_opts=(0,0,0,0,4,1,1))
# Decompression
with h5py.File('test.h5', 'r') as f:
    data = f['data']
    print(data[()])
    print(data._filters)
    print(data.id.get_create_plist().get_nfilters())
    print(data.id.get_create_plist().get_filter(0))

[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
{'32001': (2, 2, 8, 400, 4, 1, 1)}
1
(32001, 1, (2, 2, 8, 400, 4, 1, 1), b'blosc')

h5dump shows filter with compression:

HDF5 "test.h5" {
GROUP "/" {
   DATASET "data" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SIMPLE { ( 100 ) / ( 100 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 50 )
         SIZE 184 (4.348:1 COMPRESSION)
      }
      FILTERS {
         USER_DEFINED_FILTER {
            FILTER_ID 32001
            COMMENT blosc
            PARAMS { 2 2 8 400 4 1 1 }
         }
      }

But this (silently) does nothing (I think this is normal h5py/hdf5 behaviour if a filter is not applicable for some reason) although the filter is reported everywhere:

# change this to some erroneous value
compression_opts=(0,0,0,0,10,1,1))

Also note the clevel related output:

`clevel` parameter must be between 0 and 9!
`clevel` parameter must be between 0 and 9!
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
{'32001': (2, 2, 8, 400, 10, 1, 1)}
1
(32001, 1, (2, 2, 8, 400, 10, 1, 1), b'blosc')

In the h5dump, we can see that the filter was not applied (no compression), although it is added. Bug or feature?

HDF5 "test.h5" {
GROUP "/" {
   DATASET "data" {
      DATATYPE  H5T_STD_I64LE
      DATASPACE  SIMPLE { ( 100 ) / ( 100 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 50 )
         SIZE 800 (1.000:1 COMPRESSION)
      }
      FILTERS {
         USER_DEFINED_FILTER {
            FILTER_ID 32001
            COMMENT blosc
            PARAMS { 2 2 8 400 10 1 1 }
         }
      }
}

The dataset is reported to have Blosc filter applied via h5py and also via netCDF4 (which also reports the wrong clevel of 10).

with nc.Dataset("test.h5") as ds:
    print(ds["data"][:])
    print(ds["data"].filters())
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
 96 97 98 99]
{'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': {'compressor': 'blosc_lz4', 'shuffle': 1}, 'shuffle': False, 'complevel': 10, 'fletcher32': False}

We get these kind of warnings (see above) for wrong clevel and shuffle, but we do not get this if we have compression out of range. It just silently does not apply the filter.

Any thoughts on that? How can we make sure to not use any problematic settings?

Would it be possible to do something like this:

import hdf5plugin

filter = hdf5plugin.from_id(32001, opts=(0, 0, 0, 0, 4, 1, 1))
f.create_dataset(
        'data', 
        data=numpy.arange(100), 
        chunks=(50,), 
        **filter)

[t20100]

In the h5dump, we can see that the filter was not applied (no compression), although it is added. Bug or feature?

h5py sets the compression filters as "optional", so this is the libhdf5 & h5py high-level behavior (see https://docs.h5py.org/en/stable/high/dataset.html#custom-compression-filters). I expect you could change it by using h5py low-level API that wraps libhdf5 API directly, but that would be a lot more code.

How can we make sure to not use any problematic settings?

This is one of the goal of hdf5plugin filter classes: provide documentation, allow to write human-readable code an provide validation of user inputs.

It works already nicely using the advertised approach using either compression=hdf5plugin.LZ4() or via **hdf5plugin.Blosc().

Why not using it then?

Now, for conciseness, I want to be able to directly use it like this:
compression = 320001 # blosc
compression_opts = (0, 0, 0, 0, 4, 1, 1)

For conciseness of the code, I'm not sure you actually gain much:

filter = hdf5plugin.from_id(32001, opts=(0, 0, 0, 0, 4, 1, 1))
# vs
filter = hdf5plugin.Blosc(cname="lz4", clevel=4, shuffle=1)
filter = hdf5plugin.Blosc("lz4", 4, 1)
filter = hdf5plugin.Blosc(clevel=4)

However, such function could also be used to read-back hdf5 dataset compression filter options in order to provide explicitly the used options (would require to add a way to get the options, e.g., filter.cname and would be nice to provide __repr__ as well).
This looks to be simple but for the Sperr, SZ, SZ3 and ZFP filters which have more complex compression_opts.

[kmuehlbauer]

Thanks @t20100 for the quick reply. Much appreciated.

h5py sets the compression filters as "optional", so this is the libhdf5 & h5py high-level behavior (see https://docs.h5py.org/en/stable/high/dataset.html#custom-compression-filters). I expect you could change it by using h5py low-level API that wraps libhdf5 API directly, but that would be a lot more code.

I'm familiar with that. But I really do not understand the idea behind.

Citing from the docs:

The underlying implementation of the compression filter will have the H5Z_FLAG_OPTIONAL flag set. This indicates that if the compression filter doesn’t compress a block while writing, no error will be thrown. The filter will then be skipped when subsequently reading the block.

I request data to be compressed, the compressor raises an error because of wrong parameters (we only see this in the stdout/stderr) and still the write succeeds but without added compression. The filter is mentioned in the dataset. My main concern is that people THINK they have done everything right, but actually this is wrong.

To understand correctly, there is nothing what can be done in hdf5plugin to prevent the above case by raising an error?

Why not using it then?

I do, but under the hood. At least if compression is a string. We get eg "blosc_lz4" from xarray. I need to split the string, get the blosc filter from hdf5plugin (.get_filters("blosc"), initialize it with "lz4" and anything meaningful I find in compression_opts.

However, such function could also be used to read-back hdf5 dataset compression filter options in order to provide explicitly the used options (would require to add a way to get the options, e.g., filter.cname and would be nice to provide __repr__ as well).

That's where I've originally started. I need to give back something meaningful in h5netcdf.Variable.compression and h5netcdf.Variable.compression_opts. Initially I wanted to roundtrip what I got back from h5py (32001 and (0, 0, 0, 0, 4, 1, 1)). This works. But then, if I allow that, people could use wrong parameters, which I, again, need to check. Otherwise things might really go wrong (no compression applied etc. etc.)

[t20100]

To understand correctly, there is nothing what can be done in hdf5plugin to prevent the above case by raising an error?

No, this is set by h5py when creating a dataset: https://github.com/h5py/h5py/blob/c32aae5e3f4c78a1a49542f6196863abef719097/h5py/_hl/filters.py#L299

people could use wrong parameters

Yes, compression_opts is really error prone.

Makes sense to me to add a way to convert compression&compression_opt to the corresponding filter class instance.
However it will only supports the compression that are provided by hdf5plugin.

[kmuehlbauer]

To understand correctly, there is nothing what can be done in hdf5plugin to prevent the above case by raising an error?

No, this is set by h5py when creating a dataset: https://github.com/h5py/h5py/blob/c32aae5e3f4c78a1a49542f6196863abef719097/h5py/_hl/filters.py#L299

Sorry for my ignorance, at which point in time is that message generated (clevel parameter must be between 0 and 9!)? I thought this is from hdf5plugin python, but probably this is from whatever source code the filter is written in? I only want to estimate, if I'd be able to fix this myself, or if I need to be patient.

Yes, compression_opts is really error prone.

Makes sense to me to add a way to convert compression&compression_opt to the corresponding filter class instance. However it will only supports the compression that are provided by hdf5plugin.

This would really be amazing! Thanks for all the efforts put into this package already! It already makes life much easier.

Should I open a feature request for the above conversion functionality? I think my usage question is fully answered!

[t20100]

at which point in time is that message generated

It's printed by c-blosc during compression:

hdf5plugin/lib/c-blosc/blosc/blosc.c

Lines 1114 to 1116 in 077d3f7

	if (clevel < 0 || clevel > 9) {
	fprintf(stderr, "`clevel` parameter must be between 0 and 9!\n");
	return -10;

hdf5plugin raises an exception when the class is instantiated:

hdf5plugin/src/hdf5plugin/_filters.py

Lines 200 to 201 in 077d3f7

	if not 0 <= clevel <= 9:
	raise ValueError("clevel must be in the range [0, 9]")

Should I open a feature request for the above conversion functionality?

Yes. It may take a bit of time to finalize it, but from a first look at it, it does not seems too complex.

[t20100]

From checking how to handle this, I started PR #366 (only checked bitshuffle and blosc for now).

[kmuehlbauer]

@t20100 Thanks for moving that quickly. Instead of creating a feature request, I rewrote the issue title. So it is all contained here.