Vary compression factor "blosc" does not make a difference above zero, is this expected?

[StefanPofahl]

Hi,
thank you for this nice package!
Today I saw that vary blosc does not seem to make a difference,
two weeks ago vary blosc had an impact.

Regards,

Stefan

P.S.:
Here is my code:
using HDF5, H5Zblosc, Printf
# ---
DIR_Export = raw"C:\tmp\tmp_test_H5Zblosc"
n_exp = 6
n_columns = 6
# ---
if ~isdir(DIR_Export)
mkpath(DIR_Export)
end
# --- calcualtions:
n_raws = Int(exp10(n_exp))
data_array = rand(Float64, (n_raws, n_columns))
# --- write array to file:
for i_compr = 0:5
FN_HDF5_Compr =
joinpath(DIR_Export, @sprintf("compr#%i_", i_compr) * "one_chunk_over_all.hdf5")
h5open(FN_HDF5_Compr, "w") do fid
fid["data_array", chunk = (n_raws, n_columns), blosc = i_compr] = data_array
end
end

[musm]

Hmm, we haven't touched the blosc package in two weeks.

By the way, you can use ``` to wrap a whole code block instead of wrapping each line with a single `

[musm]

Could you clarify by what you mean when you say it has no impact?

[StefanPofahl]

In a previous experiment around three weeks ago, the file size was dependent in the parameter `blosc`, but now the file size changes only between `blosc=0` and `blosc=1` Am Mi., 18. Mai 2022 um 19:40 Uhr schrieb Mustafa M < ***@***.***>:

…

Could you clarify by what you mean when you say it has no impact? — Reply to this email directly, view it on GitHub <#940 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APREBEYQAN2NE64OSTT7YHDVKUTO5ANCNFSM5WH3GPVA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[mkitti]

Reformatting to make this easier to read:

using HDF5, H5Zblosc, Printf
# ---

DIR_Export = raw"C:\tmp\tmp_test_H5Zblosc"
n_exp = 6
n_columns = 6
# ---
if ~isdir(DIR_Export)
mkpath(DIR_Export)
end

# --- calculations:
n_raws = Int(exp10(n_exp))
data_array = rand(Float64, (n_raws, n_columns))

# --- write array to file:
for i_compr = 0:5
    FN_HDF5_Compr =
    joinpath(DIR_Export, @sprintf("compr#%i_", i_compr) * "one_chunk_over_all.hdf5")
    h5open(FN_HDF5_Compr, "w") do fid
        fid["data_array", chunk = (n_raws, n_columns), blosc = i_compr] = data_array
    end
end

[mkitti]

This is using the old blosc property rather than the filter property. I wonder if something happened with set_blosc!.

[simonbyrne]

This appears to be an issue with Blosc:

julia> data_array = rand(100,100);

julia> [length(Blosc.compress(data_array; level=i)) for i = 0:9]
10-element Vector{Int64}:
 80016
 70103
 70103
 70103
 70103
 70103
 70103
 70103
 70103
 70103

[simonbyrne]

The change appears to be due to differences in the Blosc binaries:

(blosc) pkg> add Blosc_jll@1.14
   Resolving package versions...
   Installed Blosc_jll ─ v1.14.3+1
  Downloaded artifact: Blosc
    Updating `~/src/HDF5.jl/test/blosc/Project.toml`
⌃ [0b7ba130] + Blosc_jll v1.14.3+1
    Updating `~/src/HDF5.jl/test/blosc/Manifest.toml`
⌃ [0b7ba130] ↓ Blosc_jll v1.21.1+0 ⇒ v1.14.3+1
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling project...
  2 dependencies successfully precompiled in 1 seconds. 7 already precompiled.

julia> using Blosc

julia> data_array = rand(1:10,100,100);

julia> [length(Blosc.compress(data_array; level=i)) for i = 0:9]
10-element Vector{Int64}:
 80016
 10395
 10395
 10395
 10395
 10395
  9393
  9393
  9393
  9393

(blosc) pkg> add Blosc_jll@1.21
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
    Updating `~/src/HDF5.jl/test/blosc/Project.toml`
  [0b7ba130] ↑ Blosc_jll v1.14.3+1 ⇒ v1.21.1+0
    Updating `~/src/HDF5.jl/test/blosc/Manifest.toml`
  [0b7ba130] ↑ Blosc_jll v1.14.3+1 ⇒ v1.21.1+0
Precompiling project...
  2 dependencies successfully precompiled in 1 seconds. 7 already precompiled.

julia> using Blosc

julia> data_array = rand(1:10,100,100);

julia> [length(Blosc.compress(data_array; level=i)) for i = 0:9]
10-element Vector{Int64}:
 80016
 10409
 10409
 10409
 10409
 10409
 10409
 10409
 10409
 10409

[mkitti]

Hmm, can you extend the levels further?

[simonbyrne]

according to Blosc.jl, the levels only go from 0:9

[mkitti]

Oh right, they map their levels. I'm guessing we are using the default compressor too, which is blosclz. Maybe try zstd?

[mkitti]

With the zstd compressor, things seem to work as expected.

julia> using Blosc

julia> data_array = rand(1:10, 100, 100);

julia> Blosc.set_compressor("zstd")
5

julia> [length(Blosc.compress(data_array; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
  4525
  4593
  4802
  4789
  4789
  4789
  4457
  4449
  4457

Resetting to the original blosclz compressor reproduces your findings.

julia> Blosc.set_compressor("blosclz")
0

julia> [length(Blosc.compress(data_array; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
 10409
 10409
 10409
 10409
 10409
 10409
 10409
 10409
 10409

Trying the other compressors

julia> Blosc.compressors()
5-element Vector{SubString{String}}:
 "blosclz"
 "lz4"
 "lz4hc"
 "zlib"
 "zstd"

julia> Blosc.set_compressor("lz4")
1

julia> [length(Blosc.compress(data_array; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
 10365
 10375
 10351
 10332
 10247
 10051
  9840
  9507
  9019

julia> Blosc.set_compressor("lz4hc")
2

julia> [length(Blosc.compress(data_array; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
  8633
  8633
  8617
  8616
  8616
  8616
  8616
  8616
  8616

julia> Blosc.set_compressor("zlib")
4

julia> [length(Blosc.compress(data_array; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
  5586
  5611
  5621
  5371
  5358
  5358
  5358
  5358
  5358

[mkitti]

Returning to blosclz4, this is highly dependent on data's entropy. It does not use a dictionary or encoding.

julia> Blosc.set_compressor("blosclz")
0

julia> data_array2 = round.(Int, randn(100,100) .+ 5);

julia> [length(Blosc.compress(data_array2; level=i)) for i in 0:9]
10-element Vector{Int64}:
 80016
 10409
 10409
  7377
  7377
  7377
  7377
  7377
  7377
  7377