ChunkCodecs.jl and JLD2.jl compat tests failing with HDF5_jll v2

[nhz2]

Here is an example of a file that cannot be read with HDF5_jll v2 but can be read with HDF5_jll v1.14
I'm not sure if this file is a valid hdf5 file, since it was not created by the actual hdf5 library but was created by https://github.com/JuliaIO/ChunkCodecs.jl/blob/a65bf6721f01a4aeb5b68b8309a82760f58867dd/test/hdf5_helpers.jl#L216-L217.

bad.h5.tar.gz

Here is a MWE using JLD2:

CC: @eschnett @JonasIsensee

using JLD2
using HDF5
using Pkg
pkg"st -m"
versioninfo()
save("example.jld2", "large_array", zeros(100); compress=true)
h5open("example.jld2", "r")

Details

Status `/tmp/jl_nH9TRK/Manifest.toml`
  [0b6fb165] ChunkCodecCore v1.0.1
  [4c0bbee4] ChunkCodecLibZlib v1.0.0
  [55437552] ChunkCodecLibZstd v1.0.0
  [34da2185] Compat v4.18.1
  [5789e2e9] FileIO v1.18.0
  [f67ccb44] HDF5 v0.18.0 `https://github.com/JuliaIO/HDF5.jl.git#497825b`
  [076d061b] HashArrayMappedTries v0.2.0
  [033835bb] JLD2 v0.6.4
  [692b3bcd] JLLWrappers v1.7.1
  [3da0fdf6] MPIPreferences v0.1.12
  [1914dd2f] MacroTools v0.5.16
  [bac558e1] OrderedCollections v1.8.1
  [aea7be01] PrecompileTools v1.3.3
  [21216c6a] Preferences v1.5.2
  [ae029012] Requires v1.3.1
  [7e506255] ScopedValues v1.6.0
  [0234f1f7] HDF5_jll v2.1.2+0
  [e33a78d0] Hwloc_jll v2.13.0+1
  [94ce4f54] Libiconv_jll v1.18.0+0
  [b5ada748] MPIABI_jll v0.1.3+0
  [7cb0a576] MPICH_jll v5.0.1+0
  [f1f71cc9] MPItrampoline_jll v5.5.5+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+3
  [fe0851c0] OpenMPI_jll v5.0.10+0
⌅ [02c8fc9c] XML2_jll v2.13.9+0
  [a65dc6b1] Xorg_libpciaccess_jll v0.18.1+0
  [3161d3a3] Zstd_jll v1.5.7+1
⌅ [2b3700d1] aws_c_auth_jll v0.9.6+0
  [70f11efc] aws_c_cal_jll v0.9.13+0
  [73048d1d] aws_c_common_jll v0.12.6+0
  [73a04cd5] aws_c_compression_jll v0.3.2+0
  [3254fc65] aws_c_http_jll v0.10.13+0
  [13c41daa] aws_c_io_jll v0.26.3+0
⌅ [bd1f34fb] aws_c_s3_jll v0.11.5+0
  [1282aa60] aws_c_sdkutils_jll v0.2.4+1
  [b2a88e68] aws_checksums_jll v0.2.10+0
  [c4b69c83] dlfcn_win32_jll v1.4.2+0
  [477f73a3] libaec_jll v1.1.6+0
  [9aeb927a] mpif_jll v0.1.5+0
  [cddc5d3d] s2n_tls_jll v1.7.2+0
  [0dad84c5] ArgTools v1.1.2
  [56f22d72] Artifacts v1.11.0
  [2a0f44e3] Base64 v1.11.0
  [ade2ca70] Dates v1.11.0
  [f43a241f] Downloads v1.7.0
  [7b1f6079] FileWatching v1.11.0
  [ac6e5ff7] JuliaSyntaxHighlighting v1.12.0
  [4af54fe1] LazyArtifacts v1.11.0
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2 v1.11.0
  [8f399da3] Libdl v1.11.0
  [56ddb016] Logging v1.11.0
  [d6f4376e] Markdown v1.11.0
  [a63ad114] Mmap v1.11.0
  [ca575930] NetworkOptions v1.3.0
  [44cfe95a] Pkg v1.12.1
  [de0858da] Printf v1.11.0
  [9a3f8284] Random v1.11.0
  [ea8e919c] SHA v0.7.0
  [f489334b] StyledStrings v1.11.0
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [cf7118a7] UUIDs v1.11.0
  [4ec0a83e] Unicode v1.11.0
  [e66e0078] CompilerSupportLibraries_jll v1.3.0+1
  [deac9b47] LibCURL_jll v8.15.0+0
  [e37daf67] LibGit2_jll v1.9.0+0
  [29816b5a] LibSSH2_jll v1.11.3+1
  [14a3606d] MozillaCACerts_jll v2025.11.4
  [458c3c95] OpenSSL_jll v3.5.4+0
  [83775a58] Zlib_jll v1.3.1+2
  [8e850ede] nghttp2_jll v1.64.0+1
  [3f19e933] p7zip_jll v17.7.0+0
Info Packages marked with ⌅ have new versions available but compatibility constraints restrict them from upgrading. To see why use `status --outdated -m`

Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 16 × Intel(R) Core(TM) i7-10700KF CPU @ 3.80GHz
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, skylake)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 16 virtual cores)
Environment:
  JULIA_PKG_DEVDIR = /home/nathan/juliadev
  JULIA_PKG_USE_CLI_GIT = true

Error showing value of type HDF5.File:

SYSTEM (REPL): showing an error caused an error
ERROR: 1-element ExceptionStack:
HDF5.API.H5Error: Error opening object //large_array
libhdf5 Stacktrace:
  [1] H5D__chunk_set_sizes: Dataset/Bad value
      stored chunk dimension encoding length does not match value calculated from chunk dimensions
   ⋮
Stacktrace:
  [1] macro expansion
    @ ~/.julia/packages/HDF5/oqkK5/src/api/error.jl:18 [inlined]
  [2] h5l_iterate(group_id::HDF5.File, idx_type::HDF5.API.H5_index_t, order::HDF5.API.H5_iter_order_t, idx::Base.RefValue{…}, op::Ptr{…}, op_data::Tuple{…})
    @ HDF5.API ~/.julia/packages/HDF5/oqkK5/src/api/functions.jl:2167
  [3] h5l_iterate(f::Any, group_id::HDF5.File, idx_type::HDF5.API.H5_index_t, order::HDF5.API.H5_iter_order_t, idx::Int64)
    @ HDF5.API ~/.julia/packages/HDF5/oqkK5/src/api/helpers.jl:501
  [4] h5l_iterate
    @ ~/.julia/packages/HDF5/oqkK5/src/api/helpers.jl:497 [inlined]
  [5] _show_tree(io::IOContext{IOBuffer}, obj::HDF5.File, indent::String; attributes::Bool, depth::Int64)
    @ HDF5 ~/.julia/packages/HDF5/oqkK5/src/show.jl:329
  [6] show_tree(io::IOContext{REPL.LimitIO{Base.TTY}}, obj::HDF5.File; kws::@Kwargs{})
    @ HDF5 ~/.julia/packages/HDF5/oqkK5/src/show.jl:348
  [7] show_tree
    @ ~/.julia/packages/HDF5/oqkK5/src/show.jl:348 [inlined]
  [8] show(io::IOContext{REPL.LimitIO{Base.TTY}}, ::MIME{Symbol("text/plain")}, obj::HDF5.File)
    @ HDF5 ~/.julia/packages/HDF5/oqkK5/src/show.jl:255
  [9] show_repl(io::IO, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:548
 [10] show_limited(io::IO, mime::MIME, x::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:517
 [11] (::REPL.var"#display##0#display##1"{REPL.REPLDisplay{…}, MIME{…}, Base.RefValue{…}})(io::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:540
 [12] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:833
 [13] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:526
 [14] display
    @ ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:546 [inlined]
 [15] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:340
 [16] (::REPL.var"#print_response##2#print_response##3")()
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:590
 [17] repl_backend_loop(backend::REPL.REPLBackend, get_module::Function)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:445
 [18] start_repl_backend(backend::REPL.REPLBackend, consumer::Any; get_module::Function)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:427
 [19] start_repl_backend
    @ ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:424 [inlined]
 [20] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool, backend::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:653
 [21] run_repl(repl::REPL.AbstractREPL, consumer::Any)
    @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:639
 [22] run_std_repl(REPL::Module, quiet::Bool, banner::Symbol, history_file::Bool)
    @ Base ./client.jl:478
 [23] run_main_repl(interactive::Bool, quiet::Bool, banner::Symbol, history_file::Bool)
    @ Base ./client.jl:499
 [24] repl_main
    @ ./client.jl:586 [inlined]
 [25] _start()
    @ Base ./client.jl:561
Stacktrace:
 [1] print_response(errio::IO, response::Any, backend::Union{…}, show_value::Bool, have_color::Bool, specialdisplay::Union{…})
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:599
 [2] (::REPL.var"#print_response##0#print_response##1"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:558
 [3] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:833
 [4] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:556
 [5] (::REPL.var"#do_respond#73"{…})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:1192
 [6] run_interface(terminal::Base.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
   @ REPL.LineEdit ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/LineEdit.jl:2854
 [7] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:1651
 [8] (::REPL.var"#61#62"{REPL.LineEditREPL, REPL.REPLBackendRef})()
   @ REPL ~/packages/julias/julia-1.12.5/share/julia/stdlib/v1.12/REPL/src/REPL.jl:650
Some type information was truncated. Use `show(err)` to see complete types.

[eschnett]

I cannot reproduce the problem:

julia> versioninfo()
Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 12 × Apple M3 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 6 virtual cores)

julia> save("example.jld2", "large_array", zeros(100); compress=true)

julia> h5open("example.jld2", "r")
🗂️ HDF5.File: (read-only) example.jld2
└─ 🔢 large_array

Your bad.h5 file is also fine for me, both with Julia and with h5ls -r and h5dump.

[nhz2]

Which version of HDF5_jll are you using?

[eschnett]

I am using HDF5_jll v1.14.6+0... I see, you were using HDF5 2!

[eschnett]

I switch to using HDF5_jll 2.1.2 and I can now reproduce your problem:

julia> versioninfo()
Julia Version 1.12.5
Commit 5fe89b8ddc1 (2026-02-09 16:05 UTC)
Build Info:
  Official https://julialang.org release
Platform Info:
  OS: macOS (arm64-apple-darwin24.0.0)
  CPU: 12 × Apple M3 Pro
  WORD_SIZE: 64
  LLVM: libLLVM-18.1.7 (ORCJIT, apple-m3)
  GC: Built with stock GC
Threads: 1 default, 1 interactive, 1 GC (on 6 virtual cores)

julia> save("example.jld2", "large_array", zeros(100); compress=true)

julia> h5open("example.jld2", "r")
Error showing value of type HDF5.File:

SYSTEM (REPL): showing an error caused an error
ERROR: 1-element ExceptionStack:
HDF5.API.H5Error: Error opening object //large_array
libhdf5 Stacktrace:
  [1] H5D__chunk_set_sizes: Dataset/Bad value
      stored chunk dimension encoding length does not match value calculated from chunk dimensions

[JonasIsensee]

I have a local sandboxed claude setup with h5debugging tools, accessible h5 format spec and some hexdump tooling.
It identified and fixed the issue.

In short: v2 is more strict about the format spec. The chunk dimension encoding length i.e. the number of bytes used to encode the dimension size is stored. This was previously set to 8 bytes in JLD2 by default.
However, spec is more specific and actually requests that this be the minimal number of bytes to represent all dimension sizes as unsigned it. h5v2 appears to be the first version to actually assert this.

In your tests you will need to change
https://github.com/JuliaIO/ChunkCodecs.jl/blob/a65bf6721f01a4aeb5b68b8309a82760f58867dd/test/hdf5_helpers.jl#L324

and following accordingly to make it work.

[nhz2]

I'm not seeing where the spec says that. Ideally this is something that can be fixed in the HDF5 C library, because any change made to JLD2.jl can't fix the already written .jld2 files.

Here is what claude code found for me in the spec:

Details

1. Spec text for "Dimension Size Encoded Length" (v3 & v4 format docs)

docs/doxygen/dox/H5.format.4.0.dox:8106-8108 (and v3 at line 7930):

▎ "This is the size in bytes used to encode Dimension Size."

That's it. No requirement that it be the minimum.

2. Compare to other variable-size fields in the same spec

The spec is explicit elsewhere when it wants minimal encoding:

• v2 B-tree Number of Records in Child Node (H5.format.4.0.dox:1302-1307):
  "the least number of bytes required to encode the maximum number of records …
  For example, if the maximum number of child records is 123, one byte will be
  used"
• Fractal heap Offset (:3719): "the minimum number of bytes necessary to
  encode the Maximum Heap Size value"
• Fractal heap Length (:3726-3729): "the minimum number of bytes needed to
  encode that value"

So the spec authors knew how to mandate minimal encoding when they wanted it —
and they didn't for Dimension Size Encoded Length.

3. H5VM_limit_enc_size() definition

src/H5VMprivate.h:447-451:
static inline unsigned H5_ATTR_UNUSED
H5VM_limit_enc_size(uint64_t limit)
{
return (H5VM_log2_gen(limit) / 8) + 1;
}
Computes the minimum bytes needed to encode values from 0 to limit. Used for
v2 B-tree record counts, fractal heap offsets, etc. — fields where the spec
text explicitly mandates minimal encoding.

4. v4 chunked layout encode/decode

Decode (src/H5Olayout.c:344-365): reads enc_bytes_per_dim byte, validates
1..8, then UINT64DECODE_VAR for each dim. Per se, this accepts any 1..8.

Encode (src/H5Olayout.c:658-664): writes whatever
mesg->u.chunk.enc_bytes_per_dim happens to hold.

5. The strict check in H5D__chunk_set_sizes — src/H5Dchunk.c:840-862

/* Compute number of bytes to use for encoding chunk dimensions /
max_enc_bytes_per_dim = 0;
for (u = 0; u < ndims; u++) {
enc_bytes_per_dim = (H5VM_log2_gen(dim[u]) + 8) / 8;
if (enc_bytes_per_dim > max_enc_bytes_per_dim)
max_enc_bytes_per_dim = enc_bytes_per_dim;
}
…
/ Set encoding length in layout */
if (dset->shared->layout.u.chunk.enc_bytes_per_dim) {
if (dset->shared->layout.u.chunk.enc_bytes_per_dim !=
max_enc_bytes_per_dim)
HGOTO_ERROR(H5E_DATASET, H5E_BADVALUE, FAIL,
"stored chunk dimension encoding length does not match
value calculated from chunk "
"dimensions");
}
else
dset->shared->layout.u.chunk.enc_bytes_per_dim = max_enc_bytes_per_dim;

This is reached on dataset open after layout decode, so any file whose stored
enc_bytes_per_dim is larger than the minimum needed to encode its largest
chunk dimension is now rejected.

6. Git history — when did this strict check appear?

git log -S "stored chunk dimension encoding length does not match" returns
exactly one commit:

1093a18c18 Add support for 64 bit chunks, including 64 bit chunk dimensions
(#5965)
Neil Fortner, 2025-11-04

Diff context shows the old logic was simply:
dset->shared->layout.u.chunk.enc_bytes_per_dim = max_enc_bytes_per_dim; /*
unconditional overwrite */

The new logic conditions on whether the field was already set (which it now
is, after a successful v4 layout decode), and errors out on mismatch.

Before this commit, the library happily accepted any enc_bytes_per_dim in 1..8
— it used the stored value to do UINT64DECODE_VAR (which works regardless of
padding), then overwrote the field with the recomputed minimum and moved on.
Files with non-minimal encoding round-tripped fine on read.

Conclusion

• The spec does NOT require minimal encoding for the v4 chunked layout
  Dimension Size Encoded Length field. It only says "the size in bytes used to
  encode Dimension Size" — and the spec is demonstrably explicit when it does
  want minimal encoding (v2 B-tree record counts, fractal heap offsets/lengths).
• The reference implementation has, since commit 1093a18c18 (2025-11-04, the
  64-bit chunks PR #5965), started rejecting files where the stored encoding
  length is not exactly the minimum required for the actual dimension values,
  with the error "stored chunk dimension encoding length does not match value
  calculated from chunk dimensions".
• This is a regression in tolerance: prior versions of the library silently
  accepted (and effectively normalized) non-minimal encodings. A spec-conformant
  writer that, say, always wrote enc_bytes_per_dim = 8 would have produced
  files readable by older HDF5 but rejected by current develop. Worth raising
  upstream — either the spec should be tightened to require minimal encoding
  (matching the v2 B-tree language), or the strict check should be relaxed back
  to a normalization.

[mkitti]

This seems like a bug to report upstream.

[mkitti]

Has anyone reported this upstream yet?

[nhz2]

It looks like the same issue has been reported in HDFGroup/hdf5#6409