HDF5.name performance issue

[matthijscox]

I discovered a performance issue with HDF5.name and the underlying HDF5.API.h5i_get_name in MAT.jl: JuliaIO/MAT.jl#237 (comment)

Somehow retrieving names takes exponentially longer for linearly growing HDF5 file sizes. Is there something we're doing wrong, or should I look into the root cause inside HDF5.jl ?

[mkitti]

Just to make sure, you are only trying to retrieve one name, right? Or are we scanning over names? If so, maybe we should use an iterator to loop over names

I am not sure yet how MAT.jl is expanding the file. Is it creating and deleting datasets on each iteration?

[matthijscox]

We were reading the name for every HDF5.Group we encountered. In an existing file. We're not writing while reading.

I think I fixed it now in MAT.jl, by only checking the name when needed: JuliaIO/MAT.jl#238.

Still curious that retrieving the name is so expensive for large files. Does it read the file from the start or something every time I request a group's name? HDF5.read_attribute doesn't have this performance problem for example.

[mkitti]

I believe this may be an upstream issue. I would be interested in seeing what upstream says at https://github.com/HDFGroup/hdf5 or https://forum.hdfgroup.org/

[matthijscox]

I see a reference here: https://forum.hdfgroup.org/t/h5i-get-name-call-is-very-slow-for-hdf5-file-5-gb/8246/6. There the advice was also to not use h5i_get_name :)

I fear that I'd have to make a low-level minimum working example to explain the issue to the HDF5 maintainers, probably to find out I shouldn't have called h5i_get_name so often. That's how I resolved it in MAT.jl now. I'm okay to close this issue. At least we're aware of the performance issue.

[mkitti]

@gheber's comments in that forum link bring quite a bit of insight.

But in that case, calling H5Iget_name is overkill, unless you are in a corner of your application where all you have is a handle (hid_t) and you would like to recover a pathname.
It’s overkill, because, as you know already, it’s a very expensive operation, and to catalog all pathnames in the file you could just use H5Ovisit and conveniently pick up the pathnames along the way.

This suggests we should build better interfaces around H5Ovisit3 or H5Literate2() when people want to walk or iterate through the HDF5 hierarchy.

• H5Ovisit3
• H5Literate2

[mkitti]

H5Literate2 is wrapped by keys(::HDF5.Group).

What Gerd is saying is that rather than passing the group handle to a function and retrieving its name via HDF5.name, you should iterate over the keys in the parent group and pass the group name and the parent's reference to a function that needs the name of the group.