HDF5 in NeuroML2

[pgleeson]

An HDF5 based equivalent for NeuroML with efficient storage/compression of positions/connections in a network would be a good alternative to serialising in XML. This should be well supported by the various NeuroML APIs so use of the format is transparent for users.

This is being worked on in:

• libNeuroML
• org.neuroml.model

Examples of HDF5 files in this format are here and here.

Some idea of current/future support:

[ccluri]

Is there some documentation regarding the HDF5 structure here?
Or is this does like xdmf-paraview kind - where the NML2 points to the the HDF5 a dataset array?

Would you consider linking via NSDF which is also HDF5 based?

[pgleeson]

@ccluri The documentation at the moment doesn't go too far beyond the examples above I'm afraid... The priority until now has been getting a format we can read and write in both Python and Java.

What is the current status of NSDF? Are there other tools for it besides the Python API? Are there examples storing network structure?

NSDF compatibility is certainly something to consider for the future, but for now it's best to keep the format as simple as possible, try to keep the apis for it as compatible as possible and see what optimisations are possible. One which is in the python api is allowing libNeuroML to create a numpy array for connectivity data/positions in memory (as opposed to creating a tree of Python objects) and then just putting that as the dataset into the HDF5 file. Makes it much quicker to read/write the info. Is that a scenario you think would be supported by NSDF?