Move toward more efficient metadata storage

[lfrank]

One challenge with working with large files of any sort is the time required to parse through them to find the data that are needed. Typically users will want particular parts of the data and do a search through the metadata to identify those chunks. If the file structure can be changed to make those searches more efficient, that would be useful. One option already suggested by Andrew and Oliver is to store metadata and large datasets in separate but linked files, and I would suggest we seriously consider implementing that or a mechanism that accomplishes the same thing relatively soon if other priorities permit.

[ajtritt]

@lfrank Thanks for putting this up. To get a better idea of what we want to do, I am going to start a list of such example queries. Please correct them as you see fit to better clarify what we want to do.

• Get all voltage traces from brain region 'CA1'
• Get spike trains for all cells during sleep epochs

[neuromusic]

this is basically the infrastructure that @dyf et all have implemented in the allensdk, by computing various features of each NWB file and storing them in a pandas dataframe that can queried to ID relevant NWB files https://github.com/AllenInstitute/allensdk

it's missing the API where you go straight to e.g. traces

I'll add my dream query:

• from BOTH ophys & ephys NWB files, get mean responses of all neurons* in region VISp in response to each natural image with ID in set (0,1,2,3,4)

[neuromusic]

note: another backend (say PostgreSQL) along with a concept in the schema that groups together sessions (say NWBSessionSet) would make this much easier, as you could run such queries at the NWBSessionSet level.

This approach is similar to what I implemented in grad school: https://github.com/gentnerlab/django-broab

Basically, each data object (say each spike train and each behavioral event) gets it's own entry in the database & you can run arbitrary queries with SQL joins.

https://github.com/gentnerlab/django-broab/blob/master/broab/models.py

[neuromusic]

I'd be so bold as to say "If you want performance querying data across files/sessions/experiments, HDF5 is simply a bad solution."

HDF5 was selected for NWB due to it's utility as an exchange format... any of the other backends that improve performance will undermine this priority.

[oruebel]

@neuromusic I agree that if you want fast query performance across large collections of files then we'll likely need some form of acceleration structure, whether it be a database index or some other form of query structure if only because the time to simply open files will kill performance. However, I would say that this is not really a problem of HDF5 but rather file-based solutions in general. Data exchange and query across files have very different requirements so it is not surprising that one storage solution may not serve all needs optimally. In addition to "pure" alternative backends (e.g., all data be routed to a database), I think as we explore this space we'll likely find that we probably want some hybrid solutions, e.g., large binary data blocks stored in HDF5 and metadata and data statistics stored in a database for query. Designing these kind of systems is a large effort in itself. My feeling right now is that this probably something we want to explore but it may not be something we want NWB to "officially" standardize at this point.

@lfrank I think rather than a full solution, what you have suggest is to take a first step to help make the pure HDF5 solution work more efficiently with queries by separating data into 1) small HDF5 files with metadata and 2) large auxiliary HDF5 files that are linked to the metadata files. What this would accomplish is that you at least would only need to touch the smaller metadata files when querying data but still have an easy way to get to the raw data blocks.

@ajtritt @neuromusic @lfrank I think as part of this ticket we probably want to focus on the proposal that @lfrank has raised and have the discussion about alternative backends to accelerate queries in a separate thread.

In terms of implementing the proposal, I think this really only affects the data write as the data read can already handle external links. As such, what would be needed are ways to write some parts of an NWBContainer to different locations. I could imagine different solution to accomplish this:

1. We could enhance the HDF/IO backend to allow us to write individual objects and provide a list of the fields it should write. Those fields could then be replaced by the corrsonding HDF5 datasets so that in the next write they would become external links. This would split the write into two main steps: 1) write the large data portions that should go to external files first and then 2) write the remaining data to a single metadata file.

2. We could design some ability to provide write hints for individual fields of an NWBContainer.

In general, the main complication I see is for attributes that are associated with an external data block. I'm not sure if we can seperate those attributes from the data block

[neuromusic]

"My feeling right now is that this probably something we want to explore but it may not be something we want NWB to "officially" standardize at this point."

agreed.

" what would be needed are ways to write some parts of an NWBContainer to different locations. "

We've done this with ainwb in a somewhat hacked way.

1. write a large NWB file w/ only LFP data (I'll call it lfp.nwb)
2. write a smaller NWB file w/ spikes & provide external link to LFP data in lfp.nwb

I'm suspicious that it would solve @lfrank's concern, but formalizing this process in pynwb would be awesome nonetheless. The proposed mechanism 1 of setting this up through HDF5IO sounds pretty intuitive to me.

[oruebel]

Just on a side-note, the following notebook also shows a "hacked" way for doing this:

https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/docs/notebooks/convert-crcns-ret-1-meisterlab.ipynb

In the notebook I used h5py to write stimulus data to external HDF5 files and then used PyNWB API to create external links to those files. This really just a workaround and I agree that having a formal mechanism for doing this sort of thing from PyNWB would be useful.

[rly]

We are (finally) experimenting with more efficient metadata storage and modular data storage with https://github.com/NeurodataWithoutBorders/lindi