Paper: NeuroConv. Streamlining Neurophysiology Data Conversion to the NWB Standard - #1088
Conversation
|
Curvenote Preview
|
rowanc1
added
paper
Co-authored-by: Ben Dichter <ben.dichter@gmail.com> Co-authored-by: Luiz Tauffer <luiz_tauffer@hotmail.com>
|
Inviting reviewers: @gartavanis and bhaskargoyal24@gmail.com |
|
Dear authors, Congratulations on your submission regarding NeuroConv! The paper is well-structred, organized and easy to follow. It addresses the issue of diverse formats in neurophysiology data and enables easier adoption of the NWB standard. The paper does a good job as a high-level description that a beginner NeuroConv user would understand, while the package documentation provides more details for expert users. I will leave specific comments for the initial review inline. |
bhaskargoyal
left a comment
bhaskargoyal
left a comment
There was a problem hiding this comment.
Overall, this is a fantastic paper and a significant contribution. The work is exemplary, and the paper is written to a very high standard. Congratulations to the authors
| To address the challenges described in {ref}`sec:introduction` (see {ref}`tbl:nwb-challenges`), we developed [NeuroConv](https://neuroconv.readthedocs.io/en/stable/index.html), an open source library that automates the ingestion and conversion of neurophysiology data from diverse formats into NWB. This section describes in detail how NeuroConv's architecture addresses each of these challenges through a modular, extensible design that maintains both flexibility and ease of use. | ||
|
|
||
| ## Handling Format Diversity | ||
| The challenge of format diversity in neurophysiology extends beyond their sheer number (47 supported at the moment). Many formats, such as Neuralynx, have multiple versions, while others, like TIFF, exhibit significant internal variability in how labs use them. NeuroConv addresses this complexity through a unified architecture centered on the DataInterface abstraction. DataInterface is an abstract class for reading data, and each supported format has a dedicated DataInterface that encapsulates the format-specific logic for data reading, metadata extraction, and NWB conversion while presenting a consistent API to users. The DataInterface serves as the fundamental building block of NeuroConv, providing a standardized pathway from diverse source formats to NWB output. This abstraction enables users to work with any supported format using identical code patterns, regardless of the underlying format complexity or internal details. The minimal conversion pipeline is illustrated in {ref}`fig:assets/minimal_conversion_pipeline`: |
There was a problem hiding this comment.
This is a fantastic point and a hallmark of a well-integrated scientific software package. Highlighting the strategy of leveraging and contributing back to the ecosystem is a major strength and a great lesson for the SciPy community.
There was a problem hiding this comment.
Thanks a lot for your kind words.
| keywords: Neurodata Without Borders, NWB, Neurophysiology, Data standardization, Data conversion, DANDI, Python, Scientific software, Large-scale data | ||
| abstract: | | ||
| Modern neurophysiology generates increasingly complex, multimodal datasets that require standardized formats for effective sharing and reuse. The Neurodata Without Borders (NWB) format has emerged as a solution for data standardization, but data conversion remains a significant bottleneck due to format heterogeneity, metadata complexity, and required technical expertise. We present NeuroConv, an open-source Python library that automates the conversion of neurophysiology data from 47 distinct formats into NWB through a unified, modular architecture. Developed through collaboration with over 50 neurophysiology laboratories, NeuroConv addresses key challenges through three core components: format-specific DataInterfaces that abstract parsing complexity, multi-stream Converters that integrate heterogeneous data modalities, and optimized writing strategies for large-scale datasets including chunked operations and cloud-compatible storage. NeuroConv's design enables researchers to convert complex, multi-modal experimental sessions with minimal code while preserving critical metadata and temporal alignment across recording systems. By removing technical barriers NeuroConv thus advances the transformation of neurophysiology toward FAIR (Findable, Accessible, Interoperable, and Reusable) data practices, facilitating reproducible research and accelerating scientific discovery. | ||
| acknowledgments: | |
There was a problem hiding this comment.
Suggestion nit: While this may be a requirement of the template, it's slightly unconventional to have the acknowledgments at the very beginning of the paper. If the style guide allows, consider moving this section to the end, just before the references, which is a more traditional placement. If it's fixed by the template, please disregard this comment.
There was a problem hiding this comment.
In the preview it appears at the end:
Is this what you mean?
|
|
||
| :::{figure} assets/conversion_comparisons.png | ||
| :label: fig:assets/conversion_comparisons | ||
| This illustrates where NeuroConv stands in regard to other conversion tools. For low-level, high-precision control, users can employ the NWB APIs directly ([PyNWB](https://pynwb.readthedocs.io) in Python, [MatNWB](https://matnwb.readthedocs.io/) in MATLAB). For a guided GUI-based experience, the [NWB GUIDE](https://nwb-guide.readthedocs.io/) offers the best option but may be too rigid for complex workflows. NeuroConv stands as a middle ground, automating the conversion of a large number of formats while still allowing for customization and flexibility. |
There was a problem hiding this comment.
This is an excellent and highly informative figure. It clearly positions NeuroConv within the broader ecosystem of NWB conversion tools and helps readers immediately understand its target audience and value proposition. Great inclusion!
|
|
||
| **Format Coverage**: Despite supporting 47 formats, the neurophysiology ecosystem continues to evolve with new acquisition systems, format versions, and processing software. Each new system often introduces proprietary formats with unique metadata structures and data organization schemes. While NeuroConv's modular architecture allows users to develop custom DataInterfaces for unsupported formats, this process requires substantial technical expertise in both the source format's internal structure and NeuroConv's interface architecture. Additionally, maintaining custom interfaces as both the source format and NeuroConv evolve presents ongoing maintenance challenges for individual laboratories. | ||
|
|
||
| **Custom Laboratory Formats**: Many laboratories develop custom data storage solutions tailored to their specific experimental workflows, often utilizing MATLAB files, custom CSV schemas, or bespoke binary formats. These custom formats frequently exhibit significant variability not only between laboratories but even within the same laboratory over time as experimental protocols evolve. The heterogeneous nature of these formats, ranging from simple tabular data to complex nested structures with laboratory-specific metadata conventions, makes automated conversion particularly challenging. While NeuroConv can handle some standardized custom formats, the diversity of these approaches often requires manual intervention or custom code development. We recommend that laboratories work with data in its original acquisition format whenever possible, as this provides the richest metadata and most reliable conversion pathway. |
There was a problem hiding this comment.
In the "Custom Laboratory Formats" point, you correctly identify the challenge. You might consider adding a single sentence with a proactive recommendation, such as: "We therefore guide users to develop their custom DataInterface for these formats and offer tutorials and support to facilitate this process." This would turn the limitation into an opportunity for community engagement.
There was a problem hiding this comment.
This is a great suggestion. Thanks for pointing this out. In fact, this is an important point that we forget to mention, building a custom data interface is a common workflow when users have behavioral data (which is rarely standardized). I added a sentence here.
|
|
||
| **NWB Standard Evolution**: We actively participate in NWB schema development and maintain close alignment with emerging standards through engagement with NWB Extensions Proposals. By monitoring and contributing to current developments, including enhanced schemas for experimental events (NWBEP001), extracellular electrophysiology (NWBEP002), and optical physiology (NWBEP003, NWBEP004), we ensure that NeuroConv incorporates the latest standard improvements as they become available for the benefit of our users. | ||
|
|
||
| **AI-Assisted Conversion**: We are exploring large language model integration to advance our core mission of automating neurophysiology data conversion. This includes using LLMs to generate DataInterfaces from source format documentation, automatically extract metadata from experimental protocols, and generate custom conversion pipelines based on natural language requirements (conversion agent). |
There was a problem hiding this comment.
This is a very exciting direction! The idea of an LLM-powered "conversion agent" is particularly compelling. To make this more concrete, you could briefly mention the type of LLM interaction you envision (e.g., "fine-tuning a code-generation model on format specifications" or "using RAG with documentation to generate pipeline code"). This isn't required but could add a bit more technical depth to this forward-looking point.
There was a problem hiding this comment.
Thanks for this suggestion.
@luiztauffer, could you help me with a sentence in this direction?
There was a problem hiding this comment.
@h-mayorquin how about this:
We are developing an LLM-driven conversion agent that uses a lightweight framework (e.g., smolagents) to invoke a specialized suite of NWB-aware tools. Using retrieval-augmented generation, the agent queries a vector store (e.g. Qdrant) populated with neuroconv’s documentation, annotated conversion examples, and NWB best-practice notes, then synthesizes the Python code required for a complete conversion pipeline. Each resulting NWB file is automatically checked with nwbinspector-specialized tool for schema compliance and metadata completeness, while a repository creator tool makes use of our cookie-cutter template to reproduce CI scaffolds, tests, and project structure. By encapsulating years of domain expertise in callable tools and grounding every generation step in verifiable documentation, the agent seeks to streamline converter development, minimize hallucinations, and promote reproducible, standards-conformant data sharing.
There was a problem hiding this comment.
Thanks Luiz but I think this is too much for this paper. Let's trim it down to 1-2 sentences and save the rest for its own paper
There was a problem hiding this comment.
Thanks again for the suggestion @bhaskargoyal, I added more words in the suggested direction.
|
@anacomesana will serve as editor for this paper. |
| :label: fig:assets/minimal_conversion_pipeline | ||
| The process begins with source data (e.g., binary recordings, metadata, and configuration files). A data-specific DataInterface object is instantiated and used to extract metadata via the `interface.get_metadata()` method. The resulting metadata can be optionally edited by the user to fill in missing or corrected fields. The finalized metadata and source data are then passed to the `interface.run_conversion()` method, which writes a complete NWB file compliant with the standard | ||
| ::: | ||
|
|
There was a problem hiding this comment.
Comments on Figure 1:
- For consistency, the title of the third panel could be changed from "Editable" to "Edit metadata".
- Again in the third panel, "adds" -> "add", "unexisting" -> "nonexistent".
- In fourth panel, "Adds" -> "Add"
- The last subtitle could be improved: "Contains… metadata, converted to an NWB-compliant format."
- Consider leaving out of the caption specific object methods, like get_metadata() etc., and keeping the caption at a high level.
There was a problem hiding this comment.
Thanks for all those suggestions. I improved the figures and applied the suggestions.
| # Add the data to an NWB File and write it to disk | ||
| interface.run_conversion(nwbfile_path="path/to/nwbfile.nwb", metadata=metadata) | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Perhaps the second code snippet, which shows the same thing for a specific DataInterface makes the first code snippet redundant.
There was a problem hiding this comment.
Yes, it's true that they are somewhat redundant. However, they serve different purposes. The first snippet is meant to be abstract and explain the general workflow, whereas the second is a concrete example from the conversion gallery. Since they play different roles in the flow of the text, I’d prefer to keep both, unless you feel strongly about removing one. In summary, I’d rather be a bit redundant than disrupt the flow here.
| As NeuroConv's format support has expanded to 47+ formats, dependency management has become increasingly complex. Each format often requires specialized libraries with potentially conflicting version requirements, creating dependency resolution challenges that can make installation difficult or impossible. For example, libraries for reading different formats may require different versions of Python or different versions of common dependencies such as numpy. Additionally, installing the dependencies of every DataInterface would create an unnecessary and inefficient environment with hundreds of packages, many of which users never need for their conversion. | ||
|
|
||
| To address these challenges, we rely on [installation extras](https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-extras) to manage installation complexity. Users can specify only the formats they need during installation: | ||
|
|
There was a problem hiding this comment.
I like this approach to manage dependency conflicts!
| :::{figure} assets/diagram_converter.png | ||
| :label: fig:assets/diagram_converter | ||
| The Converter orchestrates multiple specialized interfaces (A, B, C), each handling different data types (electrophysiology, imaging, and behavior). Individual interfaces extract metadata from their respective data sources, which the Converter combines into a single, user-editable metadata structure through its `converter.get_metadata()` method. The Converter's `converter.run_conversion()` method then coordinates all interfaces to produce a unified NWB file containing all data modalities. This design pattern enables flexible, modular integration of heterogeneous neuroscience data into a single standardized format. | ||
| ::: |
There was a problem hiding this comment.
Again, the caption could be more high-level by omitting specific method names. The interfaces in the figure could be named with their proper class names (e.g. OpenEphysRecordingInterface rather than Interface A). Finally, the figure refers to a Converter, but the code in the snippet below and the main text refer to it as ConverterPipe.
There was a problem hiding this comment.
The caption was changed to be less specific and more high-level.
We have a ConverterPipe but that is not the only type of Converter that we have.
| The DataInterface abstracts away format-specific complexities: from parsing proprietary binary structures to extracting embedded metadata. | ||
|
|
||
| Currently supporting 47 distinct input formats (@tbl:formats), each DataInterface is comprehensively documented and demonstrated in the [Conversion Gallery](https://neuroconv.readthedocs.io/en/stable/conversion_examples_gallery/index.html), where users can find complete examples requiring only ~5 lines of code to perform full data conversion. Throughout the conversion process, NeuroConv enforces NWB Best Practices for metadata and data organization while optimizing data storage for both archival purposes and cloud computing requirements. | ||
|
|
There was a problem hiding this comment.
It's great that you link to the conversion gallery. It would also be helpful to link to the index page of the documentation somewhere in the paper (perhaps in the introduction).
There was a problem hiding this comment.
Yes, we have a link to the documentation in the first at the beginning of this section: line 55.
| For storage optimization, NeuroConv leverages chunking and compression in HDF5 and Zarr, the currently supported backends in NWB. Chunking allows large datasets to be read in manageable chunk sizes, and lossless compression algorithms allow us to reduce file size without altering the values of the data. There are many options for compression algorithms, and they present a trade-off between storage space and access speed [@alessio_compression_2023]. NeuroConv exposes an easy-to-use [API](https://neuroconv.readthedocs.io/en/stable/user_guide/backend_configuration.html) for configuring chunking and compression settings at the dataset level, allowing for quick experimentation while providing sensible defaults that work for most users. | ||
|
|
||
| Determining optimal chunk parameters involves complex tradeoffs [@zarr_performance]. Large chunks minimize the number of read operations but may require decompressing unnecessary data when accessing small subsets. Small chunks provide more precise access but increase overhead, particularly for cloud storage where each chunk requires a separate HTTP range request. Generally, appropriate chunking requires understanding the most common data access patterns. By understanding common analysis workflows and visualization patterns, it becomes feasible to implement evidence-based heuristics for chunk sizing across common data types, such as voltage recordings and imaging datasets. | ||
|
|
There was a problem hiding this comment.
Is there any available data on how performance scales with chunking? Also, are there benchmarks for how performance scales in high-volume datasets?
There was a problem hiding this comment.
Unfortunately not yet that I am aware of, but the community would like to move forward in that direction:
|
Hi, @anacomesana @rowanc1 How is the calendar looking for the next steps here? Given the time line it seems that that we are behind? https://github.com/scipy-conference/scipy_proceedings?tab=readme-ov-file#timeline-for-2025 Is there something that we are missing on our side? |
|
@h-mayorquin - we are good to go on this. |
|
This looks great! Thank you everyone for all your hard work. |
|
Approved and complete. |
Co-authored-by: Paul Adkisson <paul.wesley.adkisson@gmail.com>
Co-authored-by: Paul Adkisson-Floro <paul.wesley.adkisson@gmail.com>
fwkoch
added
approved
12 participants
If you are creating this PR in order to submit a draft of your paper, please name your PR with
Paper: <title>. An editor will then add adraftlabel; this will trigger GitHub Actions to run automated checks on your paper and build a preview. You may then work to resolve failed checks and ensure the preview build looks correct. If you have any questions, please tag the proceedings team in a comment on your PR with@scipy-conference/2025-proceedings.See the project readme for more information.