Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,13 @@
* Fixed the global compression test failing against `hdf5plugin>=7.0.0`. The test asserted on each filter's free-text description string, which 7.0.0 reworded for LZ4 and Zstd when switching to The HDF Group's bundled implementations (the filter IDs are unchanged). The test now asserts only on the stable numeric filter ID, which is what actually proves the requested filter was applied. [PR #1766](https://github.com/catalystneuro/neuroconv/pull/1766)

## Features
* Added HERD brain-region annotation. Anatomical `location` fields (the electrodes table `location` column, `ElectrodeGroup.location`, and `ImagingPlane.location`) are annotated with machine-readable ontology references stored in-file under `/general/external_resources`. Each location is resolved first against a user-defined `metadata["BrainRegions"]` mapping — where each brain area maps to one or more ontology terms given as `{"id": ..., "uri": ...}` (ontology-agnostic, so it applies to any species and supports mapping one area to several ontologies, e.g. MBA + UBERON) — and then against an offline Allen brain-atlas lookup for the subject's species — the Allen Mouse Brain Atlas for *Mus musculus* and the Allen Human Brain Atlas for *Homo sapiens*. The species and brain-region controlled vocabularies are shipped as curated LinkML TermSet YAML files (the format used by HDMF's `TermSet`, read directly so no extra dependency is added). Species and brain-region annotation are unified behind an overridable `OntologyAnnotationMixin` (methods `add_species_external_resource` and `add_brain_region_external_resources`) on `BaseDataInterface`/`NWBConverter`, applied at write time. [PR #1776](https://github.com/catalystneuro/neuroconv/pull/1776)
* Added `CSVEventsInterface` for converting discrete events from a CSV file into `ndx-events` objects. [PR #1755](https://github.com/catalystneuro/neuroconv/pull/1755)
* Added `TDTEventsInterface` for converting discrete events (epocs) from a TDT tank folder, depending on the `ndx-events` extension. [PR #1751](https://github.com/catalystneuro/neuroconv/pull/1751)
* Added `VameInterface` for converting VAME behavioral segmentation data to NWB using the `ndx-vame` extension. Writes per-frame motif labels (`MotifSeries`), optional latent-space embeddings (`LatentSpaceSeries`), optional community labels (`CommunitySeries`), and serializes the VAME project config as JSON. [PR #1737](https://github.com/catalystneuro/neuroconv/pull/1737)
* Added `AxonIntracellularInterface` for converting intracellular electrophysiology recorded in Axon Binary Format (.abf). Each instance handles one electrode in one file, writing its response as a single continuous `PatchClampSeries` (with an optional paired stimulus taken either from a recorded monitor channel or from a reconstructed protocol command) and one `IntracellularRecordings` row per sweep, addressed by `(start_index, count)` ranges and tagged with the run's `sequence` and `stimulus_type` columns. The clamp mode (`voltage_clamp`, `current_clamp`, or `izero`) is selected explicitly because Axon clamp-mode metadata is unreliable. It uses the dict-based metadata format keyed by `metadata_key`. The interface writes only the per-sweep recordings rows; the upper icephys hierarchy tables (`SimultaneousRecordings` and above) are assembled separately once the full set of channels and files is known. [PR #1746](https://github.com/catalystneuro/neuroconv/pull/1746)
* Added `neuroconv.tools.ontology` with `validate_species`/`get_species_suggestion`, an offline recommender that maps common species names and typos to their canonical Latin binomial and NCBITaxon identifier. `make_nwbfile_from_metadata` now emits a non-blocking suggestion when `Subject.species` is a recognized common name (e.g. `"mouse"` → `Mus musculus` / `NCBITaxon:10090`) or a likely typo. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771)
* Added HERD external-resource integration for `Subject.species`. When the species is a recognized Latin binomial, `make_nwbfile_from_metadata` now attaches a machine-readable NCBITaxon reference to the file (stored in-file under `/general/external_resources` via `add_species_external_resource`), so downstream tools can resolve the term without guessing. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771)
* Added HERD external-resource integration for `Subject.species`. When the species is a recognized Latin binomial, a conversion attaches a machine-readable NCBITaxon reference to the file (stored in-file under `/general/external_resources` via `add_species_external_resource`), so downstream tools can resolve the term without guessing. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771)
* Added `XClustSortingInterface` for converting XClust (.CEL) spike sorting data, using the `XClustSortingExtractor` from SpikeInterface. [PR #1691](https://github.com/catalystneuro/neuroconv/pull/1691)
* Added `IntanStimInterface` for converting electrical stimulation current data from Intan RHS2000 systems. Stimulation channels (one per amplifier channel, named `{channel}_STIM`) are written as a `TimeSeries` with `unit="A"` (Amperes), with the conversion factor derived automatically from `stim_step_size` in the .rhs file header. [PR #1711](https://github.com/catalystneuro/neuroconv/pull/1711)
* Added a `saved_files_are_split` parameter to `IntanRecordingInterface`, `IntanAnalogInterface`, and `IntanStimInterface` to support Intan RHX recordings saved with the "create a new save file every N minutes" option. When enabled, all sibling `.rhd`/`.rhs` files in the session folder are concatenated in filename order. When disabled, the interface warns if sibling rotation files are detected next to the one the user pointed at. [PR #1724](https://github.com/catalystneuro/neuroconv/pull/1724)
Expand All @@ -68,7 +69,7 @@
* Added `metadata_key` support to recording interfaces (`BaseRecordingExtractorInterface`, inherited by `BaseLFPExtractorInterface` and all recording interfaces) and `use_new_metadata_format` parameter to `get_metadata()` for the dict-based metadata pipeline. `metadata_key` defaults to the value of `es_key`. [PR #1747](https://github.com/catalystneuro/neuroconv/pull/1747)

## Improvements
* Added documentation for the species ontology tools: a user guide page (`docs/user_guide/species_ontology.rst`) covering the non-blocking species suggestion and the in-file NCBITaxon (HERD) annotation, plus an API reference page for `neuroconv.tools.ontology`. [PR #1772](https://github.com/catalystneuro/neuroconv/pull/1772)
* Added documentation for the ontology tools: a user guide page (`docs/user_guide/ontology.rst`) covering the non-blocking species suggestion, the in-file NCBITaxon (HERD) species annotation, and the brain-region (Allen Mouse Brain Atlas) annotation, plus an API reference page for `neuroconv.tools.ontology`. [PR #1772](https://github.com/catalystneuro/neuroconv/pull/1772)
* Migrated `MockPoseEstimationInterface` to the dict-based metadata shape used by the rest of the unified pipeline: top-level `metadata["Devices"]`, with `metadata["Behavior"]["Pose"]["PoseEstimations"][metadata_key]` and `metadata["Behavior"]["Pose"]["Skeletons"][metadata_key]` sub-registries linked via `device_metadata_key` and `skeleton_metadata_key`. The constructor argument `pose_estimation_metadata_key` was renamed to `metadata_key` for consistency with the other dict-based interfaces. This anchors the target shape for the upcoming DLC, Lightning Pose, and SLEAP interface migrations. [PR #1745](https://github.com/catalystneuro/neuroconv/pull/1745)
* `MiniscopeBehaviorInterface` no longer depends on `ndx-miniscope` for raw format parsing; `ndx-miniscope` is used only for NWB construction. Legacy V3 Miniscope folders now raise a clear error on both Miniscope interfaces pointing users to the issue tracker so V3 support can be added against real fixtures. [PR #1725](https://github.com/catalystneuro/neuroconv/pull/1725)
* Added a conversion gallery example for aligning ScanImage imaging data with external DAQ sync pulses, demonstrating use of `get_original_frame_indices` and `slice_samples` from `ScanImageImagingExtractor`. [PR #1709](https://github.com/catalystneuro/neuroconv/pull/1709)
Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
include *.yml
include *.json
include src/neuroconv/schemas/*
include src/neuroconv/tools/ontology/term_sets/*.yaml
include src/neuroconv/tools/testing/_path_expander_demo_ibl_filepaths.txt
2 changes: 1 addition & 1 deletion docs/user_guide/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ and synchronize data across multiple sources.
temporal_alignment
csvs
expand_path
species_ontology
ontology
backend_configuration
linking_sorted_data
yaml
Expand Down
222 changes: 222 additions & 0 deletions docs/user_guide/ontology.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,222 @@
Ontology Annotation
===================

NeuroConv can attach machine-readable **ontology references** to a written NWB file, so downstream
tools such as the `DANDI Archive <https://dandiarchive.org/>`_ can resolve exactly what a value
means instead of guessing from free text. References are stored **in-file** under
``/general/external_resources`` using HDMF's HERD (External Resources Data), so they travel with the
file.

Two kinds of value are annotated automatically by a conversion:

- the subject's **species**, mapped to `NCBITaxon <https://bioregistry.io/registry/ncbitaxon>`_;
- anatomical **brain regions** (``location`` fields), mapped to the
`Allen Mouse Brain Atlas <https://bioregistry.io/registry/mba>`_ (MBA) for mouse subjects and the
`Allen Human Brain Atlas <https://bioregistry.io/registry/hba>`_ (HBA) for human subjects, or to
any ontology you specify in metadata.

The recognized terms live in curated `LinkML <https://linkml.io/>`_ TermSet files shipped with
NeuroConv (one per vocabulary, the same format used by
`HDMF's TermSet <https://hdmf.readthedocs.io/en/stable/tutorials/plot_term_set.html>`_), so the
mappings are transparent and editable.

Both annotations are applied at write time by the overridable
:py:class:`~neuroconv.tools.ontology.OntologyAnnotationMixin`, which ``BaseDataInterface`` and
``NWBConverter`` inherit (see `Customizing the annotation`_ below). In-file HERD storage requires
``pynwb >= 4.0.0``, which is NeuroConv's minimum supported version.

Species
-------

NWB stores a subject's species in :py:attr:`Subject.species <pynwb.file.Subject.species>` as a
binomial Latin name (e.g. ``"Mus musculus"``) or a taxonomy URL. NeuroConv helps standardize it in
two complementary ways, backed by a small, curated, offline table of common neuroscience species
(:py:data:`~neuroconv.tools.ontology.SPECIES_TERMS`). The table needs no network access and no extra
dependencies, and it is high-precision: it only speaks up when confident, and valid-but-uncommon
binomials pass through silently.

Suggesting a standardized term
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When ``Subject.species`` is a recognized common name (e.g. ``"mouse"``) or a likely typo of a known
binomial (e.g. ``"Homo sapien"``), NeuroConv emits a ``UserWarning`` recommending the canonical
Latin binomial and its NCBITaxon identifier while the metadata is processed in
:py:func:`~neuroconv.tools.nwb_helpers.make_nwbfile_from_metadata`. This never raises and never
blocks a conversion.

.. code-block:: python

from neuroconv.tools.ontology import validate_species

validate_species("mouse")
# UserWarning: Subject species 'mouse' is a common name. Consider using the Latin binomial
# 'Mus musculus' (NCBITaxon:10090) for interoperability. See https://bioregistry.io/NCBITaxon:10090

validate_species("Homo sapien")
# UserWarning: Subject species 'Homo sapien' closely matches a known species name. Consider using
# the Latin binomial 'Homo sapiens' (NCBITaxon:9606) for interoperability. ...

validate_species("Mus musculus") # already canonical -> no warning, returns the term
validate_species("Octodon degus") # valid but not in the table -> no warning, returns None

To resolve a value to its canonical term without emitting a warning, use
:py:func:`~neuroconv.tools.ontology.get_species_term`, which also succeeds on exact canonical
matches:

.. code-block:: python

from neuroconv.tools.ontology import get_species_term

term = get_species_term("rhesus macaque")
term.canonical_name # 'Macaca mulatta'
term.ncbitaxon_id # 'NCBITaxon:9544'
term.entity_uri # 'http://purl.obolibrary.org/obo/NCBITaxon_9544'

Annotating the file with an NCBITaxon reference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When the species resolves to a recognized term, NeuroConv attaches a reference mapping
``Subject.species`` to its NCBITaxon entity:

.. code-block:: python

from neuroconv.tools.ontology import add_species_external_resource

# nwbfile.subject.species == "Mus musculus"
added = add_species_external_resource(nwbfile) # returns True
nwbfile.external_resources # now carries a Mus musculus -> NCBITaxon:10090 reference

The call is a no-op (returns ``False``) when there is no subject or the species is not recognized,
and it is idempotent: an existing ``external_resources`` HERD is extended in place rather than
replaced, and a species that is already annotated is not added twice.

Brain regions
-------------

Anatomical locations are stored in NWB as free-text strings: the ``location`` column of the
electrodes table (ecephys), ``ElectrodeGroup.location``, and ``ImagingPlane.location`` (ophys). For
a **mouse** or **human** subject, NeuroConv can attach an Allen-atlas reference to each of these, so
downstream tools can resolve the exact structure instead of guessing from an acronym. This runs at
write time, once the electrodes table and imaging planes have been populated.

How locations are resolved
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Each distinct ``location`` string is resolved in two steps:

1. **Metadata mapping (takes precedence).** If ``metadata["BrainRegions"]`` provides an entry for
the exact location string, those terms are used. The mapping is ontology-agnostic (each term is
an explicit ``id`` and ``uri``), so it applies to **any** species and can attach more than one
term to a region. This is how you annotate a region the offline lookup does not recognize, or
override one it does.
2. **Offline lookup (per species).** Otherwise NeuroConv consults the curated atlas for the
subject's species -- the Allen Mouse Brain Atlas for *Mus musculus* and the Allen Human Brain
Atlas for *Homo sapiens* -- matching an exact Allen acronym (case-sensitive, e.g. ``"CA1"``,
``"VISp"``), a canonical structure name (case-insensitive, e.g. ``"caudoputamen"``), or a common
informal name or abbreviation (e.g. ``"hippocampus"``, ``"V1"``).

Locations that resolve to neither (including the ``"unknown"`` placeholder) are left unannotated.
The lookup is species-specific because the same acronym denotes different structures across atlases
(e.g. ``"MB"`` is the mouse midbrain but the human mammillary body); a subject whose species has no
supported atlas is annotated only through the metadata mapping.

Automatic annotation
~~~~~~~~~~~~~~~~~~~~~~

No configuration is required for recognized regions. Given a mouse recording whose electrodes carry
Allen acronyms as their ``location`` (for a SpikeInterface recording, this is the ``brain_area``
property), a conversion writes an NCBITaxon reference for the species and an MBA reference for each
recognized region:

.. code-block:: python

# recording.set_property("brain_area", ["CA1", "CA1", "VISp"]) upstream
metadata["Subject"] = dict(subject_id="m1", species="Mus musculus", sex="M", age="P30D")

nwbfile = interface.create_nwbfile(metadata=metadata)
nwbfile.external_resources.to_dataframe()[["key", "entity_id", "entity_uri"]]
# key entity_id entity_uri
# Mus musculus NCBITaxon:10090 http://purl.obolibrary.org/obo/NCBITaxon_10090
# CA1 MBA:382 https://purl.brain-bican.org/ontology/mbao/MBA_382
# VISp MBA:385 https://purl.brain-bican.org/ontology/mbao/MBA_385

Defining the mapping in metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When a location string is not recognized (a lab-specific label, a subregion outside the curated
table, a non-standard spelling, or a non-mouse species), map it under ``metadata["BrainRegions"]``.
Each brain area (the key) maps to an ontology term given as a dict with an ``id`` and a resolvable
``uri``. Because the term is explicit rather than MBA-specific, this mapping generalizes to any
ontology and any species:

.. code-block:: python

metadata["BrainRegions"] = {
"my recording site": {
"id": "MBA:382",
"uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382",
},
}

interface.run_conversion(nwbfile_path="out.nwb", metadata=metadata)

To annotate one brain area with **several** ontologies (e.g. both the Allen atlas and UBERON), map
it to a list of terms:

.. code-block:: python

metadata["BrainRegions"] = {
"CA1": [
{"id": "MBA:382", "uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382"},
{"id": "UBERON:0003881", "uri": "http://purl.obolibrary.org/obo/UBERON_0003881"},
],
}

The metadata mapping takes precedence over the offline lookup, so you can also use it to override a
string the mouse lookup would otherwise resolve differently.

Customizing the annotation
--------------------------

Both annotations are overridable methods provided by
:py:class:`~neuroconv.tools.ontology.OntologyAnnotationMixin`, which ``BaseDataInterface`` and
``NWBConverter`` inherit:

- ``add_species_external_resource(nwbfile, metadata=None)`` — the subject species;
- ``add_brain_region_external_resources(nwbfile, metadata=None)`` — anatomical locations.

Each runs at write time, once the interface/converter data has been added to the file. Override one
in your interface or converter subclass to change or disable that annotation — for example to use a
different atlas, annotate additional objects, or turn it off:

.. code-block:: python

class MyConverter(NWBConverter):
def add_brain_region_external_resources(self, nwbfile, metadata=None):
return 0 # disable brain-region annotation

# or extend the default behavior:
class MyOtherConverter(NWBConverter):
def add_brain_region_external_resources(self, nwbfile, metadata=None):
number_added = super().add_brain_region_external_resources(nwbfile, metadata=metadata)
# ... attach additional references here ...
return number_added

Using the lookups directly
--------------------------

The resolution and annotation functions are available in :py:mod:`neuroconv.tools.ontology`:

.. code-block:: python

from neuroconv.tools.ontology import get_brain_region_term, add_brain_region_external_resources

term = get_brain_region_term("caudoputamen") # species defaults to "Mus musculus"
term.acronym # 'CP'
term.curie # 'MBA:672'
term.entity_uri # 'https://purl.brain-bican.org/ontology/mbao/MBA_672'

get_brain_region_term("CA1", species="Homo sapiens").curie # 'HBA:12892'

# Annotate an already-populated in-memory NWBFile (no-op unless the subject has a supported atlas):
number_added = add_brain_region_external_resources(nwbfile, metadata=metadata)
Loading
Loading