diff --git a/CHANGELOG.md b/CHANGELOG.md index 2fc708c89..9c3758204 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,12 +46,13 @@ * Fixed the global compression test failing against `hdf5plugin>=7.0.0`. The test asserted on each filter's free-text description string, which 7.0.0 reworded for LZ4 and Zstd when switching to The HDF Group's bundled implementations (the filter IDs are unchanged). The test now asserts only on the stable numeric filter ID, which is what actually proves the requested filter was applied. [PR #1766](https://github.com/catalystneuro/neuroconv/pull/1766) ## Features +* Added HERD brain-region annotation. Anatomical `location` fields (the electrodes table `location` column, `ElectrodeGroup.location`, and `ImagingPlane.location`) are annotated with machine-readable ontology references stored in-file under `/general/external_resources`. Each location is resolved first against a user-defined `metadata["BrainRegions"]` mapping — where each brain area maps to one or more ontology terms given as `{"id": ..., "uri": ...}` (ontology-agnostic, so it applies to any species and supports mapping one area to several ontologies, e.g. MBA + UBERON) — and then against an offline Allen brain-atlas lookup for the subject's species — the Allen Mouse Brain Atlas for *Mus musculus* and the Allen Human Brain Atlas for *Homo sapiens*. The species and brain-region controlled vocabularies are shipped as curated LinkML TermSet YAML files (the format used by HDMF's `TermSet`, read directly so no extra dependency is added). Species and brain-region annotation are unified behind an overridable `OntologyAnnotationMixin` (methods `add_species_external_resource` and `add_brain_region_external_resources`) on `BaseDataInterface`/`NWBConverter`, applied at write time. [PR #1776](https://github.com/catalystneuro/neuroconv/pull/1776) * Added `CSVEventsInterface` for converting discrete events from a CSV file into `ndx-events` objects. [PR #1755](https://github.com/catalystneuro/neuroconv/pull/1755) * Added `TDTEventsInterface` for converting discrete events (epocs) from a TDT tank folder, depending on the `ndx-events` extension. [PR #1751](https://github.com/catalystneuro/neuroconv/pull/1751) * Added `VameInterface` for converting VAME behavioral segmentation data to NWB using the `ndx-vame` extension. Writes per-frame motif labels (`MotifSeries`), optional latent-space embeddings (`LatentSpaceSeries`), optional community labels (`CommunitySeries`), and serializes the VAME project config as JSON. [PR #1737](https://github.com/catalystneuro/neuroconv/pull/1737) * Added `AxonIntracellularInterface` for converting intracellular electrophysiology recorded in Axon Binary Format (.abf). Each instance handles one electrode in one file, writing its response as a single continuous `PatchClampSeries` (with an optional paired stimulus taken either from a recorded monitor channel or from a reconstructed protocol command) and one `IntracellularRecordings` row per sweep, addressed by `(start_index, count)` ranges and tagged with the run's `sequence` and `stimulus_type` columns. The clamp mode (`voltage_clamp`, `current_clamp`, or `izero`) is selected explicitly because Axon clamp-mode metadata is unreliable. It uses the dict-based metadata format keyed by `metadata_key`. The interface writes only the per-sweep recordings rows; the upper icephys hierarchy tables (`SimultaneousRecordings` and above) are assembled separately once the full set of channels and files is known. [PR #1746](https://github.com/catalystneuro/neuroconv/pull/1746) * Added `neuroconv.tools.ontology` with `validate_species`/`get_species_suggestion`, an offline recommender that maps common species names and typos to their canonical Latin binomial and NCBITaxon identifier. `make_nwbfile_from_metadata` now emits a non-blocking suggestion when `Subject.species` is a recognized common name (e.g. `"mouse"` → `Mus musculus` / `NCBITaxon:10090`) or a likely typo. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771) -* Added HERD external-resource integration for `Subject.species`. When the species is a recognized Latin binomial, `make_nwbfile_from_metadata` now attaches a machine-readable NCBITaxon reference to the file (stored in-file under `/general/external_resources` via `add_species_external_resource`), so downstream tools can resolve the term without guessing. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771) +* Added HERD external-resource integration for `Subject.species`. When the species is a recognized Latin binomial, a conversion attaches a machine-readable NCBITaxon reference to the file (stored in-file under `/general/external_resources` via `add_species_external_resource`), so downstream tools can resolve the term without guessing. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771) * Added `XClustSortingInterface` for converting XClust (.CEL) spike sorting data, using the `XClustSortingExtractor` from SpikeInterface. [PR #1691](https://github.com/catalystneuro/neuroconv/pull/1691) * Added `IntanStimInterface` for converting electrical stimulation current data from Intan RHS2000 systems. Stimulation channels (one per amplifier channel, named `{channel}_STIM`) are written as a `TimeSeries` with `unit="A"` (Amperes), with the conversion factor derived automatically from `stim_step_size` in the .rhs file header. [PR #1711](https://github.com/catalystneuro/neuroconv/pull/1711) * Added a `saved_files_are_split` parameter to `IntanRecordingInterface`, `IntanAnalogInterface`, and `IntanStimInterface` to support Intan RHX recordings saved with the "create a new save file every N minutes" option. When enabled, all sibling `.rhd`/`.rhs` files in the session folder are concatenated in filename order. When disabled, the interface warns if sibling rotation files are detected next to the one the user pointed at. [PR #1724](https://github.com/catalystneuro/neuroconv/pull/1724) @@ -68,7 +69,7 @@ * Added `metadata_key` support to recording interfaces (`BaseRecordingExtractorInterface`, inherited by `BaseLFPExtractorInterface` and all recording interfaces) and `use_new_metadata_format` parameter to `get_metadata()` for the dict-based metadata pipeline. `metadata_key` defaults to the value of `es_key`. [PR #1747](https://github.com/catalystneuro/neuroconv/pull/1747) ## Improvements -* Added documentation for the species ontology tools: a user guide page (`docs/user_guide/species_ontology.rst`) covering the non-blocking species suggestion and the in-file NCBITaxon (HERD) annotation, plus an API reference page for `neuroconv.tools.ontology`. [PR #1772](https://github.com/catalystneuro/neuroconv/pull/1772) +* Added documentation for the ontology tools: a user guide page (`docs/user_guide/ontology.rst`) covering the non-blocking species suggestion, the in-file NCBITaxon (HERD) species annotation, and the brain-region (Allen Mouse Brain Atlas) annotation, plus an API reference page for `neuroconv.tools.ontology`. [PR #1772](https://github.com/catalystneuro/neuroconv/pull/1772) * Migrated `MockPoseEstimationInterface` to the dict-based metadata shape used by the rest of the unified pipeline: top-level `metadata["Devices"]`, with `metadata["Behavior"]["Pose"]["PoseEstimations"][metadata_key]` and `metadata["Behavior"]["Pose"]["Skeletons"][metadata_key]` sub-registries linked via `device_metadata_key` and `skeleton_metadata_key`. The constructor argument `pose_estimation_metadata_key` was renamed to `metadata_key` for consistency with the other dict-based interfaces. This anchors the target shape for the upcoming DLC, Lightning Pose, and SLEAP interface migrations. [PR #1745](https://github.com/catalystneuro/neuroconv/pull/1745) * `MiniscopeBehaviorInterface` no longer depends on `ndx-miniscope` for raw format parsing; `ndx-miniscope` is used only for NWB construction. Legacy V3 Miniscope folders now raise a clear error on both Miniscope interfaces pointing users to the issue tracker so V3 support can be added against real fixtures. [PR #1725](https://github.com/catalystneuro/neuroconv/pull/1725) * Added a conversion gallery example for aligning ScanImage imaging data with external DAQ sync pulses, demonstrating use of `get_original_frame_indices` and `slice_samples` from `ScanImageImagingExtractor`. [PR #1709](https://github.com/catalystneuro/neuroconv/pull/1709) diff --git a/MANIFEST.in b/MANIFEST.in index dbf19a4c5..b087b6435 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,4 +1,5 @@ include *.yml include *.json include src/neuroconv/schemas/* +include src/neuroconv/tools/ontology/term_sets/*.yaml include src/neuroconv/tools/testing/_path_expander_demo_ibl_filepaths.txt diff --git a/docs/user_guide/index.rst b/docs/user_guide/index.rst index 408907bdd..faed55a3e 100644 --- a/docs/user_guide/index.rst +++ b/docs/user_guide/index.rst @@ -24,7 +24,7 @@ and synchronize data across multiple sources. temporal_alignment csvs expand_path - species_ontology + ontology backend_configuration linking_sorted_data yaml diff --git a/docs/user_guide/ontology.rst b/docs/user_guide/ontology.rst new file mode 100644 index 000000000..4f9118117 --- /dev/null +++ b/docs/user_guide/ontology.rst @@ -0,0 +1,222 @@ +Ontology Annotation +=================== + +NeuroConv can attach machine-readable **ontology references** to a written NWB file, so downstream +tools such as the `DANDI Archive `_ can resolve exactly what a value +means instead of guessing from free text. References are stored **in-file** under +``/general/external_resources`` using HDMF's HERD (External Resources Data), so they travel with the +file. + +Two kinds of value are annotated automatically by a conversion: + +- the subject's **species**, mapped to `NCBITaxon `_; +- anatomical **brain regions** (``location`` fields), mapped to the + `Allen Mouse Brain Atlas `_ (MBA) for mouse subjects and the + `Allen Human Brain Atlas `_ (HBA) for human subjects, or to + any ontology you specify in metadata. + +The recognized terms live in curated `LinkML `_ TermSet files shipped with +NeuroConv (one per vocabulary, the same format used by +`HDMF's TermSet `_), so the +mappings are transparent and editable. + +Both annotations are applied at write time by the overridable +:py:class:`~neuroconv.tools.ontology.OntologyAnnotationMixin`, which ``BaseDataInterface`` and +``NWBConverter`` inherit (see `Customizing the annotation`_ below). In-file HERD storage requires +``pynwb >= 4.0.0``, which is NeuroConv's minimum supported version. + +Species +------- + +NWB stores a subject's species in :py:attr:`Subject.species ` as a +binomial Latin name (e.g. ``"Mus musculus"``) or a taxonomy URL. NeuroConv helps standardize it in +two complementary ways, backed by a small, curated, offline table of common neuroscience species +(:py:data:`~neuroconv.tools.ontology.SPECIES_TERMS`). The table needs no network access and no extra +dependencies, and it is high-precision: it only speaks up when confident, and valid-but-uncommon +binomials pass through silently. + +Suggesting a standardized term +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``Subject.species`` is a recognized common name (e.g. ``"mouse"``) or a likely typo of a known +binomial (e.g. ``"Homo sapien"``), NeuroConv emits a ``UserWarning`` recommending the canonical +Latin binomial and its NCBITaxon identifier while the metadata is processed in +:py:func:`~neuroconv.tools.nwb_helpers.make_nwbfile_from_metadata`. This never raises and never +blocks a conversion. + +.. code-block:: python + + from neuroconv.tools.ontology import validate_species + + validate_species("mouse") + # UserWarning: Subject species 'mouse' is a common name. Consider using the Latin binomial + # 'Mus musculus' (NCBITaxon:10090) for interoperability. See https://bioregistry.io/NCBITaxon:10090 + + validate_species("Homo sapien") + # UserWarning: Subject species 'Homo sapien' closely matches a known species name. Consider using + # the Latin binomial 'Homo sapiens' (NCBITaxon:9606) for interoperability. ... + + validate_species("Mus musculus") # already canonical -> no warning, returns the term + validate_species("Octodon degus") # valid but not in the table -> no warning, returns None + +To resolve a value to its canonical term without emitting a warning, use +:py:func:`~neuroconv.tools.ontology.get_species_term`, which also succeeds on exact canonical +matches: + +.. code-block:: python + + from neuroconv.tools.ontology import get_species_term + + term = get_species_term("rhesus macaque") + term.canonical_name # 'Macaca mulatta' + term.ncbitaxon_id # 'NCBITaxon:9544' + term.entity_uri # 'http://purl.obolibrary.org/obo/NCBITaxon_9544' + +Annotating the file with an NCBITaxon reference +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When the species resolves to a recognized term, NeuroConv attaches a reference mapping +``Subject.species`` to its NCBITaxon entity: + +.. code-block:: python + + from neuroconv.tools.ontology import add_species_external_resource + + # nwbfile.subject.species == "Mus musculus" + added = add_species_external_resource(nwbfile) # returns True + nwbfile.external_resources # now carries a Mus musculus -> NCBITaxon:10090 reference + +The call is a no-op (returns ``False``) when there is no subject or the species is not recognized, +and it is idempotent: an existing ``external_resources`` HERD is extended in place rather than +replaced, and a species that is already annotated is not added twice. + +Brain regions +------------- + +Anatomical locations are stored in NWB as free-text strings: the ``location`` column of the +electrodes table (ecephys), ``ElectrodeGroup.location``, and ``ImagingPlane.location`` (ophys). For +a **mouse** or **human** subject, NeuroConv can attach an Allen-atlas reference to each of these, so +downstream tools can resolve the exact structure instead of guessing from an acronym. This runs at +write time, once the electrodes table and imaging planes have been populated. + +How locations are resolved +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each distinct ``location`` string is resolved in two steps: + +1. **Metadata mapping (takes precedence).** If ``metadata["BrainRegions"]`` provides an entry for + the exact location string, those terms are used. The mapping is ontology-agnostic (each term is + an explicit ``id`` and ``uri``), so it applies to **any** species and can attach more than one + term to a region. This is how you annotate a region the offline lookup does not recognize, or + override one it does. +2. **Offline lookup (per species).** Otherwise NeuroConv consults the curated atlas for the + subject's species -- the Allen Mouse Brain Atlas for *Mus musculus* and the Allen Human Brain + Atlas for *Homo sapiens* -- matching an exact Allen acronym (case-sensitive, e.g. ``"CA1"``, + ``"VISp"``), a canonical structure name (case-insensitive, e.g. ``"caudoputamen"``), or a common + informal name or abbreviation (e.g. ``"hippocampus"``, ``"V1"``). + +Locations that resolve to neither (including the ``"unknown"`` placeholder) are left unannotated. +The lookup is species-specific because the same acronym denotes different structures across atlases +(e.g. ``"MB"`` is the mouse midbrain but the human mammillary body); a subject whose species has no +supported atlas is annotated only through the metadata mapping. + +Automatic annotation +~~~~~~~~~~~~~~~~~~~~~~ + +No configuration is required for recognized regions. Given a mouse recording whose electrodes carry +Allen acronyms as their ``location`` (for a SpikeInterface recording, this is the ``brain_area`` +property), a conversion writes an NCBITaxon reference for the species and an MBA reference for each +recognized region: + +.. code-block:: python + + # recording.set_property("brain_area", ["CA1", "CA1", "VISp"]) upstream + metadata["Subject"] = dict(subject_id="m1", species="Mus musculus", sex="M", age="P30D") + + nwbfile = interface.create_nwbfile(metadata=metadata) + nwbfile.external_resources.to_dataframe()[["key", "entity_id", "entity_uri"]] + # key entity_id entity_uri + # Mus musculus NCBITaxon:10090 http://purl.obolibrary.org/obo/NCBITaxon_10090 + # CA1 MBA:382 https://purl.brain-bican.org/ontology/mbao/MBA_382 + # VISp MBA:385 https://purl.brain-bican.org/ontology/mbao/MBA_385 + +Defining the mapping in metadata +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When a location string is not recognized (a lab-specific label, a subregion outside the curated +table, a non-standard spelling, or a non-mouse species), map it under ``metadata["BrainRegions"]``. +Each brain area (the key) maps to an ontology term given as a dict with an ``id`` and a resolvable +``uri``. Because the term is explicit rather than MBA-specific, this mapping generalizes to any +ontology and any species: + +.. code-block:: python + + metadata["BrainRegions"] = { + "my recording site": { + "id": "MBA:382", + "uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382", + }, + } + + interface.run_conversion(nwbfile_path="out.nwb", metadata=metadata) + +To annotate one brain area with **several** ontologies (e.g. both the Allen atlas and UBERON), map +it to a list of terms: + +.. code-block:: python + + metadata["BrainRegions"] = { + "CA1": [ + {"id": "MBA:382", "uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382"}, + {"id": "UBERON:0003881", "uri": "http://purl.obolibrary.org/obo/UBERON_0003881"}, + ], + } + +The metadata mapping takes precedence over the offline lookup, so you can also use it to override a +string the mouse lookup would otherwise resolve differently. + +Customizing the annotation +-------------------------- + +Both annotations are overridable methods provided by +:py:class:`~neuroconv.tools.ontology.OntologyAnnotationMixin`, which ``BaseDataInterface`` and +``NWBConverter`` inherit: + +- ``add_species_external_resource(nwbfile, metadata=None)`` — the subject species; +- ``add_brain_region_external_resources(nwbfile, metadata=None)`` — anatomical locations. + +Each runs at write time, once the interface/converter data has been added to the file. Override one +in your interface or converter subclass to change or disable that annotation — for example to use a +different atlas, annotate additional objects, or turn it off: + +.. code-block:: python + + class MyConverter(NWBConverter): + def add_brain_region_external_resources(self, nwbfile, metadata=None): + return 0 # disable brain-region annotation + + # or extend the default behavior: + class MyOtherConverter(NWBConverter): + def add_brain_region_external_resources(self, nwbfile, metadata=None): + number_added = super().add_brain_region_external_resources(nwbfile, metadata=metadata) + # ... attach additional references here ... + return number_added + +Using the lookups directly +-------------------------- + +The resolution and annotation functions are available in :py:mod:`neuroconv.tools.ontology`: + +.. code-block:: python + + from neuroconv.tools.ontology import get_brain_region_term, add_brain_region_external_resources + + term = get_brain_region_term("caudoputamen") # species defaults to "Mus musculus" + term.acronym # 'CP' + term.curie # 'MBA:672' + term.entity_uri # 'https://purl.brain-bican.org/ontology/mbao/MBA_672' + + get_brain_region_term("CA1", species="Homo sapiens").curie # 'HBA:12892' + + # Annotate an already-populated in-memory NWBFile (no-op unless the subject has a supported atlas): + number_added = add_brain_region_external_resources(nwbfile, metadata=metadata) diff --git a/docs/user_guide/species_ontology.rst b/docs/user_guide/species_ontology.rst deleted file mode 100644 index 4fadc645f..000000000 --- a/docs/user_guide/species_ontology.rst +++ /dev/null @@ -1,79 +0,0 @@ -Species Ontology -================ - -NWB stores a subject's species in :py:attr:`Subject.species ` as a -binomial Latin name (e.g. ``"Mus musculus"``) or a taxonomy URL. Consistent, standardized species -values make files interoperable and allow downstream tools such as the -`DANDI Archive `_ to resolve the exact organism without guessing. - -NeuroConv helps with this in two complementary ways, both applied automatically by -:py:func:`~neuroconv.tools.nwb_helpers.make_nwbfile_from_metadata` (and therefore by every -conversion that builds an NWB file through it): - -1. A **non-blocking suggestion** when the species looks like a common name or a typo. -2. A **machine-readable NCBITaxon annotation** attached in-file when the species is recognized. - -The lookup is a small, curated, offline table of common neuroscience species -(:py:data:`~neuroconv.tools.ontology.SPECIES_TERMS`). It requires no network access and no extra -dependencies, and it is intentionally high-precision: it only speaks up when it is confident there -is a better term. Valid-but-uncommon binomials pass through silently. - -Suggesting a standardized term ------------------------------- - -When ``Subject.species`` is a recognized common name (e.g. ``"mouse"``) or a likely typo of a known -binomial (e.g. ``"Homo sapien"``), NeuroConv emits a ``UserWarning`` recommending the canonical -Latin binomial and its NCBITaxon identifier. This never raises and never blocks a conversion. - -.. code-block:: python - - from neuroconv.tools.ontology import validate_species - - validate_species("mouse") - # UserWarning: Subject species 'mouse' is a common name. Consider using the Latin binomial - # 'Mus musculus' (NCBITaxon:10090) for interoperability. See https://bioregistry.io/NCBITaxon:10090 - - validate_species("Homo sapien") - # UserWarning: Subject species 'Homo sapien' closely matches a known species name. Consider using - # the Latin binomial 'Homo sapiens' (NCBITaxon:9606) for interoperability. ... - - validate_species("Mus musculus") # already canonical -> no warning, returns the term - validate_species("Octodon degus") # valid but not in the table -> no warning, returns None - -To resolve a value to its canonical term without emitting a warning, use -:py:func:`~neuroconv.tools.ontology.get_species_term`, which also succeeds on exact canonical -matches: - -.. code-block:: python - - from neuroconv.tools.ontology import get_species_term - - term = get_species_term("rhesus macaque") - term.canonical_name # 'Macaca mulatta' - term.ncbitaxon_id # 'NCBITaxon:9544' - term.entity_uri # 'http://purl.obolibrary.org/obo/NCBITaxon_9544' - -Annotating the file with an NCBITaxon reference ------------------------------------------------ - -When the species resolves to a recognized term, NeuroConv attaches a machine-readable external -resource reference mapping ``Subject.species`` to its NCBITaxon entity. This is stored **in-file** -under ``/general/external_resources`` using HDMF's HERD (External Resources Data), so the link -travels with the file: - -.. code-block:: python - - from neuroconv.tools.ontology import add_species_external_resource - - # nwbfile.subject.species == "Mus musculus" - added = add_species_external_resource(nwbfile) # returns True - nwbfile.external_resources # now carries a Mus musculus -> NCBITaxon:10090 reference - -The call is a no-op (returns ``False``) when there is no subject or the species is not recognized, -and it is idempotent: an existing ``external_resources`` HERD is extended in place rather than -replaced, and a species that is already annotated is not added twice. - -.. note:: - - In-file HERD storage requires ``pynwb >= 4.0.0``, which is NeuroConv's minimum supported - version. diff --git a/pyproject.toml b/pyproject.toml index b2fba9258..89f847c09 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -475,7 +475,7 @@ known-first-party = ["neuroconv"] [tool.codespell] skip = '.git*,*.pdf,*.css,*.svg' check-hidden = true -ignore-words-list = 'assertin,sortings,Tye,trough' +ignore-words-list = 'assertin,sortings,Tye,trough,ect' [dependency-groups] test = [ diff --git a/src/neuroconv/basedatainterface.py b/src/neuroconv/basedatainterface.py index 617a511e9..003fc64d3 100644 --- a/src/neuroconv/basedatainterface.py +++ b/src/neuroconv/basedatainterface.py @@ -21,6 +21,7 @@ _resolve_backend, configure_and_write_nwbfile, ) +from .tools.ontology import OntologyAnnotationMixin from .utils import ( get_json_schema_from_method_signature, load_dict_from_file, @@ -29,7 +30,7 @@ from .utils.json_schema import _NWBMetaDataEncoder, _NWBSourceDataEncoder -class BaseDataInterface(ABC): +class BaseDataInterface(OntologyAnnotationMixin, ABC): """Abstract class defining the structure of all DataInterfaces.""" display_name: str | None = None @@ -151,6 +152,12 @@ def create_nwbfile(self, metadata: dict | None = None, **conversion_options) -> nwbfile = make_nwbfile_from_metadata(metadata=metadata) self.add_to_nwbfile(nwbfile=nwbfile, metadata=metadata, **conversion_options) + # Annotate the assembled file with ontology references (in-file HERD). Runs after data is + # added so the electrodes table and imaging planes exist. Both hooks are overridable + # (see OntologyAnnotationMixin). + self.add_species_external_resource(nwbfile, metadata=metadata) + self.add_brain_region_external_resources(nwbfile, metadata=metadata) + return nwbfile @abstractmethod @@ -267,6 +274,8 @@ def _write_nwbfile( """ if nwbfile is not None: self.add_to_nwbfile(nwbfile=nwbfile, metadata=metadata, **conversion_options) + self.add_species_external_resource(nwbfile, metadata=metadata) + self.add_brain_region_external_resources(nwbfile, metadata=metadata) else: nwbfile = self.create_nwbfile(metadata=metadata, **conversion_options) diff --git a/src/neuroconv/nwbconverter.py b/src/neuroconv/nwbconverter.py index d8459975c..776f00a60 100644 --- a/src/neuroconv/nwbconverter.py +++ b/src/neuroconv/nwbconverter.py @@ -22,6 +22,7 @@ make_nwbfile_from_metadata, ) from .tools.nwb_helpers._metadata_and_file_helpers import _resolve_backend +from .tools.ontology import OntologyAnnotationMixin from .utils import ( dict_deep_update, fill_defaults, @@ -37,7 +38,7 @@ ) -class NWBConverter: +class NWBConverter(OntologyAnnotationMixin): """Primary class for all NWB conversion classes.""" display_name: str | None = None @@ -227,6 +228,13 @@ def create_nwbfile(self, metadata: dict | None = None, conversion_options: dict nwbfile = make_nwbfile_from_metadata(metadata=metadata) self.add_to_nwbfile(nwbfile=nwbfile, metadata=metadata, conversion_options=conversion_options) + + # Annotate the assembled file with ontology references (in-file HERD). Runs after data is + # added so the electrodes table and imaging planes exist. Both hooks are overridable + # (see OntologyAnnotationMixin). + self.add_species_external_resource(nwbfile, metadata=metadata) + self.add_brain_region_external_resources(nwbfile, metadata=metadata) + return nwbfile def add_to_nwbfile(self, nwbfile: NWBFile, metadata: dict | None = None, conversion_options: dict | None = None): @@ -356,6 +364,8 @@ def _write_nwbfile( """ if nwbfile is not None: self.add_to_nwbfile(nwbfile=nwbfile, metadata=metadata, conversion_options=conversion_options) + self.add_species_external_resource(nwbfile, metadata=metadata) + self.add_brain_region_external_resources(nwbfile, metadata=metadata) else: nwbfile = self.create_nwbfile(metadata=metadata, conversion_options=conversion_options) diff --git a/src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py b/src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py index a2fce92cd..505f14403 100644 --- a/src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py +++ b/src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py @@ -19,7 +19,7 @@ configure_backend, get_default_backend_configuration, ) -from ..ontology import add_species_external_resource, validate_species +from ..ontology import validate_species from ...utils.dict import DeepDict, load_dict_from_file from ...utils.json_schema import validate_metadata @@ -143,10 +143,6 @@ def make_nwbfile_from_metadata(metadata: dict) -> NWBFile: nwbfile = NWBFile(**nwbfile_kwargs) - # Attach a machine-readable NCBITaxon reference for the subject species (in-file HERD). - # No-op when there is no subject or the species is not recognized. - add_species_external_resource(nwbfile) - return nwbfile diff --git a/src/neuroconv/tools/ontology/__init__.py b/src/neuroconv/tools/ontology/__init__.py index a9d586ca1..b3eae86ee 100644 --- a/src/neuroconv/tools/ontology/__init__.py +++ b/src/neuroconv/tools/ontology/__init__.py @@ -1,6 +1,16 @@ """Tools for recommending standardized ontology terms for NWB metadata.""" -from ._external_resources import add_species_external_resource +from ._brain_regions import ( + HBA_TERMS, + MBA_TERMS, + BrainRegionTerm, + get_brain_region_term, +) +from ._external_resources import ( + OntologyAnnotationMixin, + add_brain_region_external_resources, + add_species_external_resource, +) from ._species import ( SPECIES_TERMS, SpeciesTerm, @@ -10,9 +20,15 @@ ) __all__ = [ + "HBA_TERMS", + "MBA_TERMS", "SPECIES_TERMS", + "BrainRegionTerm", + "OntologyAnnotationMixin", "SpeciesTerm", + "add_brain_region_external_resources", "add_species_external_resource", + "get_brain_region_term", "get_species_suggestion", "get_species_term", "validate_species", diff --git a/src/neuroconv/tools/ontology/_brain_regions.py b/src/neuroconv/tools/ontology/_brain_regions.py new file mode 100644 index 000000000..0594ad8bf --- /dev/null +++ b/src/neuroconv/tools/ontology/_brain_regions.py @@ -0,0 +1,159 @@ +"""Lightweight, offline recognition of brain regions as Allen brain-atlas terms. + +NWB stores an anatomical location as a free-text string (e.g. the ``location`` column of the +electrodes table, ``ElectrodeGroup.location``, or ``ImagingPlane.location``). The Allen brain +atlases provide standard controlled vocabularies of brain structures per species: the Allen Mouse +Brain Atlas (``MBA``) for mouse and the Allen Human Brain Atlas (``HBA``) for human. Recognizing a +location string as an atlas term lets NeuroConv attach a machine-readable reference (``MBA:`` / +``HBA:``) so downstream tools can resolve the exact structure. + +The lookup is species-specific because the same acronym denotes different structures across atlases +(e.g. ``MB`` is the mouse midbrain but the human mammillary body). The terms live in the curated +TermSet files ``term_sets/mouse_brain_atlas.yaml`` and ``term_sets/human_brain_atlas.yaml`` +(identifiers taken from the Allen structure graphs, registered in the Bioregistry under the ``MBA`` +https://bioregistry.io/registry/mba and ``HBA`` https://bioregistry.io/registry/hba prefixes). This +module adds the offline resolution of a free-text location string to one of those terms. +""" + +from dataclasses import dataclass + +from ._species import get_species_term +from ._term_sets import load_term_set + +__all__ = [ + "HBA_TERMS", + "MBA_TERMS", + "SUPPORTED_ATLAS_SPECIES", + "BrainRegionTerm", + "get_brain_region_term", +] + + +@dataclass(frozen=True) +class BrainRegionTerm: + """An Allen brain-atlas structure and its ontology reference.""" + + acronym: str + name: str + curie: str # entity CURIE, e.g. "MBA:382" + entity_uri: str # resolvable entity URI (usable as a HERD ``entity_uri``) + + +# Common informal names and abbreviations -> Allen acronym, per atlas. Compared case-insensitively. +_MBA_ALIAS_TO_ACRONYM: dict[str, str] = { + "hippocampus": "HIP", + "entorhinal cortex": "ENT", + "primary visual cortex": "VISp", + "v1": "VISp", + "primary motor cortex": "MOp", + "m1": "MOp", + "primary somatosensory cortex": "SSp", + "s1": "SSp", + "barrel cortex": "SSp-bfd", + "neocortex": "Isocortex", + "dorsal striatum": "CP", + "locus coeruleus": "LC", + "substantia nigra pars compacta": "SNc", + "substantia nigra pars reticulata": "SNr", + "periaqueductal grey": "PAG", +} + +_HBA_ALIAS_TO_ACRONYM: dict[str, str] = { + "hippocampus": "HiF", + "caudate": "Cd", + "midbrain": "MES", + "medulla": "MY", + "medulla oblongata": "MY", + "cingulate cortex": "CgG", + "locus coeruleus": "LC", +} + + +@dataclass(frozen=True) +class _BrainAtlas: + """A curated, offline lookup of one species' brain-atlas terms.""" + + terms: dict[str, BrainRegionTerm] # acronym -> term + name_to_acronym: dict[str, str] # lower-cased canonical name -> acronym + alias_to_acronym: dict[str, str] # lower-cased informal name -> acronym + + def resolve(self, location: str) -> BrainRegionTerm | None: + """Resolve a location string to a term via exact acronym, canonical name, or alias.""" + stripped = location.strip() + if stripped == "": + return None + # Exact acronym (case-sensitive: acronyms like "VISp"/"CgG" are case-specific). + if stripped in self.terms: + return self.terms[stripped] + lowered = stripped.lower() + acronym = self.name_to_acronym.get(lowered) or self.alias_to_acronym.get(lowered) + return self.terms.get(acronym) if acronym is not None else None + + +def _build_atlas(term_set_file: str, alias_to_acronym: dict) -> _BrainAtlas: + terms = { + info.value: BrainRegionTerm( + acronym=info.value, name=info.description, curie=info.curie, entity_uri=info.entity_uri + ) + for info in load_term_set(term_set_file).values() + } + name_to_acronym = {term.name.lower(): term.acronym for term in terms.values()} + return _BrainAtlas(terms=terms, name_to_acronym=name_to_acronym, alias_to_acronym=alias_to_acronym) + + +_MBA_ATLAS = _build_atlas("mouse_brain_atlas.yaml", _MBA_ALIAS_TO_ACRONYM) +_HBA_ATLAS = _build_atlas("human_brain_atlas.yaml", _HBA_ALIAS_TO_ACRONYM) + +# Canonical species binomial -> its brain atlas. +_SPECIES_TO_ATLAS: dict[str, _BrainAtlas] = { + "Mus musculus": _MBA_ATLAS, + "Homo sapiens": _HBA_ATLAS, +} + +#: Canonical species names for which an offline brain-atlas lookup is available. +SUPPORTED_ATLAS_SPECIES: frozenset = frozenset(_SPECIES_TO_ATLAS) + +#: Acronym -> :class:`BrainRegionTerm` for the Allen Mouse Brain Atlas. +MBA_TERMS: dict[str, BrainRegionTerm] = _MBA_ATLAS.terms +#: Acronym -> :class:`BrainRegionTerm` for the Allen Human Brain Atlas. +HBA_TERMS: dict[str, BrainRegionTerm] = _HBA_ATLAS.terms + + +def _atlas_for_species(species: str | None) -> _BrainAtlas | None: + """Return the brain atlas for a species value (resolving common names), or ``None``.""" + species_term = get_species_term(species) + canonical_name = species_term.canonical_name if species_term is not None else species + return _SPECIES_TO_ATLAS.get(canonical_name) + + +def get_brain_region_term(location: str, species: str = "Mus musculus") -> BrainRegionTerm | None: + """ + Resolve a free-text location string to an Allen brain-atlas term for a species. + + The lookup is high-precision: it matches an exact Allen acronym (case-sensitive, e.g. + ``"CA1"``), a canonical structure name (case-insensitive, e.g. ``"caudoputamen"``), or a small + set of common informal names and abbreviations (e.g. ``"hippocampus"``, ``"V1"``). Anything it + does not recognize returns ``None``. + + The atlas is chosen from ``species``: ``"Mus musculus"`` (default) uses the Allen Mouse Brain + Atlas and ``"Homo sapiens"`` the Allen Human Brain Atlas. Common names (e.g. ``"mouse"``, + ``"human"``) are accepted. A species without a supported atlas returns ``None``. + + Parameters + ---------- + location : str + The anatomical location string as written to an NWB ``location`` field. + species : str, default: "Mus musculus" + The subject species selecting which atlas to resolve against. + + Returns + ------- + BrainRegionTerm or None + The recognized atlas term, or ``None`` when the string cannot be resolved. + """ + if not isinstance(location, str): + return None + atlas = _atlas_for_species(species) + if atlas is None: + return None + return atlas.resolve(location) diff --git a/src/neuroconv/tools/ontology/_external_resources.py b/src/neuroconv/tools/ontology/_external_resources.py index c3a4c9947..146eef843 100644 --- a/src/neuroconv/tools/ontology/_external_resources.py +++ b/src/neuroconv/tools/ontology/_external_resources.py @@ -2,8 +2,9 @@ HERD (HDMF External Resources Data) lets an NWB file carry machine-readable links from its metadata values to entities in external ontologies. NeuroConv uses it to annotate values it -can recognize -- currently ``Subject.species`` -> NCBITaxon -- so downstream tools (e.g. the -DANDI archive) can resolve the term without guessing. +can recognize -- ``Subject.species`` -> NCBITaxon, and (for mice) anatomical ``location`` +fields -> Allen Mouse Brain Atlas -- so downstream tools (e.g. the DANDI archive) can resolve +the term without guessing. The reference is stored in-file under ``/general/external_resources``, which requires ``pynwb >= 4.0.0`` (guaranteed by NeuroConv's dependency pin). @@ -11,9 +12,14 @@ from pynwb import NWBFile, get_type_map +from ._brain_regions import SUPPORTED_ATLAS_SPECIES, get_brain_region_term from ._species import get_species_term -__all__ = ["add_species_external_resource"] +__all__ = [ + "OntologyAnnotationMixin", + "add_brain_region_external_resources", + "add_species_external_resource", +] def _species_already_annotated(herd, subject) -> bool: @@ -75,3 +81,233 @@ def add_species_external_resource(nwbfile: NWBFile) -> bool: if is_new_herd: nwbfile.external_resources = herd return True + + +def _subject_atlas_species(nwbfile: NWBFile) -> str | None: + """Canonical species name if the subject has a supported brain atlas (mouse/human), else ``None``.""" + subject = getattr(nwbfile, "subject", None) + if subject is None: + return None + species_term = get_species_term(getattr(subject, "species", None)) + if species_term is None or species_term.canonical_name not in SUPPORTED_ATLAS_SPECIES: + return None + return species_term.canonical_name + + +def _brain_region_mapping_from_metadata(metadata: dict | None) -> dict: + """Parse ``metadata["BrainRegions"]`` into ``{location string: [(entity_id, entity_uri), ...]}``. + + Each brain area maps to one or more ontology terms, each given as a ``dict`` with an ``id`` + (a CURIE such as ``"MBA:382"`` or ``"UBERON:0003881"``) and a resolvable ``uri``. A single + ``dict`` or a list of them is accepted, so one area can be annotated with several ontologies + (e.g. both MBA and UBERON). This representation is ontology-agnostic, so it applies to any + species, not just mouse. + """ + if not isinstance(metadata, dict): + return {} + raw_mapping = metadata.get("BrainRegions") + if not isinstance(raw_mapping, dict): + return {} + + mapping = {} + for location, value in raw_mapping.items(): + terms = value if isinstance(value, list) else [value] + entities = [] + for term in terms: + if not isinstance(term, dict): + raise TypeError( + f"Each metadata['BrainRegions'] term must be a dict with 'id' and 'uri' keys; " + f"got {type(term).__name__} for brain area {location!r}." + ) + entity_id = term.get("id") + entity_uri = term.get("uri") + if not entity_id or not entity_uri: + raise ValueError( + f"Each metadata['BrainRegions'] term for brain area {location!r} must define " + "both 'id' and 'uri'." + ) + entities.append((str(entity_id), str(entity_uri))) + mapping[location] = entities + return mapping + + +def _brain_region_annotation_sites(nwbfile: NWBFile) -> list: + """Collect ``(container, attribute, relative_path, location string)`` tuples to annotate. + + Covers the electrodes table ``location`` column (ecephys), each ``ElectrodeGroup.location``, + and each ``ImagingPlane.location`` (ophys). Duplicate location strings within the electrodes + column are collapsed to one reference per column. + + ``container`` is the object HERD records the reference against (the ``location`` column, a + ``VectorData``, for the electrodes table; the group / plane itself otherwise). ``attribute`` + and ``relative_path`` are how that value is addressed for :meth:`HERD.add_ref` / + :meth:`HERD.get_key` -- ``None`` / ``""`` for the standalone column, and ``"location"`` for the + scalar attribute of a group or plane. + """ + sites = [] + + electrodes = nwbfile.electrodes + if electrodes is not None and "location" in electrodes.colnames: + location_column = electrodes["location"] + for location in dict.fromkeys(location_column.data): # unique, order-preserving + sites.append((location_column, None, "", location)) + + for electrode_group in nwbfile.electrode_groups.values(): + sites.append((electrode_group, "location", "location", electrode_group.location)) + + for imaging_plane in nwbfile.imaging_planes.values(): + sites.append((imaging_plane, "location", "location", imaging_plane.location)) + + return sites + + +def _existing_external_resource_refs(herd) -> set: + """The ``(object_id, key, entity_id)`` references already present in ``herd`` (idempotency).""" + if len(herd.entities[:]) == 0: + return set() + dataframe = herd.to_dataframe() + return set(zip(dataframe["object_id"].tolist(), dataframe["key"].tolist(), dataframe["entity_id"].tolist())) + + +def _find_existing_key(herd, container, relative_path: str, key_string: str): + """Return the ``Key`` already recorded for ``(container, relative_path, key_string)``, or ``None``.""" + try: + key = herd.get_key(key_string, container=container, relative_path=relative_path) + except ValueError: + return None + if isinstance(key, list): + return key[0] if key else None + return key + + +def add_brain_region_external_resources(nwbfile: NWBFile, metadata: dict | None = None) -> int: + """ + Annotate anatomical ``location`` fields with brain-region ontology entities via HERD. + + Resolves each ``location`` string on the electrodes table, electrode groups, and imaging planes + to one or more ontology terms and attaches machine-readable references (stored in-file under + ``/general/external_resources``). Each location is resolved by: + + 1. the ``metadata["BrainRegions"]`` mapping, if it provides an entry (this takes precedence and + is ontology-agnostic, so it applies to any species and may map one area to several terms, + e.g. both MBA and UBERON); then + 2. the offline Allen brain-atlas lookup for the subject's species -- the Allen Mouse Brain Atlas + for *Mus musculus* and the Allen Human Brain Atlas for *Homo sapiens*. + + Locations resolving to neither are left untouched. This is a no-op (returns ``0``) when the + subject's species has no supported atlas and no metadata mapping is provided. + + Parameters + ---------- + nwbfile : NWBFile + The file whose anatomical locations should be annotated. Modified in place. + metadata : dict, optional + Conversion metadata. ``metadata["BrainRegions"]`` maps a brain area (location string) to a + term ``{"id": ..., "uri": ...}`` or a list of such terms. + + Returns + ------- + int + The number of external-resource references added. + """ + custom_mapping = _brain_region_mapping_from_metadata(metadata) + atlas_species = _subject_atlas_species(nwbfile) + if not custom_mapping and atlas_species is None: + return 0 + + from hdmf.common import HERD + + herd = nwbfile.external_resources + is_new_herd = herd is None + if is_new_herd: + herd = HERD(type_map=get_type_map()) + + already_annotated = _existing_external_resource_refs(herd) + number_added = 0 + for container, attribute, relative_path, location in _brain_region_annotation_sites(nwbfile): + if not isinstance(location, str) or location.strip() == "": + continue + + entities = custom_mapping.get(location) + if entities is None and atlas_species is not None: + term = get_brain_region_term(location, species=atlas_species) + entities = [(term.curie, term.entity_uri)] if term is not None else None + if not entities: + continue + + # All terms for a given location share one HERD key; reuse the key object across the + # location's entities so a single object<->key link carries every ontology reference. + key = None + for entity_id, entity_uri in entities: + if (container.object_id, location, entity_id) in already_annotated: + continue + if key is None: + key = _find_existing_key(herd, container, relative_path, location) + if key is None: + herd.add_ref( + container=container, attribute=attribute, key=location, entity_id=entity_id, entity_uri=entity_uri + ) + key = herd.get_key(location, container=container, relative_path=relative_path) + else: + herd.add_ref( + container=container, attribute=attribute, key=key, entity_id=entity_id, entity_uri=entity_uri + ) + already_annotated.add((container.object_id, location, entity_id)) + number_added += 1 + + if number_added > 0 and is_new_herd: + nwbfile.external_resources = herd + return number_added + + +class OntologyAnnotationMixin: + """Mixin adding overridable hooks that annotate a written file with ontology references (HERD). + + ``BaseDataInterface`` and ``NWBConverter`` inherit this. Each hook is called once the + interface/converter data has been added to the file, and delegates to the corresponding + ``neuroconv.tools.ontology`` function by default. Override a method in a subclass to customize + or disable a particular annotation (e.g. use a different brain atlas, or turn off species + annotation). + """ + + def add_species_external_resource(self, nwbfile: NWBFile, metadata: dict | None = None) -> bool: + """ + Attach a species (NCBITaxon) reference for the subject to ``nwbfile`` (HERD). + + Override to customize. The default implementation delegates to + :func:`neuroconv.tools.ontology.add_species_external_resource`. + + Parameters + ---------- + nwbfile : NWBFile + The populated file to annotate, modified in place. + metadata : dict, optional + Conversion metadata (unused by the default implementation; available to overrides). + + Returns + ------- + bool + Whether a reference was added. + """ + return add_species_external_resource(nwbfile) + + def add_brain_region_external_resources(self, nwbfile: NWBFile, metadata: dict | None = None) -> int: + """ + Attach brain-region ontology references to ``nwbfile`` (HERD). Override to customize. + + The default implementation delegates to + :func:`neuroconv.tools.ontology.add_brain_region_external_resources`. + + Parameters + ---------- + nwbfile : NWBFile + The populated file to annotate, modified in place. + metadata : dict, optional + Conversion metadata (see the delegated function for the ``"BrainRegions"`` mapping). + + Returns + ------- + int + The number of external-resource references added. + """ + return add_brain_region_external_resources(nwbfile, metadata=metadata) diff --git a/src/neuroconv/tools/ontology/_species.py b/src/neuroconv/tools/ontology/_species.py index 07dc07e2a..c2f7dcc52 100644 --- a/src/neuroconv/tools/ontology/_species.py +++ b/src/neuroconv/tools/ontology/_species.py @@ -15,6 +15,8 @@ import warnings from dataclasses import dataclass +from ._term_sets import load_term_set + __all__ = [ "SPECIES_TERMS", "SpeciesTerm", @@ -30,43 +32,13 @@ class SpeciesTerm: canonical_name: str ncbitaxon_id: str - - @property - def entity_uri(self) -> str: - """Resolvable URI for the NCBITaxon entity (usable as a HERD ``entity_uri``).""" - taxon_number = self.ncbitaxon_id.split(":", 1)[1] - return f"http://purl.obolibrary.org/obo/NCBITaxon_{taxon_number}" + entity_uri: str # resolvable URI for the NCBITaxon entity (usable as a HERD ``entity_uri``) -# Canonical Latin binomial -> NCBITaxon identifier (CURIE) for common neuroscience species. +# Canonical Latin binomial -> SpeciesTerm, from the curated species TermSet. SPECIES_TERMS: dict[str, SpeciesTerm] = { - canonical_name: SpeciesTerm(canonical_name=canonical_name, ncbitaxon_id=ncbitaxon_id) - for canonical_name, ncbitaxon_id in { - "Mus musculus": "NCBITaxon:10090", - "Rattus norvegicus": "NCBITaxon:10116", - "Homo sapiens": "NCBITaxon:9606", - "Macaca mulatta": "NCBITaxon:9544", - "Macaca fascicularis": "NCBITaxon:9541", - "Callithrix jacchus": "NCBITaxon:9483", - "Mustela putorius furo": "NCBITaxon:9669", - "Danio rerio": "NCBITaxon:7955", - "Drosophila melanogaster": "NCBITaxon:7227", - "Caenorhabditis elegans": "NCBITaxon:6239", - "Gallus gallus": "NCBITaxon:9031", - "Sus scrofa": "NCBITaxon:9823", - "Oryctolagus cuniculus": "NCBITaxon:9986", - "Cavia porcellus": "NCBITaxon:10141", - "Felis catus": "NCBITaxon:9685", - "Canis lupus familiaris": "NCBITaxon:9615", - "Ovis aries": "NCBITaxon:9940", - "Mesocricetus auratus": "NCBITaxon:10036", - "Monodelphis domestica": "NCBITaxon:13616", - "Taeniopygia guttata": "NCBITaxon:59729", - "Xenopus laevis": "NCBITaxon:8355", - "Ambystoma mexicanum": "NCBITaxon:8296", - "Carassius auratus": "NCBITaxon:7957", - "Apis mellifera": "NCBITaxon:7460", - }.items() + info.value: SpeciesTerm(canonical_name=info.value, ncbitaxon_id=info.curie, entity_uri=info.entity_uri) + for info in load_term_set("species.yaml").values() } diff --git a/src/neuroconv/tools/ontology/_term_sets.py b/src/neuroconv/tools/ontology/_term_sets.py new file mode 100644 index 000000000..b08ac3cdb --- /dev/null +++ b/src/neuroconv/tools/ontology/_term_sets.py @@ -0,0 +1,62 @@ +"""Read the curated ontology term sets shipped with NeuroConv. + +Each controlled vocabulary (species, mouse brain atlas, human brain atlas) is stored as a +`LinkML `_ TermSet YAML file under ``term_sets/`` -- the same format used by +HDMF's :class:`~hdmf.term_set.TermSet` (https://hdmf.readthedocs.io/en/stable/tutorials/plot_term_set.html). +These files are the source of truth mapping each permissible value to an ontology entity. + +The format is small and stable, so NeuroConv reads it directly with PyYAML rather than taking a +dependency on ``linkml-runtime``; the files remain valid TermSets and can be loaded with +``hdmf.term_set.TermSet(term_schema_path=...)`` by anyone who wants to. +""" + +from dataclasses import dataclass +from functools import lru_cache +from pathlib import Path + +import yaml + +_TERM_SET_DIRECTORY = Path(__file__).parent / "term_sets" + + +@dataclass(frozen=True) +class TermInfo: + """One permissible value of a TermSet and its resolved ontology entity.""" + + value: str # the permissible value (e.g. "Mus musculus" or "CA1") + curie: str # the entity CURIE from the term's ``meaning`` (e.g. "NCBITaxon:10090") + entity_uri: str # the resolvable entity URI (the CURIE expanded via the schema ``prefixes``) + description: str + + +@lru_cache(maxsize=None) +def load_term_set(file_name: str) -> dict[str, TermInfo]: + """ + Load a LinkML TermSet YAML file into a ``{permissible value: TermInfo}`` mapping. + + Parameters + ---------- + file_name : str + The TermSet file name under ``neuroconv/tools/ontology/term_sets`` (e.g. ``"species.yaml"``). + + Returns + ------- + dict of str to TermInfo + The permissible values, each with its CURIE and expanded entity URI. + """ + with open(_TERM_SET_DIRECTORY / file_name, encoding="utf-8") as file: + schema = yaml.safe_load(file) + + prefixes = schema.get("prefixes", {}) + # A NeuroConv TermSet defines a single enumeration of permissible values. + (enumeration,) = schema["enums"].values() + + term_set = {} + for value, term in enumeration["permissible_values"].items(): + curie = term["meaning"] + prefix, local_identifier = curie.split(":", 1) + entity_uri = prefixes[prefix] + local_identifier + term_set[value] = TermInfo( + value=value, curie=curie, entity_uri=entity_uri, description=term.get("description", "") + ) + return term_set diff --git a/src/neuroconv/tools/ontology/term_sets/human_brain_atlas.yaml b/src/neuroconv/tools/ontology/term_sets/human_brain_atlas.yaml new file mode 100644 index 000000000..198bf382e --- /dev/null +++ b/src/neuroconv/tools/ontology/term_sets/human_brain_atlas.yaml @@ -0,0 +1,194 @@ +id: https://neuroconv.readthedocs.io/termset/humanbrainatlas +name: HumanBrainAtlas +version: 0.0.1 +prefixes: + HBA: https://purl.brain-bican.org/ontology/hbao/HBA_ +imports: +- linkml:types +default_range: string +enums: + BrainRegion: + permissible_values: + Tel: + description: telencephalon + meaning: HBA:4007 + Cx: + description: cerebral cortex + meaning: HBA:4008 + FL: + description: frontal lobe + meaning: HBA:4009 + PL: + description: parietal lobe + meaning: HBA:4084 + TL: + description: temporal lobe + meaning: HBA:4132 + OL: + description: occipital lobe + meaning: HBA:4180 + LL: + description: limbic lobe + meaning: HBA:4219 + Ins: + description: insula + meaning: HBA:4268 + DiE: + description: diencephalon + meaning: HBA:4391 + MES: + description: mesencephalon + meaning: HBA:9001 + MET: + description: metencephalon + meaning: HBA:4833 + MY: + description: myelencephalon + meaning: HBA:9512 + PrG: + description: precentral gyrus + meaning: HBA:4010 + SFG: + description: superior frontal gyrus + meaning: HBA:4021 + MFG: + description: middle frontal gyrus + meaning: HBA:4028 + IFG: + description: inferior frontal gyrus + meaning: HBA:4035 + GRe: + description: gyrus rectus + meaning: HBA:4047 + PoG: + description: postcentral gyrus + meaning: HBA:4085 + SMG: + description: supramarginal gyrus + meaning: HBA:4104 + AnG: + description: angular gyrus + meaning: HBA:4111 + PCu: + description: precuneus + meaning: HBA:4118 + STG: + description: superior temporal gyrus + meaning: HBA:4133 + MTG: + description: middle temporal gyrus + meaning: HBA:4140 + ITG: + description: inferior temporal gyrus + meaning: HBA:4147 + FuG: + description: fusiform gyrus + meaning: HBA:4156 + Cun: + description: cuneus + meaning: HBA:4184 + LiG: + description: lingual gyrus + meaning: HBA:4191 + CgG: + description: cingulate gyrus + meaning: HBA:4220 + PHG: + description: parahippocampal gyrus + meaning: HBA:4242 + HiF: + description: hippocampal formation + meaning: HBA:4249 + DG: + description: dentate gyrus + meaning: HBA:12891 + CA1: + description: CA1 field + meaning: HBA:12892 + CA2: + description: CA2 field + meaning: HBA:12893 + CA3: + description: CA3 field + meaning: HBA:12894 + CA4: + description: CA4 field + meaning: HBA:12895 + S: + description: subiculum + meaning: HBA:12896 + Cd: + description: caudate nucleus + meaning: HBA:4278 + Pu: + description: putamen + meaning: HBA:4287 + Acb: + description: nucleus accumbens + meaning: HBA:4290 + GP: + description: globus pallidus + meaning: HBA:4293 + BF: + description: basal forebrain + meaning: HBA:4300 + Cl: + description: claustrum + meaning: HBA:4321 + SptN: + description: septal nuclei + meaning: HBA:13002 + SI: + description: substantia innominata + meaning: HBA:13003 + Amg: + description: amygdala + meaning: HBA:4327 + CeA: + description: central nucleus + meaning: HBA:4359 + TH: + description: thalamus + meaning: HBA:4392 + DT: + description: dorsal thalamus + meaning: HBA:4393 + Hy: + description: hypothalamus + meaning: HBA:4540 + MB: + description: mammillary body + meaning: HBA:12909 + PIN: + description: pineal gland + meaning: HBA:4532 + SN: + description: substantia nigra + meaning: HBA:9072 + VTA: + description: ventral tegmental area + meaning: HBA:9066 + RN: + description: red nucleus + meaning: HBA:9053 + Pons: + description: pons + meaning: HBA:9131 + LC: + description: locus ceruleus + meaning: HBA:9148 + IO: + description: inferior olivary complex + meaning: HBA:9560 + Cb: + description: cerebellum + meaning: HBA:4696 + CbCx: + description: cerebellar cortex + meaning: HBA:4697 + Dt: + description: dentate nucleus + meaning: HBA:12946 + cc: + description: corpus callosum + meaning: HBA:9222 diff --git a/src/neuroconv/tools/ontology/term_sets/mouse_brain_atlas.yaml b/src/neuroconv/tools/ontology/term_sets/mouse_brain_atlas.yaml new file mode 100644 index 000000000..fd77bb0db --- /dev/null +++ b/src/neuroconv/tools/ontology/term_sets/mouse_brain_atlas.yaml @@ -0,0 +1,305 @@ +id: https://neuroconv.readthedocs.io/termset/mousebrainatlas +name: MouseBrainAtlas +version: 0.0.1 +prefixes: + MBA: https://purl.brain-bican.org/ontology/mbao/MBA_ +imports: +- linkml:types +default_range: string +enums: + BrainRegion: + permissible_values: + root: + description: root + meaning: MBA:997 + grey: + description: Basic cell groups and regions + meaning: MBA:8 + CH: + description: Cerebrum + meaning: MBA:567 + BS: + description: Brain stem + meaning: MBA:343 + CTX: + description: Cerebral cortex + meaning: MBA:688 + CNU: + description: Cerebral nuclei + meaning: MBA:623 + Isocortex: + description: Isocortex + meaning: MBA:315 + MOp: + description: Primary motor area + meaning: MBA:985 + MOs: + description: Secondary motor area + meaning: MBA:993 + SSp: + description: Primary somatosensory area + meaning: MBA:322 + SSs: + description: Supplemental somatosensory area + meaning: MBA:378 + SSp-bfd: + description: Primary somatosensory area, barrel field + meaning: MBA:329 + VISp: + description: Primary visual area + meaning: MBA:385 + VISl: + description: Lateral visual area + meaning: MBA:409 + VISal: + description: Anterolateral visual area + meaning: MBA:402 + VISam: + description: Anteromedial visual area + meaning: MBA:394 + VISpm: + description: posteromedial visual area + meaning: MBA:533 + VISrl: + description: Rostrolateral visual area + meaning: MBA:417 + VISpl: + description: Posterolateral visual area + meaning: MBA:425 + VISpor: + description: Postrhinal area + meaning: MBA:312782628 + AUDp: + description: Primary auditory area + meaning: MBA:1002 + AUDd: + description: Dorsal auditory area + meaning: MBA:1011 + ACAd: + description: Anterior cingulate area, dorsal part + meaning: MBA:39 + ACAv: + description: Anterior cingulate area, ventral part + meaning: MBA:48 + PL: + description: Prelimbic area + meaning: MBA:972 + ILA: + description: Infralimbic area + meaning: MBA:44 + ORBl: + description: Orbital area, lateral part + meaning: MBA:723 + ORBm: + description: Orbital area, medial part + meaning: MBA:731 + RSPd: + description: Retrosplenial area, dorsal part + meaning: MBA:879 + RSPv: + description: Retrosplenial area, ventral part + meaning: MBA:886 + AId: + description: Agranular insular area, dorsal part + meaning: MBA:104 + TEa: + description: Temporal association areas + meaning: MBA:541 + PERI: + description: Perirhinal area + meaning: MBA:922 + ECT: + description: Ectorhinal area + meaning: MBA:895 + GU: + description: Gustatory areas + meaning: MBA:1057 + VISC: + description: Visceral area + meaning: MBA:677 + PTLp: + description: Posterior parietal association areas + meaning: MBA:22 + HPF: + description: Hippocampal formation + meaning: MBA:1089 + HIP: + description: Hippocampal region + meaning: MBA:1080 + CA1: + description: Field CA1 + meaning: MBA:382 + CA2: + description: Field CA2 + meaning: MBA:423 + CA3: + description: Field CA3 + meaning: MBA:463 + DG: + description: Dentate gyrus + meaning: MBA:726 + SUB: + description: Subiculum + meaning: MBA:502 + ProS: + description: Prosubiculum + meaning: MBA:484682470 + POST: + description: Postsubiculum + meaning: MBA:1037 + PRE: + description: Presubiculum + meaning: MBA:1084 + PAR: + description: Parasubiculum + meaning: MBA:843 + ENT: + description: Entorhinal area + meaning: MBA:909 + ENTl: + description: Entorhinal area, lateral part + meaning: MBA:918 + ENTm: + description: Entorhinal area, medial part, dorsal zone + meaning: MBA:926 + OLF: + description: Olfactory areas + meaning: MBA:698 + MOB: + description: Main olfactory bulb + meaning: MBA:507 + PIR: + description: Piriform area + meaning: MBA:961 + AON: + description: Anterior olfactory nucleus + meaning: MBA:159 + BLA: + description: Basolateral amygdalar nucleus + meaning: MBA:295 + CEA: + description: Central amygdalar nucleus + meaning: MBA:536 + MEA: + description: Medial amygdalar nucleus + meaning: MBA:403 + LA: + description: Lateral amygdalar nucleus + meaning: MBA:131 + STR: + description: Striatum + meaning: MBA:477 + CP: + description: Caudoputamen + meaning: MBA:672 + ACB: + description: Nucleus accumbens + meaning: MBA:56 + PAL: + description: Pallidum + meaning: MBA:803 + LSr: + description: Lateral septal nucleus, rostral (rostroventral) part + meaning: MBA:258 + TH: + description: Thalamus + meaning: MBA:549 + VPM: + description: Ventral posteromedial nucleus of the thalamus + meaning: MBA:733 + VPL: + description: Ventral posterolateral nucleus of the thalamus + meaning: MBA:718 + VM: + description: Ventral medial nucleus of the thalamus + meaning: MBA:685 + VAL: + description: Ventral anterior-lateral complex of the thalamus + meaning: MBA:629 + MD: + description: Mediodorsal nucleus of thalamus + meaning: MBA:362 + LGd: + description: Dorsal part of the lateral geniculate complex + meaning: MBA:170 + LP: + description: Lateral posterior nucleus of the thalamus + meaning: MBA:218 + PO: + description: Posterior complex of the thalamus + meaning: MBA:1020 + RT: + description: Reticular nucleus of the thalamus + meaning: MBA:262 + HY: + description: Hypothalamus + meaning: MBA:1097 + LHA: + description: Lateral hypothalamic area + meaning: MBA:194 + PVH: + description: Paraventricular hypothalamic nucleus + meaning: MBA:38 + ZI: + description: Zona incerta + meaning: MBA:797 + MB: + description: Midbrain + meaning: MBA:313 + SCs: + description: Superior colliculus, sensory related + meaning: MBA:302 + SCm: + description: Superior colliculus, motor related + meaning: MBA:294 + IC: + description: Inferior colliculus + meaning: MBA:4 + PAG: + description: Periaqueductal gray + meaning: MBA:795 + VTA: + description: Ventral tegmental area + meaning: MBA:749 + SNr: + description: Substantia nigra, reticular part + meaning: MBA:381 + SNc: + description: Substantia nigra, compact part + meaning: MBA:374 + RN: + description: Red nucleus + meaning: MBA:214 + APN: + description: Anterior pretectal nucleus + meaning: MBA:215 + MRN: + description: Midbrain reticular nucleus + meaning: MBA:128 + P: + description: Pons + meaning: MBA:771 + PB: + description: Parabrachial nucleus + meaning: MBA:867 + PG: + description: Pontine gray + meaning: MBA:931 + DR: + description: Dorsal nucleus raphe + meaning: MBA:872 + LC: + description: Locus ceruleus + meaning: MBA:147 + MY: + description: Medulla + meaning: MBA:354 + NTS: + description: Nucleus of the solitary tract + meaning: MBA:651 + IO: + description: Inferior olivary complex + meaning: MBA:83 + CB: + description: Cerebellum + meaning: MBA:512 diff --git a/src/neuroconv/tools/ontology/term_sets/species.yaml b/src/neuroconv/tools/ontology/term_sets/species.yaml new file mode 100644 index 000000000..892bc6234 --- /dev/null +++ b/src/neuroconv/tools/ontology/term_sets/species.yaml @@ -0,0 +1,84 @@ +id: https://neuroconv.readthedocs.io/termset/species +name: Species +version: 0.0.1 +prefixes: + NCBITaxon: http://purl.obolibrary.org/obo/NCBITaxon_ +imports: + - linkml:types +default_range: string + +enums: + Species: + permissible_values: + Mus musculus: + description: house mouse + meaning: NCBITaxon:10090 + Rattus norvegicus: + description: Norway rat + meaning: NCBITaxon:10116 + Homo sapiens: + description: human + meaning: NCBITaxon:9606 + Macaca mulatta: + description: rhesus macaque + meaning: NCBITaxon:9544 + Macaca fascicularis: + description: crab-eating macaque + meaning: NCBITaxon:9541 + Callithrix jacchus: + description: common marmoset + meaning: NCBITaxon:9483 + Mustela putorius furo: + description: domestic ferret + meaning: NCBITaxon:9669 + Danio rerio: + description: zebrafish + meaning: NCBITaxon:7955 + Drosophila melanogaster: + description: fruit fly + meaning: NCBITaxon:7227 + Caenorhabditis elegans: + description: roundworm + meaning: NCBITaxon:6239 + Gallus gallus: + description: chicken + meaning: NCBITaxon:9031 + Sus scrofa: + description: pig + meaning: NCBITaxon:9823 + Oryctolagus cuniculus: + description: rabbit + meaning: NCBITaxon:9986 + Cavia porcellus: + description: guinea pig + meaning: NCBITaxon:10141 + Felis catus: + description: domestic cat + meaning: NCBITaxon:9685 + Canis lupus familiaris: + description: domestic dog + meaning: NCBITaxon:9615 + Ovis aries: + description: sheep + meaning: NCBITaxon:9940 + Mesocricetus auratus: + description: golden hamster + meaning: NCBITaxon:10036 + Monodelphis domestica: + description: gray short-tailed opossum + meaning: NCBITaxon:13616 + Taeniopygia guttata: + description: zebra finch + meaning: NCBITaxon:59729 + Xenopus laevis: + description: African clawed frog + meaning: NCBITaxon:8355 + Ambystoma mexicanum: + description: axolotl + meaning: NCBITaxon:8296 + Carassius auratus: + description: goldfish + meaning: NCBITaxon:7957 + Apis mellifera: + description: western honey bee + meaning: NCBITaxon:7460 diff --git a/tests/test_minimal/test_tools/test_ontology_brain_region_external_resources.py b/tests/test_minimal/test_tools/test_ontology_brain_region_external_resources.py new file mode 100644 index 000000000..2845919eb --- /dev/null +++ b/tests/test_minimal/test_tools/test_ontology_brain_region_external_resources.py @@ -0,0 +1,317 @@ +"""Tests for attaching Allen brain-atlas references (mouse MBA / human HBA) via HERD.""" + +from datetime import datetime + +import pytest +from dateutil.tz import tzutc +from pynwb import NWBFile +from pynwb.file import Subject + +from neuroconv.tools.ontology import add_brain_region_external_resources + + +def _make_nwbfile(species="Mus musculus", with_subject=True) -> NWBFile: + nwbfile = NWBFile( + session_description="d", + identifier="id", + session_start_time=datetime(2020, 1, 1, tzinfo=tzutc()), + ) + if with_subject: + nwbfile.subject = Subject(subject_id="s1", species=species) + return nwbfile + + +def _add_electrodes(nwbfile: NWBFile, locations) -> None: + device = nwbfile.create_device(name="probe") + group = nwbfile.create_electrode_group(name="group0", description="d", location="unknown", device=device) + for index, location in enumerate(locations): + nwbfile.add_electrode(location=location, group=group, id=index) + + +class TestNoOps: + def test_no_subject(self): + nwbfile = _make_nwbfile(with_subject=False) + _add_electrodes(nwbfile, ["CA1"]) + assert add_brain_region_external_resources(nwbfile) == 0 + assert nwbfile.external_resources is None + + def test_species_without_atlas_is_skipped(self): + # Rat has no offline atlas (only mouse and human are supported). + nwbfile = _make_nwbfile(species="Rattus norvegicus") + _add_electrodes(nwbfile, ["CA1", "VISp"]) + assert add_brain_region_external_resources(nwbfile) == 0 + assert nwbfile.external_resources is None + + def test_no_recognized_locations(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["unknown", "not a region"]) + assert add_brain_region_external_resources(nwbfile) == 0 + assert nwbfile.external_resources is None + + +class TestElectrodeAnnotation: + def test_distinct_locations_are_annotated_once_each(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1", "CA1", "VISp", "unknown"]) + + # "mouse" common name also gates as Mus musculus. + assert add_brain_region_external_resources(nwbfile) == 2 + + dataframe = nwbfile.external_resources.to_dataframe() + assert sorted(dataframe["key"].tolist()) == ["CA1", "VISp"] + by_key = dict(zip(dataframe["key"], dataframe["entity_id"])) + assert by_key == {"CA1": "MBA:382", "VISp": "MBA:385"} + uris = dict(zip(dataframe["key"], dataframe["entity_uri"])) + assert uris["CA1"] == "https://purl.brain-bican.org/ontology/mbao/MBA_382" + + def test_reference_points_at_electrodes_location_column(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1"]) + add_brain_region_external_resources(nwbfile) + + # HERD records the reference against the ``location`` column (a VectorData), not the table. + objects = nwbfile.external_resources.objects.to_dataframe() + assert nwbfile.electrodes["location"].object_id in objects["object_id"].tolist() + + def test_common_name_subject_still_resolves_as_mouse(self): + nwbfile = _make_nwbfile(species="mouse") + _add_electrodes(nwbfile, ["CA1"]) + assert add_brain_region_external_resources(nwbfile) == 1 + + +class TestHumanAtlasAnnotation: + def test_human_locations_are_annotated_with_hba(self): + nwbfile = _make_nwbfile(species="Homo sapiens") + _add_electrodes(nwbfile, ["CA1", "cerebral cortex", "unknown"]) + + assert add_brain_region_external_resources(nwbfile) == 2 + dataframe = nwbfile.external_resources.to_dataframe() + by_key = dict(zip(dataframe["key"], dataframe["entity_id"])) + # Same "CA1" acronym resolves to the human atlas id, not the mouse one. + assert by_key == {"CA1": "HBA:12892", "cerebral cortex": "HBA:4008"} + uris = dict(zip(dataframe["key"], dataframe["entity_uri"])) + assert uris["CA1"] == "https://purl.brain-bican.org/ontology/hbao/HBA_12892" + + +class TestElectrodeGroupAndImagingPlaneAnnotation: + def test_electrode_group_location_is_annotated(self): + nwbfile = _make_nwbfile() + device = nwbfile.create_device(name="probe") + nwbfile.create_electrode_group(name="g0", description="d", location="VISp", device=device) + + assert add_brain_region_external_resources(nwbfile) == 1 + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["key"].tolist() == ["VISp"] + assert dataframe["entity_id"].tolist() == ["MBA:385"] + + def test_imaging_plane_location_is_annotated(self): + nwbfile = _make_nwbfile() + device = nwbfile.create_device(name="scope") + optical_channel = _optical_channel() + nwbfile.create_imaging_plane( + name="plane0", + optical_channel=optical_channel, + description="d", + device=device, + excitation_lambda=600.0, + indicator="GCaMP", + location="CA1", + imaging_rate=30.0, + ) + + assert add_brain_region_external_resources(nwbfile) == 1 + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["key"].tolist() == ["CA1"] + assert dataframe["entity_id"].tolist() == ["MBA:382"] + + +class TestMetadataMapping: + def test_metadata_mapping_annotates_unrecognized_location(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["my special area"]) + metadata = { + "BrainRegions": { + "my special area": {"id": "MBA:382", "uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382"} + } + } + + assert add_brain_region_external_resources(nwbfile, metadata=metadata) == 1 + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["key"].tolist() == ["my special area"] + assert dataframe["entity_id"].tolist() == ["MBA:382"] + assert dataframe["entity_uri"].tolist() == ["https://purl.brain-bican.org/ontology/mbao/MBA_382"] + + def test_metadata_mapping_takes_precedence_over_offline_lookup(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1"]) + # Override the offline result (MBA:382) with an explicit term. + metadata = {"BrainRegions": {"CA1": {"id": "MBA:999", "uri": "https://example.org/MBA_999"}}} + + add_brain_region_external_resources(nwbfile, metadata=metadata) + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["entity_id"].tolist() == ["MBA:999"] + + def test_metadata_and_offline_combine(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1", "my special area"]) + metadata = {"BrainRegions": {"my special area": {"id": "MBA:42", "uri": "https://example.org/MBA_42"}}} + + assert add_brain_region_external_resources(nwbfile, metadata=metadata) == 2 + dataframe = nwbfile.external_resources.to_dataframe() + assert dict(zip(dataframe["key"], dataframe["entity_id"])) == { + "CA1": "MBA:382", + "my special area": "MBA:42", + } + + def test_maps_one_area_to_multiple_ontology_terms(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1"]) + metadata = { + "BrainRegions": { + "CA1": [ + {"id": "MBA:382", "uri": "https://purl.brain-bican.org/ontology/mbao/MBA_382"}, + {"id": "UBERON:0003881", "uri": "http://purl.obolibrary.org/obo/UBERON_0003881"}, + ] + } + } + + assert add_brain_region_external_resources(nwbfile, metadata=metadata) == 2 + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["key"].tolist() == ["CA1", "CA1"] + assert sorted(dataframe["entity_id"].tolist()) == ["MBA:382", "UBERON:0003881"] + + def test_metadata_mapping_applies_to_species_without_an_atlas(self): + # The explicit metadata mapping is ontology-agnostic and is not gated on species; a rat + # subject has no offline atlas, so only the metadata-defined region is annotated. + nwbfile = _make_nwbfile(species="Rattus norvegicus") + _add_electrodes(nwbfile, ["my region", "CA1"]) + metadata = { + "BrainRegions": { + "my region": {"id": "UBERON:0002436", "uri": "http://purl.obolibrary.org/obo/UBERON_0002436"} + } + } + + assert add_brain_region_external_resources(nwbfile, metadata=metadata) == 1 + dataframe = nwbfile.external_resources.to_dataframe() + assert dataframe["key"].tolist() == ["my region"] + assert dataframe["entity_id"].tolist() == ["UBERON:0002436"] + + @pytest.mark.parametrize( + "bad_value", + ["MBA:382", {"id": "MBA:382"}, {"uri": "https://example.org/1"}, {"id": "", "uri": ""}], + ) + def test_malformed_metadata_term_raises(self, bad_value): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["area"]) + metadata = {"BrainRegions": {"area": bad_value}} + with pytest.raises((TypeError, ValueError)): + add_brain_region_external_resources(nwbfile, metadata=metadata) + + +class TestHERDLifecycle: + def test_idempotent(self): + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1", "VISp"]) + assert add_brain_region_external_resources(nwbfile) == 2 + assert add_brain_region_external_resources(nwbfile) == 0 + assert len(nwbfile.external_resources.entities[:]) == 2 + + def test_extends_existing_herd_in_place(self): + from hdmf.common import HERD + from pynwb import get_type_map + + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1"]) + herd = HERD(type_map=get_type_map()) + herd.add_ref( + container=nwbfile.subject, + attribute="subject_id", + key="s1", + entity_id="EXAMPLE:1", + entity_uri="https://example.org/1", + ) + nwbfile.external_resources = herd + + assert add_brain_region_external_resources(nwbfile) == 1 + assert nwbfile.external_resources is herd # extended in place, not replaced + assert len(herd.entities[:]) == 2 + + def test_round_trips_through_file(self, tmp_path): + from pynwb import NWBHDF5IO + + nwbfile = _make_nwbfile() + _add_electrodes(nwbfile, ["CA1", "VISp"]) + add_brain_region_external_resources(nwbfile) + + path = tmp_path / "brain_region_herd.nwb" + with NWBHDF5IO(path, "w") as io: + io.write(nwbfile) + with NWBHDF5IO(path, "r") as io: + read_nwbfile = io.read() + dataframe = read_nwbfile.external_resources.to_dataframe() + + assert sorted(dataframe["key"].tolist()) == ["CA1", "VISp"] + assert set(dataframe["entity_id"].tolist()) == {"MBA:382", "MBA:385"} + + +class TestOverridableHook: + def _mouse_recording_interface(self, brain_areas): + from neuroconv.tools.testing.mock_interfaces import MockRecordingInterface + + interface = MockRecordingInterface(num_channels=len(brain_areas), durations=(0.1,)) + interface.recording_extractor.set_property("brain_area", list(brain_areas)) + return interface + + def _mouse_metadata(self, interface): + metadata = interface.get_metadata() + metadata["Subject"] = dict(subject_id="m1", species="Mus musculus", sex="M", age="P30D") + return metadata + + def test_default_hooks_annotate_species_and_brain_region_through_create_nwbfile(self): + interface = self._mouse_recording_interface(["CA1", "VISp"]) + nwbfile = interface.create_nwbfile(metadata=self._mouse_metadata(interface)) + + entity_ids = set(nwbfile.external_resources.to_dataframe()["entity_id"].tolist()) + # Species (NCBITaxon) and brain-region (MBA) references are both attached by the mixin. + assert "NCBITaxon:10090" in entity_ids + assert {"MBA:382", "MBA:385"}.issubset(entity_ids) + + def test_subclass_can_override_brain_region_hook(self): + from neuroconv.tools.testing.mock_interfaces import MockRecordingInterface + + class NoBrainRegionInterface(MockRecordingInterface): + def add_brain_region_external_resources(self, nwbfile, metadata=None): + # Override to disable brain-region annotation entirely. + return 0 + + interface = NoBrainRegionInterface(num_channels=2, durations=(0.1,)) + interface.recording_extractor.set_property("brain_area", ["CA1", "VISp"]) + nwbfile = interface.create_nwbfile(metadata=self._mouse_metadata(interface)) + + # No brain-region references were added, but the species reference still is. + entity_ids = nwbfile.external_resources.to_dataframe()["entity_id"].tolist() + assert not any(entity_id.startswith("MBA:") for entity_id in entity_ids) + assert "NCBITaxon:10090" in entity_ids + + def test_subclass_can_override_species_hook(self): + from neuroconv.tools.testing.mock_interfaces import MockRecordingInterface + + class NoSpeciesInterface(MockRecordingInterface): + def add_species_external_resource(self, nwbfile, metadata=None): + # Override to disable species annotation entirely. + return False + + interface = NoSpeciesInterface(num_channels=2, durations=(0.1,)) + interface.recording_extractor.set_property("brain_area", ["CA1", "VISp"]) + nwbfile = interface.create_nwbfile(metadata=self._mouse_metadata(interface)) + + # No species reference was added, but the brain-region references still are. + entity_ids = nwbfile.external_resources.to_dataframe()["entity_id"].tolist() + assert not any(entity_id.startswith("NCBITaxon:") for entity_id in entity_ids) + assert {"MBA:382", "MBA:385"}.issubset(set(entity_ids)) + + +def _optical_channel(): + from pynwb.ophys import OpticalChannel + + return OpticalChannel(name="channel0", description="d", emission_lambda=500.0) diff --git a/tests/test_minimal/test_tools/test_ontology_brain_regions.py b/tests/test_minimal/test_tools/test_ontology_brain_regions.py new file mode 100644 index 000000000..035e8e7a9 --- /dev/null +++ b/tests/test_minimal/test_tools/test_ontology_brain_regions.py @@ -0,0 +1,104 @@ +"""Tests for the offline Allen brain-atlas (mouse MBA / human HBA) region lookup.""" + +import pytest + +from neuroconv.tools.ontology import ( + HBA_TERMS, + MBA_TERMS, + BrainRegionTerm, + get_brain_region_term, +) + + +class TestAtlasTermTables: + @pytest.mark.parametrize("terms, prefix", [(MBA_TERMS, "MBA"), (HBA_TERMS, "HBA")]) + def test_tables_are_populated(self, terms, prefix): + assert len(terms) > 50 + + @pytest.mark.parametrize("terms, prefix", [(MBA_TERMS, "MBA"), (HBA_TERMS, "HBA")]) + def test_entries_are_self_consistent(self, terms, prefix): + for acronym, term in terms.items(): + assert isinstance(term, BrainRegionTerm) + assert term.acronym == acronym + assert term.name + assert term.curie.startswith(f"{prefix}:") + + def test_mouse_curie_and_uri(self): + term = MBA_TERMS["CA1"] + assert term.curie == "MBA:382" + assert term.entity_uri == "https://purl.brain-bican.org/ontology/mbao/MBA_382" + + def test_human_curie_and_uri(self): + term = HBA_TERMS["CA1"] + assert term.curie == "HBA:12892" + assert term.entity_uri == "https://purl.brain-bican.org/ontology/hbao/HBA_12892" + + @pytest.mark.parametrize("terms", [MBA_TERMS, HBA_TERMS]) + def test_curies_are_unique_within_an_atlas(self, terms): + curies = [term.curie for term in terms.values()] + assert len(curies) == len(set(curies)) + + +class TestGetBrainRegionTermMouse: + @pytest.mark.parametrize( + "location, expected_acronym", + [ + ("CA1", "CA1"), # exact acronym + ("VISp", "VISp"), # case-specific acronym + ("SSp-bfd", "SSp-bfd"), # acronym with a hyphen + ("Field CA1", "CA1"), # canonical name + ("caudoputamen", "CP"), # canonical name, case-insensitive + ("Nucleus accumbens", "ACB"), + ("hippocampus", "HIP"), # informal alias + ("V1", "VISp"), # abbreviation alias + ("barrel cortex", "SSp-bfd"), + ("locus coeruleus", "LC"), # alternate spelling alias + ], + ) + def test_recognized_locations(self, location, expected_acronym): + term = get_brain_region_term(location) # default species is mouse + assert term is not None + assert term.acronym == expected_acronym + assert term.curie.startswith("MBA:") + + def test_whitespace_is_stripped(self): + assert get_brain_region_term(" CA1 ").acronym == "CA1" + + @pytest.mark.parametrize("location", ["", " ", "not a region", "ca1", "visp"]) + def test_unrecognized_locations_return_none(self, location): + # Note: acronym matching is case-sensitive, so lowercase "ca1"/"visp" are not recognized. + assert get_brain_region_term(location) is None + + @pytest.mark.parametrize("location", [None, 382, ["CA1"]]) + def test_non_string_returns_none(self, location): + assert get_brain_region_term(location) is None + + +class TestGetBrainRegionTermHuman: + @pytest.mark.parametrize( + "location, expected_curie", + [ + ("CA1", "HBA:12892"), # same acronym as mouse, different atlas/id + ("cerebral cortex", "HBA:4008"), # canonical name + ("Precuneus", "HBA:4118"), # canonical name, case-insensitive + ("hippocampus", "HBA:4249"), # alias -> hippocampal formation + ("caudate", "HBA:4278"), # alias + ("nucleus accumbens", "HBA:4290"), + ], + ) + def test_recognized_human_locations(self, location, expected_curie): + term = get_brain_region_term(location, species="Homo sapiens") + assert term is not None + assert term.curie == expected_curie + + def test_same_acronym_resolves_per_species(self): + # "MB" is the mouse midbrain but the human mammillary body. + assert get_brain_region_term("MB", species="Mus musculus").curie == "MBA:313" + assert get_brain_region_term("MB", species="Homo sapiens").curie == "HBA:12909" + + def test_common_name_species_is_accepted(self): + assert get_brain_region_term("cerebral cortex", species="human").curie == "HBA:4008" + + @pytest.mark.parametrize("species", ["Rattus norvegicus", "Danio rerio", None]) + def test_species_without_atlas_returns_none(self, species): + assert get_brain_region_term("CA1", species=species) is None