HERD integration: species ontology terms

CHANGELOG.md

@@ -50,6 +50,8 @@
 * Added `TDTEventsInterface` for converting discrete events (epocs) from a TDT tank folder, depending on the `ndx-events` extension. [PR #1751](https://github.com/catalystneuro/neuroconv/pull/1751)
 * Added `VameInterface` for converting VAME behavioral segmentation data to NWB using the `ndx-vame` extension. Writes per-frame motif labels (`MotifSeries`), optional latent-space embeddings (`LatentSpaceSeries`), optional community labels (`CommunitySeries`), and serializes the VAME project config as JSON. [PR #1737](https://github.com/catalystneuro/neuroconv/pull/1737)
 * Added `AxonIntracellularInterface` for converting intracellular electrophysiology recorded in Axon Binary Format (.abf). Each instance handles one electrode in one file, writing its response as a single continuous `PatchClampSeries` (with an optional paired stimulus taken either from a recorded monitor channel or from a reconstructed protocol command) and one `IntracellularRecordings` row per sweep, addressed by `(start_index, count)` ranges and tagged with the run's `sequence` and `stimulus_type` columns. The clamp mode (`voltage_clamp`, `current_clamp`, or `izero`) is selected explicitly because Axon clamp-mode metadata is unreliable. It uses the dict-based metadata format keyed by `metadata_key`. The interface writes only the per-sweep recordings rows; the upper icephys hierarchy tables (`SimultaneousRecordings` and above) are assembled separately once the full set of channels and files is known. [PR #1746](https://github.com/catalystneuro/neuroconv/pull/1746)
+* Added `neuroconv.tools.ontology` with `validate_species`/`get_species_suggestion`, an offline recommender that maps common species names and typos to their canonical Latin binomial and NCBITaxon identifier. `make_nwbfile_from_metadata` now emits a non-blocking suggestion when `Subject.species` is a recognized common name (e.g. `"mouse"` → `Mus musculus` / `NCBITaxon:10090`) or a likely typo. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771)
+* Added HERD external-resource integration for `Subject.species`. When the species is a recognized Latin binomial, `make_nwbfile_from_metadata` now attaches a machine-readable NCBITaxon reference to the file (stored in-file under `/general/external_resources` via `add_species_external_resource`), so downstream tools can resolve the term without guessing. [PR #1771](https://github.com/catalystneuro/neuroconv/pull/1771)
 * Added `XClustSortingInterface` for converting XClust (.CEL) spike sorting data, using the `XClustSortingExtractor` from SpikeInterface. [PR #1691](https://github.com/catalystneuro/neuroconv/pull/1691)
 * Added `IntanStimInterface` for converting electrical stimulation current data from Intan RHS2000 systems. Stimulation channels (one per amplifier channel, named `{channel}_STIM`) are written as a `TimeSeries` with `unit="A"` (Amperes), with the conversion factor derived automatically from `stim_step_size` in the .rhs file header. [PR #1711](https://github.com/catalystneuro/neuroconv/pull/1711)
 * Added a `saved_files_are_split` parameter to `IntanRecordingInterface`, `IntanAnalogInterface`, and `IntanStimInterface` to support Intan RHX recordings saved with the "create a new save file every N minutes" option. When enabled, all sibling `.rhd`/`.rhs` files in the session folder are concatenated in filename order. When disabled, the interface warns if sibling rotation files are detected next to the one the user pointed at. [PR #1724](https://github.com/catalystneuro/neuroconv/pull/1724)
@@ -66,6 +68,7 @@
 * Added `metadata_key` support to recording interfaces (`BaseRecordingExtractorInterface`, inherited by `BaseLFPExtractorInterface` and all recording interfaces) and `use_new_metadata_format` parameter to `get_metadata()` for the dict-based metadata pipeline. `metadata_key` defaults to the value of `es_key`. [PR #1747](https://github.com/catalystneuro/neuroconv/pull/1747)
 
 ## Improvements
+* Added documentation for the species ontology tools: a user guide page (`docs/user_guide/species_ontology.rst`) covering the non-blocking species suggestion and the in-file NCBITaxon (HERD) annotation, plus an API reference page for `neuroconv.tools.ontology`. [PR #1772](https://github.com/catalystneuro/neuroconv/pull/1772)
 * Migrated `MockPoseEstimationInterface` to the dict-based metadata shape used by the rest of the unified pipeline: top-level `metadata["Devices"]`, with `metadata["Behavior"]["Pose"]["PoseEstimations"][metadata_key]` and `metadata["Behavior"]["Pose"]["Skeletons"][metadata_key]` sub-registries linked via `device_metadata_key` and `skeleton_metadata_key`. The constructor argument `pose_estimation_metadata_key` was renamed to `metadata_key` for consistency with the other dict-based interfaces. This anchors the target shape for the upcoming DLC, Lightning Pose, and SLEAP interface migrations. [PR #1745](https://github.com/catalystneuro/neuroconv/pull/1745)
 * `MiniscopeBehaviorInterface` no longer depends on `ndx-miniscope` for raw format parsing; `ndx-miniscope` is used only for NWB construction. Legacy V3 Miniscope folders now raise a clear error on both Miniscope interfaces pointing users to the issue tracker so V3 support can be added against real fixtures. [PR #1725](https://github.com/catalystneuro/neuroconv/pull/1725)
 * Added a conversion gallery example for aligning ScanImage imaging data with external DAQ sync pulses, demonstrating use of `get_original_frame_indices` and `slice_samples` from `ScanImageImagingExtractor`. [PR #1709](https://github.com/catalystneuro/neuroconv/pull/1709)

docs/api/tools.ontology.rst

@@ -0,0 +1,4 @@
+Ontology
+========
+
+.. automodule:: neuroconv.tools.ontology

docs/api/tools.rst

@@ -11,6 +11,7 @@ Tools
     tools.testing
     tools.iterative_write
     tools.path_expansion
+    tools.ontology
     tools.signal_processing
     tools.data_transfers
     tools.nwb_helpers

docs/user_guide/index.rst

@@ -24,6 +24,7 @@ and synchronize data across multiple sources.
   temporal_alignment
   csvs
   expand_path
+  species_ontology
   backend_configuration
   linking_sorted_data
   yaml

docs/user_guide/species_ontology.rst

@@ -0,0 +1,79 @@
+Species Ontology
+================
+
+NWB stores a subject's species in :py:attr:`Subject.species <pynwb.file.Subject.species>` as a
+binomial Latin name (e.g. ``"Mus musculus"``) or a taxonomy URL. Consistent, standardized species
+values make files interoperable and allow downstream tools such as the
+`DANDI Archive <https://dandiarchive.org/>`_ to resolve the exact organism without guessing.
+
+NeuroConv helps with this in two complementary ways, both applied automatically by
+:py:func:`~neuroconv.tools.nwb_helpers.make_nwbfile_from_metadata` (and therefore by every
+conversion that builds an NWB file through it):
+
+1. A **non-blocking suggestion** when the species looks like a common name or a typo.
+2. A **machine-readable NCBITaxon annotation** attached in-file when the species is recognized.
+
+The lookup is a small, curated, offline table of common neuroscience species
+(:py:data:`~neuroconv.tools.ontology.SPECIES_TERMS`). It requires no network access and no extra
+dependencies, and it is intentionally high-precision: it only speaks up when it is confident there
+is a better term. Valid-but-uncommon binomials pass through silently.
+
+Suggesting a standardized term
+------------------------------
+
+When ``Subject.species`` is a recognized common name (e.g. ``"mouse"``) or a likely typo of a known
+binomial (e.g. ``"Homo sapien"``), NeuroConv emits a ``UserWarning`` recommending the canonical
+Latin binomial and its NCBITaxon identifier. This never raises and never blocks a conversion.
+
+.. code-block:: python
+
+    from neuroconv.tools.ontology import validate_species
+
+    validate_species("mouse")
+    # UserWarning: Subject species 'mouse' is a common name. Consider using the Latin binomial
+    # 'Mus musculus' (NCBITaxon:10090) for interoperability. See https://bioregistry.io/NCBITaxon:10090
+
+    validate_species("Homo sapien")
+    # UserWarning: Subject species 'Homo sapien' closely matches a known species name. Consider using
+    # the Latin binomial 'Homo sapiens' (NCBITaxon:9606) for interoperability. ...
+
+    validate_species("Mus musculus")  # already canonical -> no warning, returns the term
+    validate_species("Octodon degus")  # valid but not in the table -> no warning, returns None
+
+To resolve a value to its canonical term without emitting a warning, use
+:py:func:`~neuroconv.tools.ontology.get_species_term`, which also succeeds on exact canonical
+matches:
+
+.. code-block:: python
+
+    from neuroconv.tools.ontology import get_species_term
+
+    term = get_species_term("rhesus macaque")
+    term.canonical_name  # 'Macaca mulatta'
+    term.ncbitaxon_id    # 'NCBITaxon:9544'
+    term.entity_uri      # 'http://purl.obolibrary.org/obo/NCBITaxon_9544'
+
+Annotating the file with an NCBITaxon reference
+-----------------------------------------------
+
+When the species resolves to a recognized term, NeuroConv attaches a machine-readable external
+resource reference mapping ``Subject.species`` to its NCBITaxon entity. This is stored **in-file**
+under ``/general/external_resources`` using HDMF's HERD (External Resources Data), so the link
+travels with the file:
+
+.. code-block:: python
+
+    from neuroconv.tools.ontology import add_species_external_resource
+
+    # nwbfile.subject.species == "Mus musculus"
+    added = add_species_external_resource(nwbfile)  # returns True
+    nwbfile.external_resources  # now carries a Mus musculus -> NCBITaxon:10090 reference
+
+The call is a no-op (returns ``False``) when there is no subject or the species is not recognized,
+and it is idempotent: an existing ``external_resources`` HERD is extended in place rather than
+replaced, and a species that is already annotated is not added twice.
+
+.. note::
+
+    In-file HERD storage requires ``pynwb >= 4.0.0``, which is NeuroConv's minimum supported
+    version.

src/neuroconv/tools/nwb_helpers/_metadata_and_file_helpers.py

@@ -19,6 +19,7 @@
     configure_backend,
     get_default_backend_configuration,
 )
+from ..ontology import add_species_external_resource, validate_species
 from ...utils.dict import DeepDict, load_dict_from_file
 from ...utils.json_schema import validate_metadata
 
@@ -109,6 +110,13 @@ def make_nwbfile_from_metadata(metadata: dict) -> NWBFile:
     schema_path = Path(__file__).resolve().parent.parent.parent / "schemas" / "base_metadata_schema.json"
     base_metadata_schema = load_dict_from_file(file_path=schema_path)
     assert metadata is not None, "Metadata is required to create an NWBFile but metadata=None was passed."
+
+    # Recommend a standardized species term (non-blocking). Runs before schema validation so
+    # that common names (e.g. "mouse") surface a helpful suggestion even though the schema's
+    # binomial pattern will subsequently reject them.
+    if isinstance(metadata.get("Subject"), dict):
+        validate_species(metadata["Subject"].get("species"))
+
     validate_metadata(metadata=metadata, schema=base_metadata_schema)
 
     nwbfile_kwargs = deepcopy(metadata["NWBFile"])
@@ -133,7 +141,13 @@ def make_nwbfile_from_metadata(metadata: dict) -> NWBFile:
             )
         nwbfile_kwargs["subject"] = Subject(**nwbfile_kwargs["subject"])
 
-    return NWBFile(**nwbfile_kwargs)
+    nwbfile = NWBFile(**nwbfile_kwargs)
+
+    # Attach a machine-readable NCBITaxon reference for the subject species (in-file HERD).
+    # No-op when there is no subject or the species is not recognized.
+    add_species_external_resource(nwbfile)
+
+    return nwbfile
 
 
 def add_device_from_metadata(nwbfile: NWBFile, modality: str = "Ecephys", metadata: dict | None = None):

src/neuroconv/tools/ontology/__init__.py

@@ -0,0 +1,19 @@
+"""Tools for recommending standardized ontology terms for NWB metadata."""
+
+from ._external_resources import add_species_external_resource
+from ._species import (
+    SPECIES_TERMS,
+    SpeciesTerm,
+    get_species_suggestion,
+    get_species_term,
+    validate_species,
+)
+
+__all__ = [
+    "SPECIES_TERMS",
+    "SpeciesTerm",
+    "add_species_external_resource",
+    "get_species_suggestion",
+    "get_species_term",
+    "validate_species",
+]

src/neuroconv/tools/ontology/_external_resources.py

@@ -0,0 +1,77 @@
+"""Attach ontology entity references to NWB files via HDMF's HERD.
+
+HERD (HDMF External Resources Data) lets an NWB file carry machine-readable links from its
+metadata values to entities in external ontologies. NeuroConv uses it to annotate values it
+can recognize -- currently ``Subject.species`` -> NCBITaxon -- so downstream tools (e.g. the
+DANDI archive) can resolve the term without guessing.
+
+The reference is stored in-file under ``/general/external_resources``, which requires
+``pynwb >= 4.0.0`` (guaranteed by NeuroConv's dependency pin).
+"""
+
+from pynwb import NWBFile, get_type_map
+
+from ._species import get_species_term
+
+__all__ = ["add_species_external_resource"]
+
+
+def _species_already_annotated(herd, subject) -> bool:
+    """Whether ``herd`` already has a species entity for ``subject`` (keeps the call idempotent)."""
+    try:
+        existing = herd.get_object_entities(subject, attribute="species")
+    except ValueError:
+        # Raised when the subject is not yet registered in the object table.
+        return False
+    return not existing.empty
+
+
+def add_species_external_resource(nwbfile: NWBFile) -> bool:
+    """
+    Annotate ``nwbfile.subject.species`` with its NCBITaxon entity via HERD.
+
+    Adds an external-resource reference mapping the subject's species (a Latin binomial) to its
+    NCBITaxon identifier, stored in-file under ``/general/external_resources``. This is a no-op
+    (returns ``False``) when there is no subject or the species is not recognized.
+
+    Parameters
+    ----------
+    nwbfile : NWBFile
+        The file whose subject species should be annotated. Modified in place.
+
+    Returns
+    -------
+    bool
+        ``True`` if a reference was added, ``False`` otherwise.
+    """
+    subject = getattr(nwbfile, "subject", None)
+    if subject is None:
+        return False
+
+    species = subject.species
+    term = get_species_term(species)
+    if term is None:
+        return False
+
+    from hdmf.common import HERD
+
+    herd = nwbfile.external_resources
+    is_new_herd = herd is None
+    if is_new_herd:
+        herd = HERD(type_map=get_type_map())
+    elif _species_already_annotated(herd, subject):
+        return False
+
+    herd.add_ref(
+        container=subject,
+        attribute="species",
+        key=species,
+        entity_id=term.ncbitaxon_id,
+        entity_uri=term.entity_uri,
+    )
+
+    # ``external_resources`` is write-once; only assign when we created the HERD, otherwise we
+    # have extended the object already linked to the file in place.
+    if is_new_herd:
+        nwbfile.external_resources = herd
+    return True