Merge branch 'main' into add_max_channel_for_phy

Author: bendichter

CHANGELOG.md

@@ -11,6 +11,7 @@
 * Add Plexon2 support [PR #918](https://github.com/catalystneuro/neuroconv/pull/918)
 * Converter working with multiple VideoInterface instances [PR #914](https://github.com/catalystneuro/neuroconv/pull/914)
 * Added helper function `neuroconv.tools.data_transfers.submit_aws_batch_job` for basic automated submission of AWS batch jobs. [PR #384](https://github.com/catalystneuro/neuroconv/pull/384)
+* Data interfaces `run_conversion` method now performs metadata validation before running the conversion. [PR #949](https://github.com/catalystneuro/neuroconv/pull/949)
 * Introduced `null_values_for_properties` to `add_units_table` to give user control over null values behavior [PR #989](https://github.com/catalystneuro/neuroconv/pull/989)
 
 
@@ -26,6 +27,7 @@
 * The `DeeplabcutInterface` now skips inferring timestamps from movie when timestamps are specified, running faster. [PR #967](https://github.com/catalystneuro/neuroconv/pull/967)
 * Improve metadata writing for SpikeGLX data interface. Added contact ids, shank ids and, remove references to shanks for neuropixels 1.0. Also deprecated the previous neuroconv exclusive property "electrode_shank_number` [PR #986](https://github.com/catalystneuro/neuroconv/pull/986)
 * Add tqdm with warning to DeepLabCut interface [PR #1006](https://github.com/catalystneuro/neuroconv/pull/1006)
+* `BaseRecordingInterface` now calls default metadata when metadata is not passing mimicking `run_conversion` behavior. [PR #1012](https://github.com/catalystneuro/neuroconv/pull/1012)
 
 
 ## v0.5.0 (July 17, 2024)

docs/user_guide/adding_trials.rst

@@ -3,11 +3,9 @@
 Adding Trials to NWB Files
 ==========================
 
-NWB allows you to store information about time intervals in a structured way. These structure are often used to store
-information about trials, epochs, or other time intervals in the data.
-You can add time intervals to an NWBFile object before writing it using PyNWB.
-Here is an example of how to add trials to an NWBFile object.
-Here is how you would add trials to an NWB file:
+NWB allows you to store information about timing information in a structured way.
+These structures are often used to store information about trials, epochs, or other time intervals in the data.
+Here is how to add trials to an NWBFile object:
 
 .. code-block:: python
 
@@ -21,10 +19,15 @@ You can also add epochs or other types of time intervals to an NWB File. See
 `PyNWB Annotating Time Intervals <https://pynwb.readthedocs.io/en/stable/tutorials/general/plot_timeintervals.html>`_
 for more information.
 
-Once this information is added, you can write the NWB file to disk.
+Once this information is added, you can write the NWB file to disk:
 
 .. code-block:: python
 
     from neuroconv.tools.nwb_helpers import configure_and_write_nwbfile
 
-    configure_and_write_nwbfile(nwbfile, save_path="path/to/destination.nwb", backend="hdf5")
+    configure_and_write_nwbfile(
+        nwbfile, save_path="path/to/destination.nwb", backend="hdf5"
+    )
+
+This will write the NWB file to disk with the added trials information, and optimize the storage settings of large
+datasets for cloud compute.

docs/user_guide/datainterfaces.rst

@@ -28,33 +28,33 @@ For instance, to install the dependencies for SpikeGLX, run:
 
 2. Construction
 ~~~~~~~~~~~~~~~
-Initialize a class and direct it to the appropriate source data. This will open
-the files and read header information, setting up the system for conversion,
-but generally will not read the underlying data.
+Initialize a class and direct it to the appropriate source data:
 
 .. code-block:: python
 
     from neuroconv.datainterfaces import SpikeGLXRecordingInterface
 
     interface = SpikeGLXRecordingInterface(file_path="path/to/towersTask_g0_t0.imec0.ap.bin")
 
+This will open the files and read header information, setting up the system for conversion,
+but generally will not read the underlying data.
+
 .. note::
 
      To get the form of source_data, run :meth:`.BaseDataInterface.get_source_schema`,
      which returns the :ref:`source schema <source_schema>` as a JSON-schema-like dictionary informing
      the user of the required and optional input arguments to the downstream readers.
 
-
 3. Get and adjust metadata
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 Each ``DataInterface`` can extract relevant metadata from the source files and
-organize it in a ``metadata`` hierarchical dictionary. This dictionary
-can be edited to include data not available in the source files.
+organize it in a ``metadata`` hierarchical dictionary:
 
 .. code-block:: python
 
     metadata = interface.get_metadata()
 
+This dictionary can be edited to include data not available in the source files.
 The DANDI Archive requires subject ID, sex, age, and species, which are rarely present in the source data. Here is how you would add them.
 
 .. code-block:: python
@@ -75,14 +75,17 @@ The DANDI Archive requires subject ID, sex, age, and species, which are rarely p
 - ``U`` for Unknown
 - ``O`` for Other
 
-``age`` follows the `ISO 8601 duration format <https://en.wikipedia.org/wiki/ISO_8601#Durations>`_. For example, ``P30D`` is 30 days old, and ``P1Y`` would be 1 year old. To express a range of ages, you can use a slash, for example ``P30D/P35D`` for 30 to 35 days old.
+``age`` follows the `ISO 8601 duration format <https://en.wikipedia.org/wiki/ISO_8601#Durations>`_.
+For example, ``P30D`` is 30 days old, and ``P1Y`` would be 1 year old.
+To express a range of ages, you can use a slash, for example ``P30D/P35D`` for 30 to 35 days old.
 
 ``species`` is the scientific Latin binomial name of the species. For example, ``Mus musculus``
 for a mouse.
 
-See :ref:`Subject Best Practices <best_practice_subject_exists>` for details
+See :ref:`Subject Best Practices <best_practice_subject_exists>` for details.
 
-The ``session_start_time`` is also required. This is sometimes found in the source data. If it is not found, you must add it.
+The ``session_start_time`` is also required. This is sometimes found in the source data.
+If it is not found, you must add it:
 
 .. code-block:: python
 
@@ -93,13 +96,15 @@ The ``session_start_time`` is also required. This is sometimes found in the sour
 
 You can use ``tz.tzlocal()`` to get the local timezone.
 
-If the ``session_start_time`` is extracted from the source data, it is often missing a timezone. This is not required but is a recommended best practice. Here is how you would add it.
+If the ``session_start_time`` is extracted from the source data, it is often missing a timezone.
+This is not required but is a recommended best practice. Here is how you would add it:
 
 .. code-block:: python
 
     metadata["NWBFile"]["session_start_time"] = metadata["NWBFile"]["session_start_time"].replace(tzinfo=ZoneInfo("US/Pacific"))
 
-NWB Best Practices also recommends several other fields that are rarely present in the extracted metadata. The metadata dictionary is the place to add this information.
+NWB Best Practices also recommends several other fields that are rarely present in the extracted metadata.
+The metadata dictionary is the place to add this information:
 
 .. code-block:: python
 
@@ -113,7 +118,9 @@ NWB Best Practices also recommends several other fields that are rarely present
         keywords=["finches", "evolution", "Galapagos"],
     )
 
-The ``metadata`` dictionary also contains metadata that pertain to the specific data being converted. In this example, the ``Ecephys`` key contains metadata that pertains to the electrophysiology data being converted. This metadata can be edited in the same way.
+The ``metadata`` dictionary also contains metadata that pertain to the specific data being converted.
+In this example, the ``Ecephys`` key contains metadata that pertains to the electrophysiology data being converted.
+This metadata can be edited in the same way:
 
 .. code-block:: python
 
@@ -134,19 +141,22 @@ The ``metadata`` dictionary also contains metadata that pertain to the specific
        'description': 'Name of the ElectrodeGroup this electrode is a part of.'},
       {'name': 'contact_shapes', 'description': 'The shape of the electrode'}]}
 
-Here we can see that ``metadata["Ecephys"]["ElectrodeGroup"][0]["location"]`` is ``unknown``. We can add this information as follows:
+Here we can see that ``metadata["Ecephys"]["ElectrodeGroup"][0]["location"]`` is ``unknown``.
+We can add this information as follows:
 
 .. code-block:: python
 
     metadata["Ecephys"]["ElectrodeGroup"]["location"] = "V1"
 
 
-Use ``.get_metadata_schema()`` to get the schema of the metadata dictionary. This schema is a JSON-schema-like dictionary that specifies required and optional fields in the metadata dictionary. See :ref:`metadata schema <metadata_schema>` for more information.
+Use ``.get_metadata_schema()`` to get the schema of the metadata dictionary.
+This schema is a JSON-schema-like dictionary that specifies required and optional fields in the metadata dictionary.
+See :ref:`metadata schema <metadata_schema>` for more information.
 
 4a. Run conversion
 ~~~~~~~~~~~~~~~~~~
 The ``.run_conversion`` method takes the (edited) metadata dictionary and
-the path of an NWB file, and launches the actual data conversion into NWB.
+the path of an NWB file, and launches the actual data conversion into NWB:
 
 .. code-block:: python
 
@@ -162,21 +172,24 @@ the file size of the output NWB file and optimizing the file for cloud compute.
 
 4b. Create an in-memory NWB file
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-If you want to create an in-memory NWB file, you can use the ``.create_nwbfile`` method.
+You can also create an in-memory NWB file:
 
 .. code-block:: python
 
     nwbfile = spikeglx_interface.create_nwbfile(metadata=metadata)
 
-This is useful for add data such as trials, epochs, or other time intervals to the NWB file. See
-:ref:`Adding Time Intervals to NWB Files <adding_trials>` for more information.
+This is useful for adding extra data such as trials, epochs, or other time intervals to the NWB file.
+See :ref:`Adding Time Intervals to NWB Files <adding_trials>` for more information.
 
-This does not load large datasets into memory. Those remain in the source files and are read piece-by-piece during the
-write process. Once you make all the modifications you want to the NWBfile, you can save it to disk. The following code
-automatically optimizes datasets for cloud compute and writes the file to disk.
+This does not load large datasets into memory.
+Those remain in the source files and are read piece-by-piece during the write process.
+Once you make all the modifications you want to the NWBfile, you can save it to disk.
+The following code automatically optimizes datasets for cloud compute and writes the file to disk:
 
 .. code-block:: python
 
     from neuroconv.tools.nwb_helpers import configure_and_write_nwbfile
 
-    configure_and_write_nwbfile(nwbfile, save_path="path/to/destination.nwb", backend="hdf5")
+    configure_and_write_nwbfile(
+        nwbfile, save_path="path/to/destination.nwb", backend="hdf5"
+    )

docs/user_guide/nwbconverter.rst

@@ -6,8 +6,8 @@ preprocessing systems with different proprietary formats in the same session.
 For instance, in a given extracellular electrophysiology experiment, you might
 have raw and processed data. The :py:class:`.NWBConverter` class streamlines this
 conversion process. This single :py:class:`.NWBConverter` object is responsible for
-combining those multiple read/write operations. An example of how to define
-a :py:class:`.NWBConverter` would be
+combining those multiple read/write operations. Here is an example definition of a
+:py:class:`.NWBConverter`:
 
 .. code-block:: python
 
@@ -42,15 +42,14 @@ keys of``data_interface_classes``.
 
     example_nwb_converter = ExampleNWBConverter(source_data)
 
-This creates an :py:class:`.NWBConverter` object that can aggregate and distribute across
-the data interfaces. To fetch metadata across all of the interfaces and merge
-them together, call.
+This creates an :py:class:`.NWBConverter`. To fetch metadata across all of the interfaces and merge
+them together, call:
 
 .. code-block:: python
 
     metadata = converter.get_metadata()
 
-The metadata can then be manually modified with any additional user-input, just like ``DataInterface`` objects.
+The metadata can then be manually modified with any additional user-input, just like ``DataInterface`` objects:
 
 .. code-block:: python
 
@@ -59,14 +58,16 @@ The metadata can then be manually modified with any additional user-input, just
     metadata["Subject"]["subject_id"] = "ID of experimental subject"
 
 The final metadata dictionary should follow the form defined by :meth:`.NWBConverter.get_metadata_schema`.
-Now run the entire conversion with.
+
+Now run the entire conversion with:
 
 .. code-block:: python
 
     converter.run_conversion(metadata=metadata, nwbfile_path="my_nwbfile.nwb")
 
-Like ``DataInterface`` objects, :py:class:`.NWBConverter` objects can output an in-memory NWBFile object by
-calling :meth:`.NWBConverter.create_nwbfile`. This can be useful for debugging or for further processing.
+Like ``DataInterface`` objects, :py:class:`.NWBConverter` objects can output an in-memory :py:class:`.NWBFile` object by
+calling :meth:`.NWBConverter.create_nwbfile`. This can be useful for debugging, for adding metadata to the file, or for
+further processing.
 
 Though this example was only for two data streams (recording and spike-sorted
 data), it can easily extend to any number of sources, including video of a

src/neuroconv/basedatainterface.py

@@ -64,14 +64,20 @@ def get_metadata(self) -> DeepDict:
 
         return metadata
 
-    def validate_metadata(self, metadata: dict) -> None:
+    def validate_metadata(self, metadata: dict, append_mode: bool = False) -> None:
         """Validate the metadata against the schema."""
         encoder = NWBMetaDataEncoder()
         # The encoder produces a serialized object, so we deserialized it for comparison
 
         serialized_metadata = encoder.encode(metadata)
         decoded_metadata = json.loads(serialized_metadata)
-        validate(instance=decoded_metadata, schema=self.get_metadata_schema())
+        metdata_schema = self.get_metadata_schema()
+        if append_mode:
+            # Eliminate required from NWBFile
+            nwbfile_schema = metdata_schema["properties"]["NWBFile"]
+            nwbfile_schema.pop("required", None)
+
+        validate(instance=decoded_metadata, schema=metdata_schema)
 
     def create_nwbfile(self, metadata: Optional[dict] = None, **conversion_options) -> NWBFile:
         """
@@ -157,6 +163,11 @@ def run_conversion(
         if metadata is None:
             metadata = self.get_metadata()
 
+        file_initially_exists = Path(nwbfile_path).exists() if nwbfile_path is not None else False
+        append_mode = file_initially_exists and not overwrite
+
+        self.validate_metadata(metadata=metadata, append_mode=append_mode)
+
         with make_or_load_nwbfile(
             nwbfile_path=nwbfile_path,
             nwbfile=nwbfile,

src/neuroconv/datainterfaces/ecephys/baserecordingextractorinterface.py

@@ -299,7 +299,7 @@ def add_to_nwbfile(
         write_electrical_series: bool = True,
         compression: Optional[str] = None,  # TODO: remove completely after 10/1/2024
         compression_opts: Optional[int] = None,
-        iterator_type: str = "v2",
+        iterator_type: Optional[str] = "v2",
         iterator_opts: Optional[dict] = None,
     ):
         """
@@ -324,10 +324,9 @@ def add_to_nwbfile(
         write_electrical_series : bool, default: True
             Electrical series are written in acquisition. If False, only device, electrode_groups,
             and electrodes are written to NWB.
-        iterator_type : {'v2', 'v1'}
+        iterator_type : {'v2'}
             The type of DataChunkIterator to use.
-            'v1' is the original DataChunkIterator of the hdmf data_utils.
-            'v2' is the locally developed RecordingExtractorDataChunkIterator, which offers full control over chunking.
+            'v2' is the locally developed RecordingExtractorDataChunkIterator, which offers full control over chunking
         iterator_opts : dict, optional
             Dictionary of options for the RecordingExtractorDataChunkIterator (iterator_type='v2').
             Valid options are:
@@ -357,6 +356,9 @@ def add_to_nwbfile(
         else:
             recording = self.recording_extractor
 
+        if metadata is None:
+            metadata = self.get_metadata()
+
         add_recording(
             recording=recording,
             nwbfile=nwbfile,

src/neuroconv/nwbconverter.py

@@ -101,13 +101,20 @@ def get_metadata(self) -> DeepDict:
             metadata = dict_deep_update(metadata, interface_metadata)
         return metadata
 
-    def validate_metadata(self, metadata: Dict[str, dict]):
+    def validate_metadata(self, metadata: Dict[str, dict], append_mode: bool = False):
         """Validate metadata against Converter metadata_schema."""
         encoder = NWBMetaDataEncoder()
         # The encoder produces a serialized object, so we deserialized it for comparison
         serialized_metadata = encoder.encode(metadata)
         decoded_metadata = json.loads(serialized_metadata)
-        validate(instance=decoded_metadata, schema=self.get_metadata_schema())
+
+        metadata_schema = self.get_metadata_schema()
+        if append_mode:
+            # Eliminate required from NWBFile
+            nwbfile_schema = metadata_schema["properties"]["NWBFile"]
+            nwbfile_schema.pop("required", None)
+
+        validate(instance=decoded_metadata, schema=metadata_schema)
         if self.verbose:
             print("Metadata is valid!")
 
@@ -206,7 +213,7 @@ def run_conversion(
         """
 
         if nwbfile_path is None:
-            warnings.warn(  # TODO: remove on or after 12/26/2024
+            warnings.warn(  # TODO: remove on or after 2024/12/26
                 "Using Converter.run_conversion without specifying nwbfile_path is deprecated. To create an "
                 "NWBFile object in memory, use Converter.create_nwbfile. To append to an existing NWBFile object,"
                 " use Converter.add_to_nwbfile."
@@ -215,10 +222,13 @@ def run_conversion(
         backend = _resolve_backend(backend, backend_configuration)
         no_nwbfile_provided = nwbfile is None  # Otherwise, variable reference may mutate later on inside the context
 
+        file_initially_exists = Path(nwbfile_path).exists() if nwbfile_path is not None else False
+        append_mode = file_initially_exists and not overwrite
+
         if metadata is None:
             metadata = self.get_metadata()
 
-        self.validate_metadata(metadata=metadata)
+        self.validate_metadata(metadata=metadata, append_mode=append_mode)
         self.validate_conversion_options(conversion_options=conversion_options)
 
         self.temporally_align_data_interfaces()