Merge branch 'dev' into planeseg_init_check

Author: rly

CHANGELOG.md

@@ -1,12 +1,28 @@
 # PyNWB Changelog
 
-## Upcoming
+## PyNWB 3.1.1 (July 16, 2025)
+
+### Bug fixes
+- Fixed reading and exporting of files written with NWB Schema < 2.9.0 that contained a reference to the electrodes table. @rly [#2112](https://github.com/NeurodataWithoutBorders/pynwb/pull/2112)
+- Skip streaming tests gracefully if offline. @rly [#2113](https://github.com/NeurodataWithoutBorders/pynwb/pull/2113)
 
-### Breaking changes
 
+## PyNWB 3.1.0 (July 8, 2025)
+
+### Breaking changes
 - Removed unused functions `prepend_string` and `_not_parent` in `core.py`, `_not_parent` in `file.py`, and `NWBBaseTypeMapper.get_nwb_file` in `io/core.py` @oruebel [#2036](https://github.com/NeurodataWithoutBorders/pynwb/pull/2036)
 
 ### Enhancements and minor changes
+- Added support for NWB Schema 2.9.0.
+  - Added `BaseImage` and `ExternalImage` as new neurodata types. The first so both `Image` and `ExternalImage` can inherit from it. The second to store external images. @rly [#2079](https://github.com/NeurodataWithoutBorders/pynwb/pull/2079)
+  - Added new `ElectrodesTable` neurodata type. @mavaylon1 [#1890](https://github.com/NeurodataWithoutBorders/pynwb/pull/1890)
+  - Formally defined and renamed `ElectrodeTable` as the `ElectrodesTable` neurodata type. @mavaylon1 [#1890](https://github.com/NeurodataWithoutBorders/pynwb/pull/1890)
+  - Formally defined bands within `DecompositionSeries` as the neurodatatype `FrequencyBandsTable`. @mavaylon1 @rly [#2063](https://github.com/NeurodataWithoutBorders/pynwb/pull/2063)
+  - Added new `DeviceModel` neurodata type to store device model information. @rly [#2088](https://github.com/NeurodataWithoutBorders/pynwb/pull/2088)
+  - Deprecated `Device.model_name`, `Device.model_number`, and `Device.manufacturer` fields in favor of `DeviceModel`. @rly [#2088](https://github.com/NeurodataWithoutBorders/pynwb/pull/2088)
+  - Added support for 2D `EventDetection.source_index` to indicate [time_index, channel_index]. @stephprince [#2091](https://github.com/NeurodataWithoutBorders/pynwb/pull/2091)
+  - Made `EventDetection.times` optional. @stephprince [#2091](https://github.com/NeurodataWithoutBorders/pynwb/pull/2091)
+  - Deprecated `EventDetection.times`. @stephprince [#2101](https://github.com/NeurodataWithoutBorders/pynwb/pull/2101)
 - Automatically add timezone information to timestamps reference time if no timezone information is specified. @stephprince [#2056](https://github.com/NeurodataWithoutBorders/pynwb/pull/2056)
 - Added option to disable typemap caching and updated type map cache location. @stephprince [#2057](https://github.com/NeurodataWithoutBorders/pynwb/pull/2057)
 - Added dictionary-like operations directly on `ProcessingModule` objects (e.g., `len(processing_module)`). @bendichter [#2020](https://github.com/NeurodataWithoutBorders/pynwb/pull/2020)
@@ -24,7 +40,8 @@
 - Fixed missing `IndexSeries.indexed_images`. @rly [#2074](https://github.com/NeurodataWithoutBorders/pynwb/pull/2074)
 - Fixed missing `__nwbfields__` and `_fieldsname` for `NWBData` and its subclasses. @rly [#2082](https://github.com/NeurodataWithoutBorders/pynwb/pull/2082)
 - Fixed caching of the type map when using HDMF 4.1.0. @rly [#2087](https://github.com/NeurodataWithoutBorders/pynwb/pull/2087)
-- Removed use of complex numbers in scratch tutorial because of incompatibilities with HDMF 4.1.0. @stephprince [#2090](https://github.com/NeurodataWithoutBorders/pynwb/pull/2090/) 
+- Removed use of complex numbers in scratch tutorial because of incompatibilities with HDMF 4.1.0. @stephprince [#2090](https://github.com/NeurodataWithoutBorders/pynwb/pull/2090/)
+- Made `ImagingPlane.description` optional to conform with the NWB Schema. @rly [#2051](https://github.com/NeurodataWithoutBorders/pynwb/pull/2051)
 
 ### Documentation and tutorial enhancements
 - Added NWB AI assistant to the home page of the documentation. @magland [#2076](https://github.com/NeurodataWithoutBorders/pynwb/pull/2076)

docs/gallery/domain/ecephys.py

@@ -80,15 +80,21 @@
 # The electrodes table references a required :py:class:`~pynwb.ecephys.ElectrodeGroup`, which is used to represent a
 # group of electrodes. Before creating an :py:class:`~pynwb.ecephys.ElectrodeGroup`, you must define a
 # :py:class:`~pynwb.device.Device` object using the method :py:meth:`.NWBFile.create_device`. The fields
-# ``description``, ``manufacturer``, ``model_number``, ``model_name``, and ``serial_number`` are optional, but
-# recommended.
+# ``description``, ``serial_number``, and ``model`` are optional, but recommended. The
+# :py:class:`~pynwb.device.DeviceModel` object stores information about the device model, which can be useful
+# when searching a set of NWB files or a data archive for all files that use a specific device model
+# (e.g., Neuropixels probe).
+device_model = nwbfile.create_device_model(
+    name="Neurovoxels 0.99",
+    manufacturer="Array Technologies",
+    model_number="PRB_1_4_0480_123",
+    description="A 12-channel array with 4 shanks and 3 channels per shank",
+)
 device = nwbfile.create_device(
     name="array",
     description="A 12-channel array with 4 shanks and 3 channels per shank",
-    manufacturer="Array Technologies",
-    model_number="PRB_1_4_0480_123",
-    model_name="Neurovoxels 0.99",
     serial_number="1234567890",
+    model=device_model,
 )
 
 #######################
@@ -238,7 +244,7 @@
 lfp = LFP(electrical_series=lfp_electrical_series)
 
 ####################
-# LFP refers to data that has been low-pass filtered, typically below 300 Hz. This data may also be downsampled. 
+# LFP refers to data that has been low-pass filtered, typically below 300 Hz. This data may also be downsampled.
 # Because it is filtered and potentially resampled, it is categorized as processed data.
 #
 # Create a processing module named ``"ecephys"`` and add the :py:class:`~pynwb.ecephys.LFP` object to it.
@@ -252,7 +258,7 @@
 
 #######################
 # If your data is filtered for frequency ranges other than LFP — such as Gamma or Theta — you should store it in an
-# :py:class:`~pynwb.ecephys.ElectricalSeries` and encapsulate it within a 
+# :py:class:`~pynwb.ecephys.ElectricalSeries` and encapsulate it within a
 # :py:class:`~pynwb.ecephys.FilteredEphys` object.
 
 from pynwb.ecephys import FilteredEphys
@@ -272,21 +278,21 @@
 ecephys_module.add(filtered_ephys)
 
 ################################
-# In some cases, you may want to further process the LFP data and decompose the signal into different frequency bands 
+# In some cases, you may want to further process the LFP data and decompose the signal into different frequency bands
 # to use for other downstream analyses. You can store the processed data from these spectral analyses using a
 # :py:class:`~pynwb.misc.DecompositionSeries` object. This object allows you to include metadata about the frequency
-# bands and metric used (e.g., power, phase, amplitude), as well as link the decomposed data to the original 
+# bands and metric used (e.g., power, phase, amplitude), as well as link the decomposed data to the original
 # :py:class:`~pynwb.base.TimeSeries` signal the data was derived from.
 
 #######################
-# .. note:: When adding data to :py:class:`~pynwb.misc.DecompositionSeries`, the ``data`` argument is assumed to be 
+# .. note:: When adding data to :py:class:`~pynwb.misc.DecompositionSeries`, the ``data`` argument is assumed to be
 #           3D where the first dimension is time, the second dimension is channels, and the third dimension is bands.
 
 
-bands = dict(theta=(4.0, 12.0), 
-             beta=(12.0, 30.0), 
+bands = dict(theta=(4.0, 12.0),
+             beta=(12.0, 30.0),
              gamma=(30.0, 80.0))  # in Hz
-phase_data = np.random.randn(50, 12, len(bands))  # 50 samples, 12 channels, 3 frequency bands  
+phase_data = np.random.randn(50, 12, len(bands))  # 50 samples, 12 channels, 3 frequency bands
 
 decomp_series = DecompositionSeries(
     name="theta",
@@ -353,7 +359,7 @@
 # While the :py:class:`~pynwb.misc.Units` table is used to store spike times and waveform data for
 # spike-sorted, single-unit activity, you may also want to store spike times and waveform snippets of
 # unsorted spiking activity (e.g., multi-unit activity detected via threshold crossings during data acquisition).
-# This information can be stored using :py:class:`~pynwb.ecephys.SpikeEventSeries` objects. 
+# This information can be stored using :py:class:`~pynwb.ecephys.SpikeEventSeries` objects.
 
 spike_snippets = np.random.rand(40, 3, 30)  # 40 events, 3 channels, 30 samples per event
 shank0 = nwbfile.create_electrode_table_region(
@@ -381,7 +387,7 @@
     name="threshold_events",
     detection_method="thresholding, 1.5 * std",
     source_electricalseries=raw_electrical_series,
-    source_idx=[1000, 2000, 3000],
+    source_idx=[[1000, 0], [2000, 4], [3000, 8]],  # indicates the time and channel indices
     times=[.033, .066, .099],
 )
 

environment-ros3.yml

@@ -13,9 +13,9 @@ dependencies:
   - python-dateutil==2.9.0
   - setuptools
   - pytest==7.4.3  # pin to pytest < 8 because of incompatibilities to be addressed
-  - fsspec==2024.12.0
+  - fsspec==2025.5.1
   - requests==2.32.3
-  - aiohttp==3.11.11
+  - aiohttp==3.12.13
   - pip
   - pip:
     - remfile==0.1.13

requirements-doc.txt

@@ -12,7 +12,7 @@ hdf5plugin
 dandi>=0.46.6
 hdmf-zarr>=0.3.0
 zarr<3  # limited to zarr<3 until hdmf-zarr resolves issues with zarr 3.0
-linkml-runtime==1.8.3; python_version >= "3.10"
-schemasheets==0.3.1; python_version >= "3.10"
-oaklib==0.6.18; python_version >= "3.10"
+linkml-runtime==1.9.3; python_version >= "3.10"
+schemasheets==0.4.0; python_version >= "3.10"
+oaklib==0.6.23; python_version >= "3.10"
 pooch

requirements-opt.txt

@@ -1,11 +1,11 @@
-linkml-runtime==1.8.3
-schemasheets==0.3.1
-oaklib==0.6.18
+linkml-runtime==1.9.3
+schemasheets==0.4.0
+oaklib==0.6.23
 
 # for streaming tests
-fsspec==2024.12.0
+fsspec==2025.5.1
 requests==2.32.4
-aiohttp==3.11.11
+aiohttp==3.12.14
 
 # For read_nwb tests
 hdmf-zarr

src/pynwb/base.py

@@ -1,5 +1,6 @@
 from warnings import warn
 from collections.abc import Iterable
+from abc import ABC
 from typing import NamedTuple
 
 import numpy as np
@@ -15,6 +16,8 @@
 __all__ = [
     'ProcessingModule',
     'TimeSeries',
+    'BaseImage',
+    'ExternalImage',
     'Image',
     'ImageReferences',
     'Images',
@@ -402,7 +405,7 @@ def get_data_in_units(self):
         .. math::
             out = data * conversion + offset
 
-        If the field 'channel_conversion' is present, the conversion factor for each channel is additionally applied 
+        If the field 'channel_conversion' is present, the conversion factor for each channel is additionally applied
         to each channel:
 
         .. math::
@@ -422,26 +425,80 @@ def get_data_in_units(self):
         return np.asarray(self.data) * scale_factor + self.offset
 
 
+@register_class('BaseImage', CORE_NAMESPACE)
+class BaseImage(NWBData, ABC):
+    """
+    An abstract base type for image data. Parent type for Image and ExternalImage types.
+    """
+    __nwbfields__ = ('description', )
+
+    @docval({'name': 'name', 'type': str, 'doc': 'The name of the image'},
+            {'name': 'data', 'type': None, 'doc': 'The image data (any type) as specified by a subclass.'},
+            {'name': 'description', 'type': str, 'doc': 'Description of the image', 'default': None},
+            allow_positional=AllowPositional.WARNING,)
+    def __init__(self, **kwargs):
+        description = kwargs.pop('description')
+        super().__init__(**kwargs)
+
+        self.description = description
+
+
 @register_class('Image', CORE_NAMESPACE)
-class Image(NWBData):
+class Image(BaseImage):
     """
-    Abstract image class. It is recommended to instead use pynwb.image.GrayscaleImage or pynwb.image.RGPImage where
-    appropriate.
+    A base type for storing image data directly. Shape can be 2-D (x, y), or 3-D where the
+    third dimension can have three or four elements, e.g. (x, y, (r, g, b)) or
+    (x, y, (r, g, b, a)).
     """
-    __nwbfields__ = ('data', 'resolution', 'description')
+    __nwbfields__ = ('resolution', )
 
-    @docval({'name': 'name', 'type': str, 'doc': 'The name of this image'},
-            {'name': 'data', 'type': ('array_data', 'data'), 'doc': 'data of image. Dimensions: x, y [, r,g,b[,a]]',
+    @docval(*get_docval(BaseImage.__init__, 'name'),
+            {'name': 'data', 'type': ('array_data', 'data'),
+             'doc': 'Data of image. Shape can be (x, y), (x, y, 3), or (x, y, 4).',
              'shape': ((None, None), (None, None, 3), (None, None, 4))},
             {'name': 'resolution', 'type': float, 'doc': 'pixels / cm', 'default': None},
-            {'name': 'description', 'type': str, 'doc': 'description of image', 'default': None},
+            *get_docval(BaseImage.__init__, 'description'),
             allow_positional=AllowPositional.WARNING,)
     def __init__(self, **kwargs):
-        args_to_set = popargs_to_dict(("resolution", "description"), kwargs)
+        resolution = kwargs.pop('resolution')
         super().__init__(**kwargs)
 
-        for key, val in args_to_set.items():
-            setattr(self, key, val)
+        self.resolution = resolution
+
+
+@register_class('ExternalImage', CORE_NAMESPACE)
+class ExternalImage(BaseImage):
+    """
+    A type for referencing an external image file.
+    """
+    __nwbfields__ = ('image_format', 'image_mode')
+
+    @docval(*get_docval(BaseImage.__init__, 'name'),
+            {'name': 'data', 'type': str, 'doc': 'Path or URL to the external image file.'},
+            *get_docval(BaseImage.__init__, 'description'),
+            {
+                'name': 'image_format',
+                'type': str,
+                'doc': (
+                    'Common name of the image file format. Only widely readable, open file formats are allowed.'
+                    'Allowed values are "PNG", "JPEG", and "GIF".'
+                ),
+                'enum': ['PNG', 'JPEG', 'GIF'],
+            },
+            {
+                'name': 'image_mode',
+                'type': str,
+                'doc': 'Image mode (color mode) of the image, e.g., "RGB", "RGBA", "grayscale", and "LA".',
+                'default': None,
+            },
+            allow_positional=AllowPositional.WARNING,)
+    def __init__(self, **kwargs):
+        image_format = kwargs.pop('image_format')
+        image_mode = kwargs.pop('image_mode')
+        super().__init__(**kwargs)
+
+        self.image_format = image_format
+        self.image_mode = image_mode
 
 
 @register_class('ImageReferences', CORE_NAMESPACE)
@@ -455,11 +512,9 @@ class ImageReferences(NWBData):
             {'name': 'data', 'type': 'array_data', 'doc': 'The images in order.'},
             allow_positional=AllowPositional.WARNING,)
     def __init__(self, **kwargs):
-        # NOTE we do not use the docval shape validator here because it will recognize a list of P MxN images as
-        # having shape (P, M, N)
-        # check type and dimensionality
         for image in kwargs['data']:
-            assert isinstance(image, Image), "Images used in ImageReferences must have type Image, not %s" % type(image)
+            assert isinstance(image, BaseImage), \
+                "Images used in ImageReferences must have type BaseImage, not %s" % type(image)
         super().__init__(**kwargs)
 
 
@@ -476,7 +531,7 @@ class Images(MultiContainerInterface):
     __clsconf__ = {
         'attr': 'images',
         'add': 'add_image',
-        'type': Image,
+        'type': BaseImage,
         'get': 'get_image',
         'create': 'create_image'
     }
@@ -485,7 +540,7 @@ class Images(MultiContainerInterface):
             {'name': 'images', 'type': 'array_data', 'doc': 'image objects', 'default': None},
             {'name': 'description', 'type': str, 'doc': 'description of images', 'default': 'no description'},
             {'name': 'order_of_images', 'type': ImageReferences,
-             'doc': 'Ordered dataset of references to Image objects stored in the parent group.', 'default': None},
+             'doc': 'Ordered dataset of references to BaseImage objects stored in the parent group.', 'default': None},
              allow_positional=AllowPositional.WARNING,)
     def __init__(self, **kwargs):