ElectrodesTable (#1890)

Co-authored-by: rly <[email protected]>
Co-authored-by: Steph Prince <[email protected]>
Fix missing __nwbfields__ for NWBData (#2082)

Author: 3 people

CHANGELOG.md

@@ -3,25 +3,27 @@
 ## Upcoming
 
 ### Breaking changes
-
 - Removed unused functions `prepend_string` and `_not_parent` in `core.py`, `_not_parent` in `file.py`, and `NWBBaseTypeMapper.get_nwb_file` in `io/core.py` @oruebel [#2036](https://github.com/NeurodataWithoutBorders/pynwb/pull/2036)
 
 ### Enhancements and minor changes
-- Added support for NWB Schema 2.9.0. @rly [#2079](https://github.com/NeurodataWithoutBorders/pynwb/pull/2079)
-  - Added `BaseImage` and `ExternalImage` as new neurodata types. The first so both `Image` and `ExternalImage` can inherit from it. The second to store external images.
+- Added support for NWB Schema 2.9.0.
+  - Added `BaseImage` and `ExternalImage` as new neurodata types. The first so both `Image` and `ExternalImage` can inherit from it. The second to store external images. @rly [#2079](https://github.com/NeurodataWithoutBorders/pynwb/pull/2079)
+  - Added new `ElectrodesTable` neurodata type. @mavaylon1 [#1890](https://github.com/NeurodataWithoutBorders/pynwb/pull/1890)
+  - Formally defined and renamed `ElectrodeTable` as the `ElectrodesTable` neurodata type. @mavaylon1 [#1890](https://github.com/NeurodataWithoutBorders/pynwb/pull/1890)
 - Automatically add timezone information to timestamps reference time if no timezone information is specified. @stephprince [#2056](https://github.com/NeurodataWithoutBorders/pynwb/pull/2056)
 - Added option to disable typemap caching and updated type map cache location. @stephprince [#2057](https://github.com/NeurodataWithoutBorders/pynwb/pull/2057)
 - Added dictionary-like operations directly on `ProcessingModule` objects (e.g., `len(processing_module)`). @bendichter [#2020](https://github.com/NeurodataWithoutBorders/pynwb/pull/2020)
 - When an external file is detected when initializing an ImageSeries and no format is provided, automatically set format to "external" instead of raising an error. @stephprince [#2060](https://github.com/NeurodataWithoutBorders/pynwb/pull/2060)
 - Added mask_type option to `mock_PlaneSegmentation`. @pauladkisson [#2067](https://github.com/NeurodataWithoutBorders/pynwb/pull/2067)
 
 ### Bug fixes
-- Fix `add_data_interface` functionality that was mistakenly removed in PyNWB 3.0. @stephprince [#2052](https://github.com/NeurodataWithoutBorders/pynwb/pull/2052)
+- Fixed `add_data_interface` functionality that was mistakenly removed in PyNWB 3.0. @stephprince [#2052](https://github.com/NeurodataWithoutBorders/pynwb/pull/2052)
 - Fixed bug in `IntracellularRecordingsTable.__init__` were `IntracellularResponsesTable` wasn't created correctly when custom category tables were provided @oruebel. [#2031](https://github.com/NeurodataWithoutBorders/pynwb/pull/2031)
 - Fixed shape check in `SpikeEventSeries.__init__` to support `AbstractDataChunkIterator` for timestamps/data. @oruebel [#2031](https://github.com/NeurodataWithoutBorders/pynwb/pull/2031)
 - Added unit tests to enhance coverage of `core.py`, `image.py`, `spec.py`, `icephys.py`, `epoch.py` and others. @oruebel [#2031](https://github.com/NeurodataWithoutBorders/pynwb/pull/2031)
-- Fix missing `IndexSeries.indexed_images`. @rly [#2074](https://github.com/NeurodataWithoutBorders/pynwb/pull/2074)
- 
+- Fixed missing `IndexSeries.indexed_images`. @rly [#2074](https://github.com/NeurodataWithoutBorders/pynwb/pull/2074)
+- Fixed missing `__nwbfields__` and `_fieldsname` for `NWBData` and its subclasses. @rly [#2082](https://github.com/NeurodataWithoutBorders/pynwb/pull/2082)
+
 ## PyNWB 3.0.0 (February 26, 2025)
 
 ### Breaking changes

src/pynwb/core.py

@@ -28,6 +28,10 @@ class NWBMixin(AbstractContainer):
 
     _data_type_attr = 'neurodata_type'
 
+    _fieldsname = '__nwbfields__'
+
+    __nwbfields__ = tuple()
+
     @docval({'name': 'neurodata_type', 'type': str, 'doc': 'the data_type to search for', 'default': None})
     def get_ancestor(self, **kwargs):
         """
@@ -72,9 +76,7 @@ def data_type(self):
 @register_class('NWBContainer', CORE_NAMESPACE)
 class NWBContainer(NWBMixin, Container):
 
-    _fieldsname = '__nwbfields__'
-
-    __nwbfields__ = tuple()
+    pass
 
 
 @register_class('NWBDataInterface', CORE_NAMESPACE)
@@ -110,8 +112,8 @@ def __getitem__(self, args):
     def append(self, arg):
         """
         Append a single element to the data
-        
-        Note: The arg to append should be 1 dimension less than the data. 
+
+        Note: The arg to append should be 1 dimension less than the data.
         For example, if the data is a 2D array, arg should be a 1D array.
         Appending to scalar data is not supported. To append multiple
         elements, use extend.
@@ -172,7 +174,7 @@ def notes(self):
         warn(("Use of ScratchData.notes has been deprecated and will be removed in PyNWB 4.0. "
               "Use ScratchData.description instead."), DeprecationWarning)
         return self.description
-    
+
     @notes.setter
     def notes(self, value):
         """

src/pynwb/ecephys.py

@@ -2,7 +2,7 @@
 import numpy as np
 from collections.abc import Iterable
 
-from hdmf.common import DynamicTableRegion
+from hdmf.common import DynamicTableRegion, DynamicTable
 from hdmf.data_utils import assertEqualShape
 from hdmf.utils import docval, popargs, get_docval, popargs_to_dict, get_data_shape, AllowPositional
 
@@ -18,7 +18,8 @@
     'EventDetection',
     'LFP',
     'FilteredEphys',
-    'FeatureExtraction'
+    'FeatureExtraction',
+    'ElectrodesTable',
 ]
 
 
@@ -67,6 +68,42 @@ def __init__(self, **kwargs):
             setattr(self, key, val)
 
 
+@register_class('ElectrodesTable', CORE_NAMESPACE)
+class ElectrodesTable(DynamicTable):
+    """A table of all electrodes (i.e. channels) used for recording. Introduced in NWB 3.0.0. Replaces the "electrodes"
+    table (neurodata_type_inc DynamicTable, no neurodata_type_def) that is part of NWBFile."""
+
+    __columns__ = (
+        {'name': 'location', 'description': 'Location of the electrode (channel).', 'required': True},
+        {'name': 'group', 'description': 'Reference to the ElectrodeGroup.', 'required': True},
+        {'name': 'group_name', 'description': 'Name of the ElectrodeGroup.', 'required': False},
+        {'name': 'x', 'description': 'x coordinate of the channel location in the brain.', 'required': False},
+        {'name': 'y', 'description': 'y coordinate of the channel location in the brain.', 'required': False},
+        {'name': 'z', 'description': 'z coordinate of the channel location in the brain.', 'required': False},
+        {'name': 'imp', 'description': 'Impedance of the channel, in ohms.', 'required': False},
+        {'name': 'filtering', 'description': 'Description of hardware filtering.', 'required': False},
+        {'name': 'rel_x', 'description': 'x coordinate in electrode group.', 'required': False},
+        {'name': 'rel_y', 'description': 'xy coordinate in electrode group.', 'required': False},
+        {'name': 'rel_z', 'description': 'z coordinate in electrode group.', 'required': False},
+        {'name': 'reference', 'description': ('Description of the reference electrode and/or reference scheme used '
+                                              'for this electrode.'), 'required': False}
+    )
+
+    @docval(*get_docval(DynamicTable.__init__, 'id', 'columns', 'colnames'))
+    def __init__(self, **kwargs):
+        kwargs['name'] = 'electrodes'
+        kwargs['description'] = 'metadata about extracellular electrodes'
+        super().__init__(**kwargs)
+
+    def copy(self):
+        """
+        Return a copy of this ElectrodesTable.
+        This is useful for linking.
+        """
+        kwargs = dict(id=self.id, columns=self.columns, colnames=self.colnames)
+        return self.__class__(**kwargs)
+
+
 @register_class('ElectricalSeries', CORE_NAMESPACE)
 class ElectricalSeries(TimeSeries):
     """
@@ -155,12 +192,12 @@ def __init__(self, **kwargs):
             # case where the data is a AbstractDataChunkIterator
             data_shape = get_data_shape(kwargs['data'], strict_no_data_load=True)
             timestamps_shape = get_data_shape(kwargs['timestamps'], strict_no_data_load=True)
-            if (data_shape is not None and 
-                timestamps_shape is not None and 
-                len(data_shape) > 0 and 
+            if (data_shape is not None and
+                timestamps_shape is not None and
+                len(data_shape) > 0 and
                 len(timestamps_shape) > 0):
-                if (data_shape[0] != timestamps_shape[0] and 
-                    data_shape[0] is not None and 
+                if (data_shape[0] != timestamps_shape[0] and
+                    data_shape[0] is not None and
                     timestamps_shape[0] is not None):
                     raise ValueError('Must provide the same number of timestamps and spike events')
         super().__init__(**kwargs)

src/pynwb/file.py

@@ -15,7 +15,7 @@
 from .base import TimeSeries, ProcessingModule
 from .device import Device
 from .epoch import TimeIntervals
-from .ecephys import ElectrodeGroup
+from .ecephys import ElectrodeGroup, ElectrodesTable
 from .icephys import (IntracellularElectrode, SweepTable, PatchClampSeries, IntracellularRecordingsTable,
                       SimultaneousRecordingsTable, SequentialRecordingsTable, RepetitionsTable,
                       ExperimentalConditionsTable)
@@ -385,12 +385,12 @@ class NWBFile(MultiContainerInterface, HERDManager):
             {'name': 'lab_meta_data', 'type': (list, tuple), 'default': None,
              'doc': 'an extension that contains lab-specific meta-data'},
             {'name': 'electrodes', 'type': DynamicTable,
-             'doc': 'the ElectrodeTable that belongs to this NWBFile', 'default': None},
+             'doc': 'the ElectrodesTable that belongs to this NWBFile', 'default': None},
             {'name': 'electrode_groups', 'type': Iterable,
              'doc': 'the ElectrodeGroups that belong to this NWBFile', 'default': None},
             {'name': 'ic_electrodes', 'type': (list, tuple),
              'doc': 'DEPRECATED use icephys_electrodes parameter instead. '
-                    'IntracellularElectrodes that belong to this NWBFile', 'default': None},  
+                    'IntracellularElectrodes that belong to this NWBFile', 'default': None},
                     # TODO remove this arg in PyNWB 4.0
             {'name': 'sweep_table', 'type': SweepTable,
              'doc': '[DEPRECATED] Use IntracellularRecordingsTable instead. '
@@ -611,7 +611,7 @@ def add_epoch(self, **kwargs):
 
     def __check_electrodes(self):
         if self.electrodes is None:
-            self.electrodes = ElectrodeTable()
+            self.electrodes = ElectrodesTable()
 
     @docval(*get_docval(DynamicTable.add_column), allow_extra=True)
     def add_electrode_column(self, **kwargs):
@@ -666,31 +666,15 @@ def add_electrode(self, **kwargs):
         # are not allowed
         if not d['location']:
             raise ValueError("The 'location' argument is required when creating an electrode.")
-        if not kwargs['group']:
+        if not d['group']:
             raise ValueError("The 'group' argument is required when creating an electrode.")
         if d.get('group_name', None) is None:
             d['group_name'] = d['group'].name
 
-        new_cols = [('x', 'the x coordinate of the position (+x is posterior)'),
-                    ('y', 'the y coordinate of the position (+y is inferior)'),
-                    ('z', 'the z coordinate of the position (+z is right)'),
-                    ('imp', 'the impedance of the electrode, in ohms'),
-                    ('filtering', 'description of hardware filtering, including the filter name and frequency cutoffs'),
-                    ('rel_x', 'the x coordinate within the electrode group'),
-                    ('rel_y', 'the y coordinate within the electrode group'),
-                    ('rel_z', 'the z coordinate within the electrode group'),
-                    ('reference', 'Description of the reference electrode and/or reference scheme used for this \
-                        electrode, e.g.,"stainless steel skull screw" or "online common average referencing".')
-                    ]
-
-        # add column if the arg is supplied and column does not yet exist
-        # do not pass arg to add_row if arg is not supplied
-        for col_name, col_doc in new_cols:
-            if kwargs[col_name] is not None:
-                if col_name not in self.electrodes:
-                    self.electrodes.add_column(col_name, col_doc)
-            else:
-                d.pop(col_name)  # remove args from d if not set
+        # remove keys that are None
+        for key in list(d.keys()):
+            if d[key] is None:
+                d.pop(key)
 
         self.electrodes.add_row(**d)
 
@@ -705,7 +689,7 @@ def create_electrode_table_region(self, **kwargs):
         for idx in region:
             if idx < 0 or idx >= len(self.electrodes):
                 raise IndexError('The index ' + str(idx) +
-                                 ' is out of range for the ElectrodeTable of length '
+                                 ' is out of range for the ElectrodesTable of length '
                                  + str(len(self.electrodes)))
         desc = getargs('description', kwargs)
         name = getargs('name', kwargs)
@@ -787,13 +771,13 @@ def add_invalid_time_interval(self, **kwargs):
         self.__check_invalid_times()
         self.invalid_times.add_interval(**kwargs)
 
-    @docval({'name': 'electrode_table', 'type': DynamicTable, 'doc': 'the ElectrodeTable for this file'})
+    @docval({'name': 'electrode_table', 'type': ElectrodesTable, 'doc': 'the ElectrodesTable for this file'})
     def set_electrode_table(self, **kwargs):
         """
-        Set the electrode table of this NWBFile to an existing ElectrodeTable
+        Set the electrode table of this NWBFile to an existing ElectrodesTable
         """
         if self.electrodes is not None:
-            msg = 'ElectrodeTable already exists, cannot overwrite'
+            msg = 'ElectrodesTable already exists, cannot overwrite'
             raise ValueError(msg)
         electrode_table = getargs('electrode_table', kwargs)
         self.electrodes = electrode_table
@@ -804,7 +788,7 @@ def _check_sweep_table(self):
         """
         if self.sweep_table is None:
             if self._in_construct_mode:
-                # Construct the SweepTable without triggering errors in construct mode because 
+                # Construct the SweepTable without triggering errors in construct mode because
                 # SweepTable has been deprecated
                 sweep_table = SweepTable.__new__(SweepTable, parent=self, in_construct_mode=True)
                 sweep_table.__init__(name='sweep_table')
@@ -1146,19 +1130,16 @@ def _tablefunc(table_name, description, columns):
     return t
 
 
-def ElectrodeTable(name='electrodes',
-                   description='metadata about extracellular electrodes'):
-    return _tablefunc(name, description,
-                      [('location', 'the location of channel within the subject e.g. brain region'),
-                       ('group', 'a reference to the ElectrodeGroup this electrode is a part of'),
-                       ('group_name', 'the name of the ElectrodeGroup this electrode is a part of')
-                       ]
-                      )
-
-
 def TrialTable(name='trials', description='metadata about experimental trials'):
     return _tablefunc(name, description, ['start_time', 'stop_time'])
 
 
 def InvalidTimesTable(name='invalid_times', description='time intervals to be removed from analysis'):
     return _tablefunc(name, description, ['start_time', 'stop_time'])
+
+
+def ElectrodeTable(name='electrodes',
+                   description='metadata about extracellular electrodes'):
+    warn("The ElectrodeTable convenience function is deprecated. Please create a new instance of "
+         "the ElectrodesTable class instead.", DeprecationWarning)
+    return ElectrodesTable()

src/pynwb/io/file.py

@@ -181,6 +181,24 @@ def scratch(self, builder, manager):
                 ret.append(manager.construct(d))
         return tuple(ret) if len(ret) > 0 else None
 
+    @ObjectMapper.constructor_arg('electrodes')
+    def electrodes(self, builder, manager):
+        try:
+            electrodes_builder = builder['general']['extracellular_ephys']['electrodes']
+        except KeyError:
+            # Note: This is here because the ObjectMapper pulls argname from docval and checks to see
+            # if there is an override even if the file doesn't have what is looking for. In this case,
+            # electrodes for NWBFile.
+            electrodes_builder = None
+        if (electrodes_builder is not None and electrodes_builder.attributes['neurodata_type'] != 'ElectrodesTable'):
+            electrodes_builder.attributes['neurodata_type'] = 'ElectrodesTable'
+            electrodes_builder.attributes['namespace'] = 'core'
+            manager.clear_cache()
+            new_container =  manager.construct(electrodes_builder)
+            return new_container
+        else:
+            return None
+
     @ObjectMapper.constructor_arg('session_start_time')
     def dateconversion(self, builder, manager):
         """Set the constructor arg for 'session_start_time' to a datetime object.

src/pynwb/nwb-schema

@@ -214,6 +214,30 @@ def _make_subject_without_age_reference():
     test_name = 'subject_no_age__reference'
     _write(test_name, nwbfile)
 
+def _make_electrodes_dynamic_table():
+    """Create a test file where electrodes is a dynamic table and not its own type."""
+    nwbfile = NWBFile(session_description='ADDME',
+                      identifier='ADDME',
+                      session_start_time=datetime.now().astimezone())
+    device = nwbfile.create_device(name="array", description="an array", manufacturer="company")
+    nwbfile.add_electrode_column(name="label", description="label of electrode")
+
+    for i in range(4):
+        electrode_group = nwbfile.create_electrode_group(
+            name=f"shank{i}",
+            description=f"electrode group for shank {i}",
+            device=device,
+            location="brain area",
+        )
+        for j in range(3):
+            nwbfile.add_electrode(
+                group=electrode_group,
+                location="brain area",
+                label=f"shank{i}electrode{j}",
+            )
+    
+    test_name = 'electrodes_dynamic_table'
+    _write(test_name, nwbfile)
 
 if __name__ == '__main__':
     # install these versions of PyNWB and run this script to generate new files
@@ -242,3 +266,6 @@ def _make_subject_without_age_reference():
 
     if __version__ == "2.2.0":
         _make_subject_without_age_reference()
+
+    if __version__ == "3.0.0":
+        _make_electrodes_dynamic_table()