Add documentation for NWB extensions and publishing process

docs/source/conf.py

@@ -164,6 +164,13 @@ def __call__(self, filename):
     'dandi': ('https://dandiarchive.org/%s', '%s'),
     "nwbinspector": ("https://nwbinspector.readthedocs.io/en/dev/%s", "%s"),
     'hdmf-zarr': ('https://hdmf-zarr.readthedocs.io/en/stable/%s', '%s'),
+    'nwb_extension_git': ('https://github.com/nwb-extensions/%s', '%s'),
+    'nwb-schema-language-docs': ('https://schema-language.readthedocs.io/en/latest/%s', '%s'),
+    'ndx-template-docs': ('https://github.com/nwb-extensions/ndx-template/%s', '%s'),
+    'nwb-schema-docs': ('https://nwb-schema.readthedocs.io/en/latest/%s', '%s'),
+    'hdmf-docutils-docs': ('https://github.com/hdmf-dev/hdmf-docutils/%s', '%s'),
+    'ndx-catalog': ('https://nwb-extensions.github.io/%s', '%s'),
+
 }
 
 nitpicky = True

docs/source/extensions.rst

@@ -0,0 +1,21 @@
+.. _extending-nwb:
+
+Extending NWB
+=============
+
+Neurophysiology is always changing as new technologies are developed. While the core NWB schema supports many of the
+most common data types in neurophysiology, we need a way to accommodate new technologies and unique metadata needs.
+Neurodata extensions (NDX) allow us to define new data types. These data types can extend core types, contain core
+types, or can be entirely new. These extensions are formally defined with a collection of YAML files following
+the `NWB Specification Language <https://schema-language.readthedocs.io/en/latest/index.html>`_.
+
+.. toctree::
+    :maxdepth: 2
+
+    extensions/create_extension
+    extensions/spec_api
+    extensions/auto_api
+    extensions/custom_api
+    extensions/documenting
+    extensions/publishing
+    extensions/examples

docs/source/extensions/auto_api.rst

@@ -0,0 +1,85 @@
+.. _extension-auto-api:
+
+Generating an API for an extension
+------------------------------------
+
+.. _extension-auto-matlabnapi:
+
+Generating a MatNWB API
+~~~~~~~~~~~~~~~~~~~~~~~
+
+In MatNWB, simply call ``generateExtension("path/to/extension/namespace.yaml");``; The class files will be generated under the ``+types/+<extension>`` module and can be accessed via standard MATLAB class semantics:
+
+.. code-block:: MATLAB
+
+    ts = types.ndx_example.TetrodeSeries(<arguments>);
+
+.. note::
+    As seen above, MatNWB will convert namespace names if they are not valid identifiers in MATLAB. See `Variable Names <https://www.mathworks.com/help/matlab/matlab_prog/variable-names.html>`_ for more information. In most cases, the conversion conforms with MATLAB's approach with `matlab.lang.makeValidName() <https://www.mathworks.com/help/matlab/ref/matlab.lang.makevalidname.html>`_
+
+.. _extension-auto-pythonapi:
+
+Generating a PyNWB API
+~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have created the extension specification, we need to create the Python interface. These classes will be
+used just like the PyNWB API to read and write NWB files using Python. There are two ways to do this: you can
+automatically generate the API classes based on the schema, or you can manually create the API classes. Here, we will
+show you how to automatically generate the API. In the next section we will discuss why and how to create custom API
+classes.
+
+
+Open up ``ndx-example/src/pynwb/ndx_example/__init__.py``, and notice the last line:
+
+.. code-block:: python
+
+    TetrodeSeries = get_class('TetrodeSeries', 'ndx-example')
+
+:py:func:`~pynwb.get_class` is a function that automatically creates a Python API object by parsing the extension
+YAML. If you create more neurodata types, simply go down the line creating each one. This is the same object that is
+created when you use the ``load_namespaces`` flag on :py:func:`~pynwb.NWBHDF5IO.__init__`.
+
+Customizing automatically generated APIs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once these classes are generated, you can customize them by dynamically adding or replacing attributes/methods (a.k.a., monkey patching).
+
+A typical example is adding methods. Let's say you wanted a method that could
+return data from only the first channel. You could add that method like this:
+
+.. code-block:: python
+
+    def data_from_first_chan(self):
+        return self.data[:, 0]
+
+    TetrodeSeries.data_from_first_chan = data_from_first_chan
+
+You can also alter existing methods by overwriting them. Lets suppose you wanted to ensure that the
+``trode_id`` field is never less than 0 for the ``TetrodeSeries`` constructor. You can do this by creating a new
+``__init__`` function and assigning it to the class.
+
+.. code-block:: python
+
+    from hdmf.utils import docval, get_docval
+    from hdmf.common.table import DynamicTableRegion
+
+    @docval(get_docval(TetrodeSeries.__init__))
+    def new_init(self, **kwargs):
+        assert kwargs['trode_id'] >=0, f"`trode_id` must be greater than or equal to 0."
+        TetrodeSeries.__init__(self, **kwargs)
+
+    TetrodeSeries.__init__ = new_init
+
+The above code creates a ``new_init`` method that runs a validation step and then calls the original ``__init__``.
+Then the class ``__init__`` is overwritten by the new method. Here we also use ``docval``, which is described in the
+next section.
+
+
+.. tip::
+    This approach is easy, but note your API will be locked to your specification. If you make changes to your
+    specification there will be corresponding changes to the API, and this is likely to break existing code.
+    Also, monkey patches can be very confusing to someone who is not aware of them. Differences
+    between the installed module and the actual behavior of the source code can lead to frustrated
+    developers. As such, this approach should be used with great care. In the
+    next section we will show you how to create your own custom API that is more robust. In the
+    :ref:`extension-custom-api` section, we'll explore how to build a fully customized API.

docs/source/extensions/create_extension.rst

@@ -0,0 +1,143 @@
+.. _extension-create:
+
+Creating an extension
+=====================
+
+Using ndx-template
+~~~~~~~~~~~~~~~~~~
+Extensions should be created in their own repository, not alongside data conversion code. This facilitates sharing
+and editing of the extension separately from the code that uses it. When starting a new extension, we highly
+recommend using the :nwb_extension_git:`ndx-template` repository, which automatically generates a repository with
+the appropriate directory structure.
+
+After you finish the instructions :nwb_extension_git:`here <ndx-template/blob/main/README.md>`,
+you should have a directory structure that looks like this
+
+.. code-block:: bash
+
+    ├── LICENSE.txt
+    ├── MANIFEST.in
+    ├── NEXTSTEPS.md
+    ├── README.md
+    ├── docs
+    │   ├── Makefile
+    │   ├── README.md
+    │   ├── make.bat
+    │   └── source
+    │       ├── _static
+    │       │   └── theme_overrides.css
+    │       ├── conf.py
+    │       ├── conf_doc_autogen.py
+    │       ├── credits.rst
+    │       ├── description.rst
+    │       ├── format.rst
+    │       ├── index.rst
+    │       └── release_notes.rst
+    ├── requirements.txt
+    ├── setup.cfg
+    ├── setup.py
+    ├── spec
+    │   ├── ndx-example.extensions.yaml
+    │   └── ndx-example.namespace.yaml
+    └── src
+        ├── matnwb
+        │   └── README.md
+        ├── pynwb
+        │   ├── README.md
+        │   ├── ndx_example
+        │   │   └── __init__.py
+        │   └── tests
+        │       ├── __init__.py
+        │       └── test_tetrodeseries.py
+        └── spec
+            └── create_extension_spec.py
+
+At its core, an NWB extension consists of YAML text files, such as those generated in the `spec`
+folder. While you can write these YAML extension files by hand, PyNWB provides a convenient API
+via the :py:mod:`~pynwb.spec` module for creating extensions.
+
+Open ``src/spec/create_extension_spec.py``. You will be
+modifying this script to create your own NWB extension. Let's first walk through each piece.
+
+Creating a namespace
+~~~~~~~~~~~~~~~~~~~~
+NWB organizes types into namespaces. You must define a new namespace before creating any new types. After following
+the instructions from the :nwb_extension_git:`ndx-template`, you should have a file
+``ndx-my-ext/src/spec/create_extension_spec.py``. The beginning of this file should look like
+
+.. code-block:: python
+
+    from pynwb.spec import NWBNamespaceBuilder, export_spec, NWBGroupSpec, NWBAttributeSpec
+    # TODO: import the following spec classes as needed
+    # from pynwb.spec import NWBDatasetSpec, NWBLinkSpec, NWBDtypeSpec, NWBRefSpec
+
+
+    def main():
+        # these arguments were auto-generated from your cookiecutter inputs
+        ns_builder = NWBNamespaceBuilder(
+            doc="my description",
+            name="ndx-my-ext",
+            version="0.1.0",
+            author="John Doe",
+            contact="[email protected]"
+        )
+
+Here, after the initial imports, we are defining meta-data of the extension.
+Pay particular attention to ``version``. If you make changes to your extension
+after the initial release, you should increase the numbers in your version
+number, so that you can keep track of what exact version of the extension was
+used for each file. We recommend using a semantic versioning approach.
+
+Including types
+~~~~~~~~~~~~~~~
+
+Next, we need to include types from the core schemas. This is analogous to
+importing classes in Python. The generated file includes some example imports.
+
+.. code-block:: python
+
+    ns_builder.include_type('ElectricalSeries', namespace='core')
+    ns_builder.include_type('TimeSeries', namespace='core')
+    ns_builder.include_type('NWBDataInterface', namespace='core')
+    ns_builder.include_type('NWBContainer', namespace='core')
+    ns_builder.include_type('DynamicTableRegion', namespace='hdmf-common')
+    ns_builder.include_type('VectorData', namespace='hdmf-common')
+    ns_builder.include_type('Data', namespace='hdmf-common')
+
+Neuroscience-specific data types are defined in the namespace ``'core'``
+(which means core NWB). More general organizational data types that are not
+specific to neuroscience and are relevant across scientific fields are defined
+in ``'hdmf-common'``. You can see which types are defined in which namespace by
+exploring the `NWB schema documentation <https://nwb-schema.readthedocs.io/en/latest/>`_
+and hdmf-common schema documentation `<https://hdmf-common-schema.readthedocs.io/en/latest/>`_.
+
+Defining new neurodata types
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Next, the ``create_extension_spec.py`` file declares an example extension
+for a new neurodata type called ``TetrodeSeries``, which extends the :py:class:`~pynwb.ecephys.ElectricalSeries`
+type. Then it creates a list of all new data types.
+
+.. code-block:: python
+
+    tetrode_series = NWBGroupSpec(
+        neurodata_type_def='TetrodeSeries',
+        neurodata_type_inc='ElectricalSeries',
+        doc=('An extension of ElectricalSeries to include the tetrode ID for '
+             'each time series.'),
+        attributes=[
+            NWBAttributeSpec(
+                name='trode_id',
+                doc='The tetrode ID.',
+                dtype='int32'
+            )
+        ],
+    )
+
+    # TODO: add all of your new data types to this list
+    new_data_types = [tetrode_series]
+
+The remainder of the file is to generate the YAML files from the spec definition, and should not be changed.
+
+After you make changes to this file, you should run it to re-generate the ``ndx-[name].extensions.yaml`` and
+``ndx-[name].namespace.yaml`` files. In the next section, :ref:`extension-spec-api`, we will go into more detail into how to create neurodata
+types.