Add batching and parallelization documentation (#67)

Author: PardhavMaradani

CHANGELOG.md

@@ -33,6 +33,7 @@ The rules for this file:
 - Added batching and parallel support for com distance widget (PR #64)
 - Minor widget enhancements (PR #65)
 - Added custom widgets documentation (PR #66)
+- Added batching and parallelization documentation (PR #67)
 
 ### Fixed
 

docs/source/adding_custom_widgets.rst

@@ -84,6 +84,8 @@ These available run methods are:
     A Widget can make the ``_run_frequency`` and ``_run_mode`` attributes dynamically
     changeable at runtime as well by making them as `Inputs`_.
 
+.. _run-frequency:
+
 _run_frequency
 ~~~~~~~~~~~~~~
 
@@ -108,6 +110,8 @@ If the ``_run_mode`` is ``serial``:
 * and ``_run_frequency`` is ``batch``,
   :meth:`~mdadash.backend.widgets.base.WidgetBase.run_batch` method is invoked.
 
+.. _run-mode:
+
 _run_mode
 ~~~~~~~~~
 
@@ -117,7 +121,7 @@ If the ``_run_mode`` is ``parallel`` for a given widget instance, a
 :meth:`~mdadash.backend.widgets.base.WidgetBase.get_parallel_job` method is invoked to
 retrieve the parallel job (a ``joblib.delayed`` tuple). A global "Parallel Jobs" under
 "Settings > Dashboard Configuration" in the dasboard controls the total number of jobs
-run in parallel during each iteration (``n_jobs`` param for ``joblib.Parallel`` call).
+run in parallel during each iteration (``n_jobs`` param for `joblib.Parallel`_ call).
 
 If a widget has ``_run_mode`` as ``parallel``, after the parallel job is completed, a
 :meth:`~mdadash.backend.widgets.base.WidgetBase.apply_parallel_results` method is invoked
@@ -336,3 +340,5 @@ for these are examples of more complex use cases.
 If you are adding a custom Widget that could be useful for others in the community, you can
 create a `pull request <https://github.com/MDAnalysis/mdadash/pulls>`_ to make it part of the
 :doc:`built_in_widgets`.
+
+.. _joblib.Parallel: https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html

docs/source/batching.rst

@@ -0,0 +1,114 @@
+Batching
+========
+
+`IMDReader`_ provides support for reading MD simulation data via the `IMDv3 Protocol`_
+in MDAnalysis since `Release 2.10.0`_.
+
+Since IMD streams data in real-time from a running simulation, it has fundamental constraints
+that differ from traditional trajectory readers and this leads to some `Important Limitations`_
+in `IMDReader`_.
+
+Buffered Access
+---------------
+
+To support buffered, time-dependent analyses in ``mdadash``, a
+:class:`~mdadash.backend.kernel.core.BufferedTrajectory` is introduced.
+
+The original trajectory is wrapped by the :class:`~mdadash.backend.kernel.core.BufferedTrajectory`
+to provide buffered access to the last ``n`` timesteps, where ``n`` is the configured batch size.
+
+.. code-block:: python
+
+    u.trajectory = BufferedTrajectory(u.trajectory, config["batch_size"])
+
+``trajectory[index]`` can be used to access individual frames. Index values can range from 0 to the
+configured batch size ``n``. The batch size ``n`` is available via the ``trajectory.buffer_size``
+attribute.
+
+When a Widget class supports :ref:`batching <run-frequency>` and implements the 
+:meth:`~mdadash.backend.widgets.base.WidgetBase.run_batch` method, the trajectory can be iterated
+this way to access the last ``n`` timesteps.
+
+Here is a typical compute batch block used in the code for :doc:`built_in_widgets`:
+
+.. code-block:: python
+
+    def _compute_batch(self):
+        """Compute for current batch"""
+        values = []
+        for i in range(self.u.trajectory.buffer_size):
+            _ = self.u.trajectory[i]  # set the trajectory to frame i
+            values.append(self._compute_current_frame())
+        return values
+
+
+AnalysisBase support
+--------------------
+
+MDAnalysis provides an `AnalysisBase`_, which is the base class for defining multi-frame analysis.
+
+A lot of built-in MDAnalysis `Analysis modules`_ derive from `AnalysisBase`_.
+
+The :class:`~mdadash.backend.kernel.core.BufferedTrajectory` enables using these analysis
+modules in the Widget classes, which are not possible with `IMDReader`_.
+
+.. note::
+
+    The total number of frames as seen by the `AnalysisBase`_-based classes will be the
+    configured Buffer / batch size during a full ``analysis.run()`` invocation.
+
+Here is an example of using an `AnalysisBase`_-based class within the Widget code by the
+:mod:`Native Contacts <mdadash.backend.analyses.native_contacts>` built-in Widget.
+
+.. code-block:: python
+
+    from MDAnalysis.analysis import contacts
+    .....
+
+    def _create_contacts(self):
+        """Update atom groups when selection phrases change"""
+        self.contacts = contacts.Contacts(
+            self.u,
+            .....
+
+    def _compute_batch(self):
+        """Compute values for current batch"""
+        self.contacts.run()
+        values = []
+        for i, (_, q) in enumerate(self.contacts.results.timeseries):
+            .....
+
+
+`AnalysisBase`_-based classes can also be used per-frame by passing the current frame as
+shown in this example:
+
+.. code-block:: python
+
+    def _compute_current_frame(self):
+        """Compute values for current frame"""
+        self.contacts.run(frames=[self.u.trajectory.frame])
+        .....
+
+
+----
+
+The list of all the Widgets that support batching can be found on the
+:doc:`built_in_widgets` page.
+
+
+.. _IMDReader: https://docs.mdanalysis.org/stable/
+    documentation_pages/coordinates/IMD.html
+
+.. _Important Limitations: https://docs.mdanalysis.org/stable/
+    documentation_pages/coordinates/IMD.html#important-limitations
+
+.. _IMDv3 Protocol: https://imdclient.readthedocs.io/en/latest/protocol_v3.html
+
+.. _Release 2.10.0: https://www.mdanalysis.org/2025/10/26/release-2.10.0/
+
+.. _AnalysisBase: https://docs.mdanalysis.org/stable/
+    documentation_pages/analysis/base.html#MDAnalysis.analysis.base.AnalysisBase
+
+.. _Analysis modules: https://docs.mdanalysis.org/stable/
+    documentation_pages/analysis_modules.html
+

docs/source/index.rst

@@ -4,7 +4,7 @@
    contain the root `toctree` directive.
 
 Welcome to ``mdadash`` documentation!
-=========================================================
+=====================================
 
 Source code and contributing instructions for this project can be found in the `GitHub Repository <https://github.com/MDAnalysis/mdadash>`_.
 
@@ -15,6 +15,8 @@ Source code and contributing instructions for this project can be found in the `
    getting_started
    built_in_widgets
    adding_custom_widgets
+   batching
+   parallelization
    api
 
 

docs/source/parallelization.rst

@@ -0,0 +1,51 @@
+Parallelization
+===============
+
+``mdadash`` supports running Widgets in parallel. It uses `Joblib`_ with the default
+``loky`` backend to run the parallel jobs as separate processes. A global "Parallel Jobs"
+under "Settings > Dashboard Configuration" in the dasboard controls the total number of jobs
+that can run in parallel (``n_jobs`` param for `joblib.Parallel`_ call).
+
+All the analyses that run in ``mdadash`` are CPU-bound and hence a Process-based parallelism
+is chosen instead of Thread-based parallelism.
+
+`IMDReader`_ uses `imdclient`_ to connect to a live MD simulation. Because of the use of a
+network socket within `imdclient`_, this is not serializable by default. ``mdadash`` patches
+`IMDReader`_ to remove `imdclient`_ from the serialization state since the trajectory is never
+iterated using `imdclient`_ within a parallel job. This makes parallelization possible within
+``mdadash`` for streaming trajectories.
+
+A Widget class that supports a ``parallel`` :ref:`_run_mode <run-mode>` must implement 
+:meth:`~mdadash.backend.widgets.base.WidgetBase.get_parallel_job` and
+:meth:`~mdadash.backend.widgets.base.WidgetBase.apply_parallel_results` methods.
+
+As mentioned in the docs for the
+:meth:`~mdadash.backend.widgets.base.WidgetBase.get_parallel_job` and
+:meth:`~mdadash.backend.widgets.base.WidgetBase.apply_parallel_results` methods, everything
+needed by the Widget class (ouputs, updated internal state, etc) must be explicitly returned
+back as return values from the parallel job and applied back to the Widget.
+
+Given the choice of Process-based parallelism, there will be serialization and de-serialization
+overheads involved when Widgets run in parallel mode. The type of analysis and the use of
+batching should be considered when choosing the ``parallel`` :ref:`_run_mode <run-mode>` for
+Widgets.
+
+.. tip::
+
+    :doc:`batching` can be used with Parallelization to limit the impact of the
+    serialization and de-serialization overhead.
+
+----
+
+The list of all the Widgets that can be run in parallel can be found on the
+:doc:`built_in_widgets` page.
+
+
+.. _Joblib: https://joblib.readthedocs.io/en/stable/
+
+.. _joblib.Parallel: https://joblib.readthedocs.io/en/stable/generated/joblib.Parallel.html
+
+.. _IMDReader: https://docs.mdanalysis.org/stable/
+    documentation_pages/coordinates/IMD.html
+
+.. _imdclient: https://imdclient.readthedocs.io/en/latest/