Distributed Analysis Tables (LorenFrankLab#1435)

* Split Mixin

* WIP: AnalysisRegistry

* WIP: dynamic externals

* WIP: Replace master instances with cls/self

* WIP: Fix cls/self references due to refactor

* WIP: Fix tests

* WIP: Fix inheritance

* WIP: Add tests

* WIP: copy files to master analysis table on export

* WIP: fix cleanup

* WIP: simplify cleanup orphan tracking

* WIP: Dynamic fetch_nwb table tuple

* WIP: Prevent multi fks to Analysis tables

* WIP: documentation

* Blackify

* Fix file reservation

* Allow update external of custom analysis tables

* Fix missing prefix var in fetch.py

* Fix test scope

* Remove hanging tests

* Fix test issue

* Error messages

* Fix var name in err msg

* PR comments

* Fix test

* Update changelog

* Add analysis file context manager

* blackify

* Update docs/src/Features/AnalysisTables.md

Co-authored-by: Samuel Bray <[email protected]>

* Remove AnalysisNwbile.cleanup safeguard

* `share_data_to_kachery` copies from parent to master

* Update export._copy_parent_to_master docstring

* 'master' -> 'common'

* Add typing import

* Func rename

* Fix failing test

* PR comments

* Fix teardown issue

---------

Co-authored-by: Samuel Bray <[email protected]>

Author: CBroz1

CHANGELOG.md

@@ -16,6 +16,7 @@ import all foreign key references.
 - Delete extra pyscripts that were renamed # 1363
 - Add note on fetching changes to setup notebook #1371
 - Revise table field docstring heading and `mermaid` diagram generation #1402
+- Add pages for custom analysis tables and class inheritance structure #1435
 
 ### Infrastructure
 
@@ -26,7 +27,9 @@ import all foreign key references.
 - Fix error from unlinked object in `AnalysisNwbfile.create` #1396
 - Sort `UserEnvironment` dict objects by key for consistency #1380
 - Fix typo in VideoFile.make #1427
-- Fix bug in TaskEpoch.make so that it correctly handles multi-row task tables from NWB #1433
+- Fix bug in TaskEpoch.make so that it correctly handles multi-row task tables
+  from NWB #1433
+- Split `SpyglassMixin` into task-specific mixins #1435
 
 ### Infrastructure
 
@@ -40,6 +43,7 @@ import all foreign key references.
 - Common
     - Add tables for storing optogenetic experiment information #1312
     - Remove wildcard matching in `Nwbfile().get_abs_path` #1382
+    - Add custom/dynamic `AnalysisNwbfile` creation #1435
 - Decoding
     - Ensure results directory is created if it doesn't exist #1362
 - Position

docs/mkdocs.yml

@@ -75,6 +75,7 @@ nav:
     - Overview: Features/index.md
     - FigURL: Features/FigURL.md
     - Merge Tables: Features/Merge.md
+    - Analysis Tables: Features/AnalysisTables.md
     - Export: Features/Export.md
     - Centralized Code: Features/Mixin.md
     - Recompute: Features/Recompute.md
@@ -84,6 +85,7 @@ nav:
     - Database Management: ForDevelopers/Management.md
     - Code Reuse: ForDevelopers/Reuse.md
     - Table Types: ForDevelopers/TableTypes.md
+    - Classes: ForDevelopers/Classes.md
     - Understanding a Schema: ForDevelopers/Schema.md
     - Custom Pipelines: ForDevelopers/CustomPipelines.md
     - Using NWB: ForDevelopers/UsingNWB.md

docs/src/Features/AnalysisTables.md

@@ -0,0 +1,264 @@
+# Analysis File Tables
+
+Spyglass uses NWB files to store both raw experimental data and analysis
+results. This guide explains how to create and manage analysis files.
+
+---
+
+## What is AnalysisNwbfile?
+
+`AnalysisNwbfile` is a DataJoint table that tracks analysis files containing
+your results (intermediate computations, final outputs, spike sorting, etc.).
+
+**Key Features**:
+
+- Creates derivative NWB files from your experimental sessions
+- Tracks analysis files in the database with checksums
+- Prevents accidental file modifications
+- Supports custom per-user tables for better performance
+
+**Lifecycle**: Analysis files follow a three-step process:
+
+```
+CREATE → POPULATE → REGISTER
+```
+
+Once registered, files are **checksummed and immutable** - any modification
+will break the checksum and cause errors.
+
+---
+
+## Table of Contents
+
+- [How to Use (Recommended)](#how-to-use-recommended)
+- [Using Custom Tables](#using-custom-tables)
+- [Legacy Pattern Comparison](#legacy-pattern-comparison)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## How to Use (Recommended)
+
+Use the `.build()` method which provides a context manager that handles the
+CREATE → POPULATE → REGISTER lifecycle automatically.
+
+### Basic Usage
+
+```python
+from spyglass.common import AnalysisNwbfile
+import datajoint as dj
+import pandas as pd
+
+schema = dj.schema("my_schema")
+
+@schema
+class MyAnalysis(dj.Computed):
+    definition = """
+    -> SomeOtherTable
+    ---
+    -> AnalysisNwbfile
+    """
+
+    def make(self, key):
+
+        my_data = ...  # Your analysis data here
+
+        nwb_file_name = key["nwb_file_name"]
+        with AnalysisNwbfile().build(nwb_file_name) as builder:
+            # Add your data using helper methods
+            builder.add_nwb_object( pd.DataFrame(my_data))
+
+            # File automatically registered on exit!
+            analysis_file_name = builder.analysis_file_name
+
+        self.insert1({**key, "analysis_file_name": analysis_file_name})
+```
+
+### Common Operations
+
+**Adding multiple objects**:
+
+```python
+with AnalysisNwbfile().build("session.nwb") as builder:
+    builder.add_nwb_object(position_data, "position")
+    builder.add_nwb_object(velocity_data, "velocity")
+    builder.add_nwb_object(metadata, "analysis_params")
+```
+
+**Adding spike sorting units**:
+
+```python
+with AnalysisNwbfile().build("session.nwb") as builder:
+    builder.add_units(
+        units={1: [0.1, 0.5, 1.2], 2: [0.2, 0.6]},
+        units_valid_times={1: [[0, 10]], 2: [[0, 10]]},
+        units_sort_interval={1: [[0, 5]], 2: [[0, 5]]},
+        metrics={"snr": {1: 5.2, 2: 3.8}}
+    )
+```
+
+**Direct NWB I/O** (for complex operations):
+
+```python
+with AnalysisNwbfile().build("session.nwb") as builder:
+    with builder.open_for_write() as io:
+        nwbf = io.read()
+        nwbf.add_unit(spike_times=[0.1, 0.5, 1.2], id=1)
+        io.write(nwbf)
+```
+
+**What happens on exception**:
+
+```python
+try:
+    with AnalysisNwbfile().build("session.nwb") as builder:
+        builder.add_nwb_object(my_data, "results")
+        raise ValueError("Something went wrong!")
+except ValueError:
+    # File created but NOT registered - logged for cleanup
+    pass
+```
+
+---
+
+## Using Custom Tables
+
+By default, all users share the common `AnalysisNwbfile` table. When multiple
+users work concurrently, this can cause database lock contention and prevent
+new table declarations. To avoid this, Spyglass supports custom per-user
+analysis tables for custom analysis pipelines.
+
+### How to Use
+
+Import from `custom_nwbfile` instead of `common_nwbfile`:
+
+```python
+import datajoint as dj
+
+# Standard (shared table)
+from spyglass.common import AnalysisNwbfile
+# ---------------- OR ----------------
+# Custom (your own table - better performance)
+from spyglass.common.custom_nwbfile import AnalysisNwbfile
+
+schema = dj.schema("my_schema")
+
+# Usage is identical
+@schema
+class MyAnalysis(dj.Computed):
+    definition = """
+    -> SomeOtherTable
+    ---
+    -> AnalysisNwbfile
+    """
+```
+
+**What happens**: Creates a user-specific schema `{username}_nwbfile` automatically,
+providing lock isolation from other users.
+
+**Team sharing**: Set `dj.config["custom"]["database.prefix"] = "teamname"` to
+share across a team (may still have some lock contention).
+
+---
+
+## Legacy Pattern Comparison
+
+**Old way** (manual lifecycle):
+
+```python
+def make(self, key):
+    # CREATE
+    file = AnalysisNwbfile().create("session.nwb")
+
+    # POPULATE
+    AnalysisNwbfile().add_nwb_object(file, data, "results")
+
+    # REGISTER (easy to forget!)
+    AnalysisNwbfile().add("session.nwb", file)
+```
+
+**New way** (automatic):
+
+```python
+with AnalysisNwbfile().build("session.nwb") as builder:
+    builder.add_nwb_object(data, "results")
+    # Auto-registered on exit
+```
+
+---
+
+## Troubleshooting
+
+### Error: "Cannot call add_nwb_object() in state: REGISTERED"
+
+**Cause**: You tried to use a helper method after the file was registered.
+
+**Solution**: Use `build()` which prevents this error:
+
+```python
+# ❌ Old way - easy to make this mistake
+file = AnalysisNwbfile().create("session.nwb")
+AnalysisNwbfile().add("session.nwb", file)  # Registered!
+AnalysisNwbfile().add_nwb_object(file, data)  # ❌ ERROR!
+
+# ✅ New way - impossible to make this mistake
+with AnalysisNwbfile().build("session.nwb") as builder:
+    builder.add_nwb_object(data, "results")
+    # Auto-registered on exit - can't call methods after
+```
+
+### Error: "File downloaded but did not pass checksum"
+
+**Cause**: The file was modified after registration.
+
+**Solutions**:
+
+1. Delete and recreate the file
+2. Discuss why the file was modified with admin to modify the checksum
+
+### Error: "Cannot call add_nwb_object() before entering context manager"
+
+**Cause**: You tried to use builder methods outside the `with` block.
+
+**Solution**:
+
+```python
+# ❌ Wrong
+builder = AnalysisNwbfile().build("session.nwb")
+builder.add_nwb_object(data, "results")  # ❌ ERROR!
+
+# ✅ Correct
+with AnalysisNwbfile().build("session.nwb") as builder:
+    builder.add_nwb_object(data, "results")
+```
+
+### When should I use the builder vs. direct NWB I/O?
+
+**Use the builder** (recommended for 90% of cases):
+
+- Adding DataFrames or arrays
+- Adding spike sorting units
+- Standard analysis workflows
+
+**Use direct I/O** (advanced, legacy code):
+
+- Custom NWB processing modules
+- Complex file modifications
+- Maintaining backward compatibility
+
+Even with direct I/O, you can still use the builder:
+
+```python
+with AnalysisNwbfile().build("session.nwb") as builder:
+    with builder.open_for_write() as io:
+        # Direct PyNWB operations here
+        pass
+```
+
+---
+
+## Related Documentation
+
+- [Database Management](../ForDevelopers/Management.md) - Cleanup, maintenance, and custom analysis tables
+- [DataJoint External Storage](https://docs.datajoint.org/python/admin/5-blob-config.html)
+- [PyNWB File I/O](https://pynwb.readthedocs.io/en/stable/tutorials/general/file.html)

docs/src/Features/Mixin.md

@@ -49,6 +49,13 @@ should be fetched from `Nwbfile` or an analysis file should be fetched from
 `AnalysisNwbfile`. If neither is foreign-key-referenced, the function will refer
 to a `_nwb_table` attribute.
 
+**Custom Analysis File Tables**: Spyglass supports individualized
+`AnalysisNwbfile` tables to address transaction lock contention in multi-team
+environments. The `fetch_nwb()` method automatically detects whether your table
+references the common `AnalysisNwbfile` table or a custom team-specific table
+and fetches from the appropriate location. See [Custom Analysis Tables](../ForDevelopers/Management.md#custom-analysis-tables)
+for details on using custom analysis file tables.
+
 ## Long-Distance Restrictions
 
 In complicated pipelines like Spyglass, there are often tables that 'bury' their

docs/src/Features/index.md

@@ -3,9 +3,11 @@
 This directory contains a series of explainers on tools that have been added to
 Spyglass.
 
+- [Analysis File Tables](./AnalysisTables.md) - Guide to creating and managing
+    analysis NWB files.
 - [Export](./Export.md) - How to export an analysis.
 - [FigURL](./FigURL.md) - How to use FigURL to share figures.
-- [Merge Tables](./Merge.md) - Tables for pipeline versioning.
+- [Merge Tables](./Merge.md) - Tables for pipeline versioning
 - [Mixin](./Mixin.md) - Spyglass-specific functionalities to DataJoint tables,
     including fetching NWB files, long-distance restrictions, and permission
     checks on delete operations.