Allow recompute via _make_file func (#1093)

* WIP: remove AnalysisNwbfileLog

* WIP: recompute

* WIP: recompute 2

* WIP: recompute 3

* WIP: recompute 4

* WIP: recompute 5, electrodes object

* WIP: recompute 6, add file hash

* WIP: recompute 7

* ✅ : recompute

* Handle groups and links

* Remove debug

* Add directory hasher

* Update directory hasher

* WIP: update hasher

* WIP: error specificity

* Add tables for recompute processing

* WIP: incorporate feedback

* WIP: enforce environment restriction

* WIP: typo

* WIP: add tests

* WIP: start add V0 hasher

* See details

v0.recompute: add make to recompute table
v0.recording:
  - disable sort group deletes during tests
  - expand _make_file to take dest dir as arg
  - expand hasher method to optionally return hasher
vX.recompute:
  - insert but don't recompute if logged at creation
  - add funcs for deleting successful recompute files
v1.recording: don't log deps at Recording stage, only recompute
nwb_hash: don't remove version, just ignore source_script
tests:
  - migrate spikesorting.v1 tests to subfolder
  - add spikesorting.v0 tests for recording only

v0 tests won't be captured in coverage for now

* WIP: test fixes

* WIP: test fixes

* WIP: add safe round trip v0

* WIP: add doc

* ✅ : Update docs

* Skip failing decoding tests

* nwb deps -> blob, future proof

* Remove extra fields

* WIP: store full env

* Update comparators

* Bypass spikeinterface version restriction

* Minor fixes

* Typo

* Fix tests

* Allow export env yml

* Add note about conda required

* Update limit docstring re random select

* Update src/spyglass/common/common_lab.py

Co-authored-by: Copilot <[email protected]>

* Review comments

* Defer recompute to after kachery/dandi fetch

* Only fetch env once

* Review suggestions @edeno

* Increase UserEnv varchar

* Warn on null restriction for 'attempt_all'

* Handle 'no conda' case. Protect from partial v0 delete case.

* More comment responses

* v0.Recording also pops v0.Versions - fix pytest

* PR review @samuelbray32

---------

Co-authored-by: Copilot <[email protected]>

Author: CBroz1

.pre-commit-config.yaml

@@ -77,7 +77,7 @@ repos:
           - tomli
 
   - repo: https://github.com/ambv/black
-    rev: 24.1.1
+    rev: 25.1.0
     hooks:
       - id: black
-        language_version: python3.9
+        language_version: python3

CHANGELOG.md

@@ -2,9 +2,30 @@
 
 ## [0.5.5] (Unreleased)
 
+### Release Notes
+
+<!-- Running draft to be removed immediately prior to release. -->
+
+<!-- When altering tables, import all foreign key references. -->
+
 Table update script
 
-```py
+```python
+# -- For recompute --
+import datajoint as dj
+from spyglass.spikesorting.v1 import recording as v1rec  # noqa
+from spyglass.spikesorting.v0 import spikesorting_recording as v0rec  # noqa
+from spyglass.linearization.v1.main import TrackGraph  # noqa
+
+dj.FreeTable(dj.conn(), "common_nwbfile.analysis_nwbfile_log").drop()
+dj.FreeTable(dj.conn(), "common_session.session_group").drop()
+TrackGraph.alter()  # Add edge map parameter
+v0rec.SpikeSortingRecording().alter()
+v0rec.SpikeSortingRecording().update_ids()
+v1rec.SpikeSortingRecording().alter()
+v1rec.SpikeSortingRecording().update_ids()
+
+# -- For LFP pipeline --
 from spyglass.lfp.lfp_imported import ImportedLFP
 from spyglass.lfp.lfp_merge import LFPOutput
 
@@ -41,6 +62,7 @@ ImportedLFP().drop()
 - Only add merge parts to `source_class_dict` if present in codebase #1237
 - Remove cli module #1250
 - Fix column error in `check_threads` method #1256
+- Add recompute ability for `SpikeSortingRecording` for both v0 and v1 #1093
 - Track Spyglass version in dedicated table for enforcing updates #1281
 - Pin to `datajoint>=0.14.4` for `dj.Top` and long make call fix #1281
 - Remove outdated code comments #1304
@@ -87,8 +109,7 @@ ImportedLFP().drop()
         and transactional insertion. #1280, #1302
     - Updating the LFPBandSelection logic with comprehensive validation and batch
         insertion for electrodes and references. #1280
-    - Implement `ImportedLFP.make()`  for ingestion from nwb files #1278, #1302
-
+    - Implement `ImportedLFP.make()` for ingestion from nwb files #1278, #1302
 
 ## [0.5.4] (December 20, 2024)
 
@@ -98,6 +119,7 @@ ImportedLFP().drop()
     #1108, #1172, #1187
 - Add docstrings to all public methods #1076
 - Update DataJoint to 0.14.2 #1081
+- Remove `AnalysisNwbfileLog` #1093
 - Allow restriction based on parent keys in `Merge.fetch_nwb()` #1086, #1126
 - Import `datajoint.dependencies.unite_master_parts` -> `topo_sort` #1116,
     #1137, #1162
@@ -109,6 +131,7 @@ ImportedLFP().drop()
 - Update DataJoint install and password instructions #1131
 - Fix dandi upload process for nwb's with video or linked objects #1095, #1151
 - Minor docs fixes #1145
+- Add Nwb hashing tool #1093
 - Test fixes
     - Remove stored hashes from pytests #1152
     - Remove mambaforge from tests #1153
@@ -168,6 +191,7 @@ ImportedLFP().drop()
     - Fix bug in `get_group_by_shank` #1096
     - Fix bug in `_compute_metric` #1099
     - Fix bug in `insert_curation` returned key #1114
+    - Add fields to `SpikeSortingRecording` to allow recompute #1093
     - Fix handling of waveform extraction sparse parameter #1132
     - Limit Artifact detection intervals to valid times #1196
 

docs/mkdocs.yml

@@ -75,6 +75,7 @@ nav:
     - Merge Tables: Features/Merge.md
     - Export: Features/Export.md
     - Centralized Code: Features/Mixin.md
+    - Recompute: Features/Recompute.md
   - For Developers:
     - Overview: ForDevelopers/index.md
     - How to Contribute: ForDevelopers/Contribute.md
@@ -93,6 +94,7 @@ extra_css:
 
 plugins:
   - search
+  - mermaid2 # for graphs
   - exclude:
       glob:
         - "temp*"
@@ -125,7 +127,6 @@ plugins:
       include_source: False
       ignore_h1_titles: True
       ignore: ["*make_pages.py", "**checkpoints**", "*/py_scripts/*"]
-  - mermaid2 # for graphs
 
 markdown_extensions:
   - attr_list
@@ -139,6 +140,8 @@ markdown_extensions:
   - pymdownx.inlinehilite
   - pymdownx.snippets
   - pymdownx.superfences
+  - pymdownx.tabbed:
+      alternate_style: true
 
 extra:
   generator: false # Disable watermark