Merge branch 'master' into import_pose

Author: edeno

CHANGELOG.md

@@ -6,7 +6,10 @@
 
 - Ensure merge tables are declared during file insertion #1205
 - Update URL for DANDI Docs #1210
-- Improve cron job documentation and script #1226
+- Add common method `get_position_interval_epoch` #1056
+- Improve cron job documentation and script #1226, #1241
+- Update export process to include `~external` tables #1239
+- Only add merge parts to `source_class_dict` if present in codebase #1237
 
 ### Pipelines
 
@@ -18,6 +21,12 @@
     - Allow population of missing `PositionIntervalMap` entries during population
         of `DLCPoseEstimation` #1208
     - Enable import of existing pose data to `ImportedPose` in position pipeline #1225
+- Spikesorting
+    - Fix compatibility bug between v1 pipeline and `SortedSpikesGroup` unit
+        filtering #1238
+
+- Behavior
+    - Implement pipeline for keypoint-moseq extraction of behavior syllables #1056
 
 ## [0.5.4] (December 20, 2024)
 

dj_local_conf_example.json

@@ -51,6 +51,11 @@
       "video": "/your/base/path/deeplabcut/video",
       "output": "/your/base/path/deeplabcut/output"
     },
+    "moseq_dirs": {
+      "base": "/your/base/path/moseq",
+      "project": "/your/base/path/moseq/projects",
+      "video": "/your/base/path/moseq/video",
+    },
     "kachery_zone": "franklab.default"
   }
 }

docs/src/Features/Export.md

@@ -165,3 +165,42 @@ Or, optionally, you can use the `RestrGraph` class to cascade hand-picked tables
 and restrictions without the background logging of `ExportMixin`. The assembled
 list of restricted free tables, `RestrGraph.all_ft`, can be passed to
 `Export.write_export` to generate a shell script for exporting the data.
+
+## Backwards Compatibility
+
+Spyglass databases that were declared before varchars were reduced to
+accommodate MySQL key length restrictions (see
+[#630](https://github.com/LorenFrankLab/spyglass/issues/630)) will have trouble
+importing exported data. Specifically, varchar mismatches will throw foreign key
+errors. To fix this, you run the following bash script on the generated `sql`
+files before importing them into the new database.
+
+<details><summary>Script</summary>
+
+```bash
+#!/bin/bash
+
+for file in ./_Pop*sql; do \
+    echo $file
+    sed -i 's/ DEFAULT CHARSET=[^ ]\w*//g' "$file"
+    sed -i 's/ DEFAULT COLLATE [^ ]\w*//g' "$file"
+    sed -i 's/ `nwb_file_name` varchar(255)/ `nwb_file_name` varchar(64)/g' "$file"
+    sed -i 's/ `analysis_file_name` varchar(255)/ `analysis_file_name` varchar(64)/g' "$file"
+    sed -i 's/ `interval_list_name` varchar(200)/ `interval_list_name` varchar(170)/g' "$file"
+    sed -i 's/ `position_info_param_name` varchar(80)/ `position_info_param_name` varchar(32)/g' "$file"
+    sed -i 's/ `mark_param_name` varchar(80)/ `mark_param_name` varchar(32)/g' "$file"
+    sed -i 's/ `artifact_removed_interval_list_name` varchar(200)/ `artifact_removed_interval_list_name` varchar(128)/g' "$file"
+    sed -i 's/ `metric_params_name` varchar(200)/ `metric_params_name` varchar(64)/g' "$file"
+    sed -i 's/ `auto_curation_params_name` varchar(200)/ `auto_curation_params_name` varchar(36)/g' "$file"
+    sed -i 's/ `sort_interval_name` varchar(200)/ `sort_interval_name` varchar(64)/g' "$file"
+    sed -i 's/ `preproc_params_name` varchar(200)/ `preproc_params_name` varchar(32)/g' "$file"
+    sed -i 's/ `sorter` varchar(200)/ `sorter` varchar(32)/g' "$file"
+    sed -i 's/ `sorter_params_name` varchar(200)/ `sorter_params_name` varchar(64)/g' "$file"
+done
+```
+
+</details>
+
+This is essentially a series of `sed` commands that adjust varchar lengths to
+their updated values. This script should be run in the directory containing the
+`_Populate*.sql` files generated by the export process.

environment_moseq_cpu.yml

@@ -0,0 +1,42 @@
+# 1. Install a conda distribution.
+#    https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html
+# 2. Run: `mamba env create -f environment.yml`
+# 3. Activate: `conda activate spyglass`
+#
+# (lines intentionally left blank)
+#
+#
+name: spyglass-moseq-cpu
+channels:
+  - conda-forge
+  # - defaults # deprecated
+  - franklab
+  - edeno
+  # - pytorch # dlc-only
+  # - anaconda  # dlc-only, for cudatoolkit
+dependencies:
+  - bottleneck
+  # - cudatoolkit=11.3 # dlc-only
+  # - ffmpeg # dlc-only
+  - ipympl
+  - jupyterlab>=3.*
+  # - libgcc # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy
+  - pip
+  - position_tools
+  - pybind11  # req by mountainsort4 -> isosplit5
+  - pydotplus
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.13
+  - pytorch<1.12.0
+  - ripple_detection
+  - seaborn
+  # - torchaudio # dlc-only
+  # - torchvision # dlc-only
+  - track_linearization>=2.3
+  - pip:
+      - ghostipy # for common_filter
+      - mountainsort4
+      - .[moseq-cpu]

environment_moseq_gpu.yml

@@ -0,0 +1,42 @@
+# 1. Install a conda distribution.
+#    https://mamba.readthedocs.io/en/latest/installation/mamba-installation.html
+# 2. Run: `mamba env create -f environment.yml`
+# 3. Activate: `conda activate spyglass`
+#
+# (lines intentionally left blank)
+#
+#
+name: spyglass-moseq-gpu
+channels:
+  - conda-forge
+  # - defaults # deprecated
+  - franklab
+  - edeno
+  # - pytorch # dlc-only
+  # - anaconda  # dlc-only, for cudatoolkit
+dependencies:
+  - bottleneck
+  # - cudatoolkit=11.3 # dlc-only
+  # - ffmpeg # dlc-only
+  - ipympl
+  - jupyterlab>=3.*
+  # - libgcc # dlc-only
+  - matplotlib
+  - non_local_detector
+  - numpy
+  - pip
+  - position_tools
+  - pybind11  # req by mountainsort4 -> isosplit5
+  - pydotplus
+  - pyfftw<=0.12.0  # ghostipy req. install from conda-forge for Mac ARM
+  - python>=3.9,<3.13
+  - pytorch<1.12.0
+  - ripple_detection
+  - seaborn
+  # - torchaudio # dlc-only
+  # - torchvision # dlc-only
+  - track_linearization>=2.3
+  - pip:
+      - ghostipy # for common_filter
+      - mountainsort4
+      - .[moseq-gpu]

maintenance_scripts/README.md

@@ -26,11 +26,40 @@ regularly as cron jobs.
 1. Clone the repository to the desired location.
 2. Set up a config file by copying `dj_local_conf_example.json` to
     `dj_local_conf.json` and filling in the necessary information.
-3. Set up a cron job to run `run_jobs.sh` at the desired interval by running
+3. Copy the `example.env` file to `.env` in the `maintenance_scripts` directory
+    and fill in the necessary information, including...
+    - `SPYGLASS_CONDA_ENV`: the name of the conda environment with Spyglass and
+        DataJoint installed.
+    - `SPYGLASS_REPO_PATH`: the path to the Spyglass repository.
+    - `SPYGLASS_LOG`: the path to the log file.
+    - Optional email settings. If not set, email notifications will not be sent.
+        - `SPYGLASS_EMAIL_SRC`: The email address from which to send notifications.
+        - `SPYGLASS_EMAIL_PASS`: the password for the email address.
+        - `SPYGLASS_EMAIL_DEST`: the email address to which to send notifications.
+4. Set up a cron job to run `run_jobs.sh` at the desired interval by running
     `crontab -e` and adding the script.
 
+Note that the log file will automatically be truncated to `SPYGLASS_MAX_LOG`
+lines on each run. 1000 lines should be sufficient.
+
+### Example Cron Job
+
 In the following example, the script is set to run every Monday at 4:00 AM.
 
 ```text
 0 4 * * 1 /path/to/run_jobs.sh
 ```
+
+### Email Service
+
+The script uses `curl` to send email notifications on failure. While this can
+work with
+[many email services](https://everything.curl.dev/usingcurl/smtp.html), Gmail is
+a common choice. To use Gmail, you will need to ...
+
+1. Turn on [2-step verification](https://myaccount.google.com/security-checkup)
+2. Turn on [less secure apps](https://myaccount.google.com/lesssecureapps)
+3. Create an [app password](https://myaccount.google.com/apppasswords)
+
+`curl` will not work with your master Gmail password, so you will need to use
+the app password instead.

maintenance_scripts/example.env

@@ -0,0 +1,8 @@
+# NOTE: Use quotes for strings with spaces
+export SPYGLASS_CONDA_ENV=spyglass # name of conda environment
+export SPYGLASS_REPO_PATH=/home/franklab/spyglass # path to spyglass repo
+export SPYGLASS_LOG=/home/franklab/spyglass.log # log file
+export SPYGLASS_EMAIL_SRC=[email protected] # optional, email sender
+export SPYGLASS_EMAIL_PASS="example password" # optional, email password
+export SPYGLASS_EMAIL_DEST=[email protected] # optional, email recipient
+export SPYGLASS_MAX_LOG=1000 # number of lines to keep in log

maintenance_scripts/run_jobs.sh

@@ -1,14 +1,52 @@
 #!/bin/bash
+# AUTHOR: Chris Brozdowski
+# DATE: 2025-02-20
+#
+# 1. Go to SPYGLASS_REPO_PATH and pull the latest changes from the master branch
+# 2. Test the SPYGLASS_CONDA_ENV conda environment
+# 3. Test the connection to the database
+# 4. Run the cleanup script
+#
+# This script is intended to be run as a cron job, weekly or more frequently.
+# It will store a log of its output in SPYGLASS_LOG.
+# If any of the operations fail, an email will be sent to SPYGLASS_EMAIL_DEST
 
-# SETUP:
-# 1. Create a conda environment with datajoint installed.
-# 2. Edit the variables below to match your setup.
-# 3. Set up the cron job (See README.md)
-# Note that the log file will be truncated to the last 1000 lines.
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$SCRIPT_DIR/.env" # load environment variables from this directory
 
-SPYGLASS_CONDA_ENV=spyglass
-SPYGLASS_REPO_PATH=/home/franklab/spyglass
-SPYGLASS_LOG=/home/franklab/spyglass/spyglass.log
+if [[ -z "${SPYGLASS_CONDA_ENV}" \
+  || -z "${SPYGLASS_REPO_PATH}" \
+  || -z "${SPYGLASS_LOG}" ]]; then
+  echo "Error: SPYGLASS_CONDA_ENV, SPYGLASS_REPO_PATH,
+        and SPYGLASS_LOG must be set in .env"
+  exit 1
+fi
+
+EMAIL_TEMPLATE=$(cat <<-EOF
+From: "Spyglass" <$SPYGLASS_EMAIL_SRC>
+To: $SPYGLASS_EMAIL_DEST
+Subject: cron fail - $(date "+%Y-%m-%d")
+
+%s
+EOF
+)
+
+on_fail() { # $1: error message. Echo message and send as email
+    echo "Error: $1"
+    if [ -z "$SPYGLASS_EMAIL_SRC" ]; then
+      return 1 # No email source, so don't send an email
+    fi
+    local error_msg="$1"
+    local content
+    content=$(printf "$EMAIL_TEMPLATE" "$error_msg")
+
+    curl -o /dev/null --ssl-reqd \
+      --url "smtps://smtp.gmail.com:465" \
+      --user "${SPYGLASS_EMAIL_SRC}:${SPYGLASS_EMAIL_PASS}" \
+      --mail-from "$SPYGLASS_EMAIL_SRC" \
+      --mail-rcpt "$SPYGLASS_EMAIL_DEST" \
+      -T <(echo "$content")
+}
 
 exec >> $SPYGLASS_LOG 2>&1
 
@@ -17,30 +55,33 @@ echo "SPYGLASS CRON JOB START: $(date +"%Y-%m-%d %H:%M:%S")"
 
 # Run from the root of the spyglass repository
 cd $SPYGLASS_REPO_PATH || \
-    { echo "Error: Could not change to the spyglass directory"; exit 1; }
+    { on_fail "Could not find repo path: $SPYGLASS_REPO_PATH"; exit 1; }
+
 
 # Update the spyglass repository
-git pull https://github.com/LorenFrankLab/spyglass.git master > /dev/null || \
-    { echo "Error: $PWD Could not update the spyglass repository"; exit 1; }
+git pull --quiet \
+  https://github.com/LorenFrankLab/spyglass.git master > /dev/null || \
+    { on_fail "Could not update the spyglass repo $PWD"; exit 1; }
 
 # Test conda environment
 if ! conda env list | grep -q $SPYGLASS_CONDA_ENV; then
-    echo "Error: Conda environment $SPYGLASS_CONDA_ENV not found"
-    exit 1
+  on_fail "Conda environment $SPYGLASS_CONDA_ENV not found"
+  exit 1
 fi
 
 # convenience function to run a command in the spyglass conda environment
 conda_run() { conda run --name $SPYGLASS_CONDA_ENV "$@"; }
 
 # Test connection to the database
-conda_run python -c "import datajoint as dj; dj.conn()" > /dev/null || \
-    { echo "Error: Could not connect to the database"; exit 1; }
+CONN_TEST="import datajoint as dj; dj.logger.setLevel('ERROR'); dj.conn()"
+conda_run python -c "$CONN_TEST" > /dev/null || \
+  { on_fail "Could not connect to the database"; exit 1; }
 
 # Run cleanup script
 conda_run python maintenance_scripts/cleanup.py
 
 echo "SPYGLASS CRON JOB END"
 
 # truncate long log file
-tail -n 1000 "$SPYGLASS_LOG" > "${SPYGLASS_LOG}.tmp" && \
-  mv "${SPYGLASS_LOG}.tmp" "$SPYGLASS_LOG"
+tail -n ${SPYGLASS_MAX_LOG:-1000} "$SPYGLASS_LOG" > "${SPYGLASS_LOG}.tmp" \
+  && mv "${SPYGLASS_LOG}.tmp" "$SPYGLASS_LOG"

notebook-images/moseq_outline.png

@@ -285,7 +285,47 @@
     "The Decoding pipeline relies on `jax` to process data with GPUs. Please see\n",
     "their conda installation steps\n",
     "[here](https://jax.readthedocs.io/en/latest/installation.html#conda-installation).\n",
-    "\n"
+    "\n",
+    "#### Deep Lab Cut (DLC)\n",
+    "\n",
+    "Spyglass provides an environment build for using the DLC pipeline. To create an \n",
+    "environment with these features, please:\n",
+    "1. navigate to your cloned spyglass repo.\n",
+    "2. build the environment from the dlc version\n",
+    "3. activate the environment to use\n",
+    "\n",
+    "```bash\n",
+    "cd /path/to/spyglass # 1\n",
+    "mamba env create -f environment_dlc.yml # 2\n",
+    "mamba activate spyglass-dlc # 3\n",
+    "```\n",
+    "\n",
+    "Alternatively, you can pip install using\n",
+    "```bash\n",
+    "pip install spyglass[dlc]\n",
+    "```\n",
+    "\n",
+    "#### Keypoint-Moseq\n",
+    "\n",
+    "Spyglass provides an environment build for using the Moseq pipeline. To create an \n",
+    "environment with these features, please:\n",
+    "1. navigate to your cloned spyglass repo.\n",
+    "2. build the environment from one of the moseq versions\n",
+    "3. activate the environment to use\n",
+    "\n",
+    "```bash\n",
+    "cd /path/to/spyglass # 1\n",
+    "mamba env create -f environment_moseq_cpu.yml # 2\n",
+    "mamba activate spyglass-moseq-cpu # 3\n",
+    "```\n",
+    "\n",
+    "Alternatively, you can pip install using\n",
+    "```bash\n",
+    "pip install spyglass[moseq-cpu]\n",
+    "```\n",
+    "\n",
+    "To use a GPU enabled version of the package, replace `cpu` with `gpu` in the above \n",
+    "commands\n"
    ]
   },
   {
@@ -593,7 +633,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "spyglass",
    "language": "python",
    "name": "python3"
   },
@@ -607,7 +647,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.9.19"
+   "version": "3.9.18"
   }
  },
  "nbformat": 4,