Merge pull request #198 from kushalbakshi/new-main

MAJOR REFACTOR: remove extra modules of imaging

Author: ttngu207

.github/workflows/test.yaml

@@ -1,10 +1,12 @@
 name: Test
 on:
   push:
+    branches:
+      - main
   pull_request:
+    branches:
+      - main
   workflow_dispatch:
-  schedule:
-    - cron: "0 8 * * 1"
 jobs:
   devcontainer-build:
     uses: datajoint/.github/.github/workflows/devcontainer-build.yaml@main
@@ -13,12 +15,7 @@ jobs:
     strategy:
       matrix:
         py_ver: ["3.9", "3.10"]
-        mysql_ver: ["8.0", "5.7"]
-        include:
-          - py_ver: "3.8"
-            mysql_ver: "5.7"
-          - py_ver: "3.7"
-            mysql_ver: "5.7"
+        mysql_ver: ["8.0"]
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python ${{matrix.py_ver}}
@@ -34,4 +31,3 @@ jobs:
           python_version=${{matrix.py_ver}}
           black element_calcium_imaging --check --verbose --target-version py${python_version//.}
           black notebooks --check --verbose --target-version py${python_version//.}
-

CHANGELOG.md

@@ -3,6 +3,11 @@
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
 [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
+## [0.11.0] - 2025-01-31
+
++ Feat - Single `imaging` module for all imaging data
++ Update - Code fixes and improvements throughout
+
 ## [0.10.1] - 2024-06-20
 
 + Fix - cleaner plotting in tutorial notebook 
@@ -220,6 +225,7 @@ Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
 + Add - `scan` and `imaging` modules
 + Add - Readers for `ScanImage`, `ScanBox`, `Suite2p`, `CaImAn`
 
+[0.11.0]: https://github.com/datajoint/element-calcium-imaging/releases/tag/0.11.0
 [0.10.0]: https://github.com/datajoint/element-calcium-imaging/releases/tag/0.10.0
 [0.9.5]: https://github.com/datajoint/element-calcium-imaging/releases/tag/0.9.5
 [0.9.4]: https://github.com/datajoint/element-calcium-imaging/releases/tag/0.9.4

README.md

@@ -1,32 +1,34 @@
 # DataJoint Element for Functional Calcium Imaging
 
-DataJoint Element for functional calcium imaging with 
-[ScanImage](https://docs.scanimage.org/), 
-[Scanbox](https://scanbox.org/),
-[Nikon NIS-Elements](https://www.microscope.healthcare.nikon.com/products/software/nis-elements), 
-and `Bruker Prairie View` acquisition software; and 
-[Suite2p](https://github.com/MouseLand/suite2p), 
+DataJoint Element for functional calcium imaging with support for
+[ScanImage](https://docs.scanimage.org/), [Scanbox](https://scanbox.org/), [Nikon
+NIS-Elements](https://www.microscope.healthcare.nikon.com/products/software/nis-elements),
+and `Bruker Prairie View` acquisition software; and
+[Suite2p](https://github.com/MouseLand/suite2p),
 [CaImAn](https://github.com/flatironinstitute/CaImAn), and
-[EXTRACT](https://github.com/schnitzer-lab/EXTRACT-public) analysis 
-software. DataJoint Elements collectively standardize and automate
-data collection and analysis for neuroscience experiments. Each Element is a modular
-pipeline for data storage and processing with corresponding database tables that can be
-combined with other Elements to assemble a fully functional pipeline. This repository 
-also provides a tutorial environment and notebooks to learn the pipeline.
+[EXTRACT](https://github.com/schnitzer-lab/EXTRACT-public) analysis software. DataJoint
+Elements collectively standardize and automate data collection and analysis for
+neuroscience experiments. Each Element is a modular pipeline for data storage and
+processing with corresponding database tables that can be combined with other Elements
+to assemble a fully functional pipeline. This repository also provides a tutorial
+environment and notebooks to learn the pipeline.
 
 ## Experiment Flowchart
 
 ![flowchart](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/flowchart.svg)
 
 ## Data Pipeline Diagram
 
-![pipeline](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/pipeline_imaging.svg)
+![pipeline](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/pipeline_imaging_no_curation.svg)
 
-+ We have designed three variations of the pipeline to handle different use cases. 
-Displayed above is the default `imaging` schema.  Details on all of the `imaging` 
-schemas can be found in the [Data 
-Pipeline](https://datajoint.com/docs/elements/element-calcium-imaging/latest/pipeline/) 
-documentation page.
+### Legacy Support
+
++ Three variations of the pipeline were designed and supported through December 2024 to
+  handle different use cases. However, based on community feedback and use cases, only
+  one pipeline will be supported after December 2024. However, all three pipeline
+  versions will remain available in their own branches for reference and use.
++ Displayed above is the default `imaging` schema. Please see other branches for the
+  `imaging` module with `Curation` table and the `imaging_preprocess` module.
 
 ## Getting Started
 
@@ -55,37 +57,64 @@ or contact our team by email at [email protected].
 
 ## Interactive Tutorial
 
-+ The easiest way to learn about DataJoint Elements is to use the tutorial notebooks within the included interactive environment configured using [Dev Container](https://containers.dev/).
++ The easiest way to learn about DataJoint Elements is to use the tutorial notebooks
+  within the included interactive environment configured using [Dev
+  Container](https://containers.dev/).
 
 ### Launch Environment
 
 Here are some options that provide a great experience:
 
-- (*recommended*) Cloud-based Environment
-  - Launch using [GitHub Codespaces](https://github.com/features/codespaces) using the `+` option which will `Create codespace on main` in the codebase repository on your fork with default options. For more control, see the `...` where you may create `New with options...`.
-  - Build time for a codespace is a few minutes. This is done infrequently and cached for convenience.
-  - Start time for a codespace is less than 1 minute. This will pull the built codespace from cache when you need it.
-  - *Tip*: Each month, GitHub renews a [free-tier](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts) quota of compute and storage. Typically we run into the storage limits before anything else since Codespaces consume storage while stopped. It is best to delete Codespaces when not actively in use and recreate when needed. We'll soon be creating prebuilds to avoid larger build times. Once any portion of your quota is reached, you will need to wait for it to be reset at the end of your cycle or add billing info to your GitHub account to handle overages.
-  - *Tip*: GitHub auto names the codespace but you can rename the codespace so that it is easier to identify later.
-
-- Local Environment
-  > *Note: Access to example data is currently limited to MacOS and Linux due to the s3fs utility. Windows users are recommended to use the above environment.*
-  - Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
-  - Install [Docker](https://docs.docker.com/get-docker/)
-  - Install [VSCode](https://code.visualstudio.com/)
-  - Install the VSCode [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
-  - `git clone` the codebase repository and open it in VSCode
-  - Use the `Dev Containers extension` to `Reopen in Container` (More info is in the `Getting started` included with the extension.)
-
-You will know your environment has finished loading once you either see a terminal open related to `Running postStartCommand` with a final message of `Done` or the `README.md` is opened in `Preview`.
++ (*recommended*) Cloud-based Environment
+  + Launch using [GitHub Codespaces](https://github.com/features/codespaces) using the
+    `+` option which will `Create codespace on main` in the codebase repository on your
+    fork with default options. For more control, see the `...` where you may create `New
+    with options...`.
+  + Build time for a codespace is a few minutes. This is done infrequently and cached
+    for convenience.
+  + Start time for a codespace is less than 1 minute. This will pull the built codespace
+    from cache when you need it.
+  + *Tip*: Each month, GitHub renews a
+    [free-tier](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts)
+    quota of compute and storage. Typically we run into the storage limits before
+    anything else since Codespaces consume storage while stopped. It is best to delete
+    Codespaces when not actively in use and recreate when needed. We'll soon be creating
+    prebuilds to avoid larger build times. Once any portion of your quota is reached,
+    you will need to wait for it to be reset at the end of your cycle or add billing
+    info to your GitHub account to handle overages.
+  + *Tip*: GitHub auto names the codespace but you can rename the codespace so that it
+    is easier to identify later.
+
++ Local Environment
+  > *Note: Access to example data is currently limited to MacOS and Linux due to the
+  > s3fs utility. Windows users are recommended to use the above environment.*
+  + Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+  + Install [Docker](https://docs.docker.com/get-docker/)
+  + Install [VSCode](https://code.visualstudio.com/)
+  + Install the VSCode [Dev Containers
+    extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
+  + `git clone` the codebase repository and open it in VSCode
+  + Use the `Dev Containers extension` to `Reopen in Container` (More info is in the
+    `Getting started` included with the extension.)
+
+You will know your environment has finished loading once you either see a terminal open
+related to `Running postStartCommand` with a final message of `Done` or the `README.md`
+is opened in `Preview`.
 
 Once the environment has launched, please run the following command in the terminal:
-```
+```sh
 MYSQL_VER=8.0 docker compose -f docker-compose-db.yaml up --build -d
 ```
 
 ### Instructions
 
-1. We recommend you start by navigating to the `notebooks` directory on the left panel and go through the `tutorial.ipynb` Jupyter notebook. Execute the cells in the notebook to begin your walk through of the tutorial.
+1. We recommend you start by navigating to the `notebooks` directory on the left panel
+   and go through the `tutorial.ipynb` Jupyter notebook. Execute the cells in the
+   notebook to begin your walk through of the tutorial.
 
-1. Once you are done, see the options available to you in the menu in the bottom-left corner. For example, in Codespace you will have an option to `Stop Current Codespace` but when running Dev Container on your own machine the equivalent option is `Reopen folder locally`. By default, GitHub will also automatically stop the Codespace after 30 minutes of inactivity.  Once the Codespace is no longer being used, we recommend deleting the Codespace.
+2. Once you are done, see the options available to you in the menu in the bottom-left
+   corner. For example, in Codespace you will have an option to `Stop Current Codespace`
+   but when running Dev Container on your own machine the equivalent option is `Reopen
+   folder locally`. By default, GitHub will also automatically stop the Codespace after
+   30 minutes of inactivity.  Once the Codespace is no longer being used, we recommend
+   deleting the Codespace.

docs/src/partnerships.md

@@ -1,6 +1,9 @@
 # Key partnerships
 
-Several labs have developed DataJoint-based data management and processing pipelines for two-photon calcium imaging. Our team collaborated with several of them during their projects. Additionally, we interviewed these teams to understand their experiment workflow, pipeline design, associated tools, and interfaces.
+Several labs have developed DataJoint-based data management and processing pipelines for
+two-photon calcium imaging. Our team collaborated with several of them during their
+projects. Additionally, we interviewed these teams to understand their experiment
+workflow, pipeline design, associated tools, and interfaces.
 
 These teams include:
 

docs/src/pipeline.md

@@ -5,33 +5,23 @@ corresponding table in the database.  Within the pipeline, Element Calcium Imagi
 connects to upstream Elements including Lab, Animal, Session, and Event. For more 
 detailed documentation on each table, see the API docs for the respective schemas.
 
-The Element is composed of two main schemas, `scan` and `imaging`. To handle
-several use cases of this pipeline, we have designed two alternatives to the `imaging` 
-schema, including `imaging_no_curation` and `imaging_preprocess`.
+The Element is composed of two main schemas, `scan` and `imaging`.
 
 ## Diagrams
 
 ### `imaging` module
 
 - Multiple scans are acquired during each session and each scan is processed independently.
 
-     ![pipeline](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/pipeline_imaging.svg)
-
-### `imaging_no_curation` module
-
-- Same as the `imaging` module, but without the `Curation` table.
-
      ![pipeline](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/pipeline_imaging_no_curation.svg)
 
-### `imaging_preprocess` module
-
-- Same as the `imaging` module, and additional pre-processing steps can be performed on each scan prior to processing with Suite2p or CaImAn.
-
-     ![pipeline](https://raw.githubusercontent.com/datajoint/element-calcium-imaging/main/images/pipeline_imaging_preprocess.svg)
-
 ### `multi-scan-processing` branch
 
-- The processing pipeline is typically performed on a per-scan basis, however, depending on the nature of the research questions, different labs may opt to perform processing/segmentation on a concatenated set of data from multiple scans. To this end, we have extended the Calcium Imaging Element and provided a design version capable of supporting a multi-scan processing scheme.
+- The processing pipeline is typically performed on a per-scan basis, however, depending
+  on the nature of the research questions, different labs may opt to perform
+  processing/segmentation on a concatenated set of data from multiple scans. To this
+  end, we have extended the Calcium Imaging Element and provided a design version
+  capable of supporting a multi-scan processing scheme.
 
 ## Table descriptions
 
@@ -89,7 +79,6 @@ schema, including `imaging_no_curation` and `imaging_preprocess`.
 | MaskType | Available labels for segmented masks |
 | ProcessingTask | Task defined by a combination of Scan and ProcessingParamSet |
 | Processing | The core table that executes a ProcessingTask |
-| Curation | Curated results |
 | MotionCorrection | Results of the motion correction procedure |
 | MotionCorrection.RigidMotionCorrection | Details of the rigid motion correction performed on the imaging data |
 | MotionCorrection.NonRigidMotionCorrection | Details of nonrigid motion correction performed on the imaging data |

docs/src/roadmap.md

@@ -18,9 +18,12 @@ the common motifs to create Element Calcium Imaging. Major features include:
 - [ ] Deepinterpolation
 - [x] Data export to NWB
 - [x] Data publishing to DANDI
-- [x] Widgets for manual ROI mask creation and curation for cell segmentation of Fluorescent voltage sensitive indicators, neurotransmitter imaging, and neuromodulator imaging
-- [ ] Expand creation widget to provide pixel weights for each mask based on Fluorescence intensity traces at each pixel
+- [x] Widgets for manual ROI mask creation and curation for cell segmentation of
+  Fluorescent voltage sensitive indicators, neurotransmitter imaging, and neuromodulator
+  imaging
+- [ ] Expand creation widget to provide pixel weights for each mask based on
+  Fluorescence intensity traces at each pixel
 
 Further development of this Element is community driven. Upon user requests and based on
-guidance from the Scientific Steering Group we will continue adding features to this 
+guidance from the Scientific Steering Group, we will continue adding features to this 
 Element.

element_calcium_imaging/__init__.py

@@ -0,0 +1,3 @@
+from . import imaging
+
+imaging_no_curation = imaging  # alias for backward compatibility