Merge pull request #45 from kabilar/main

Add quality metrics

Author: kushalbakshi

.github/workflows/u24_element_release_call.yaml

@@ -25,4 +25,4 @@ jobs:
       py_ver: 3.9
     secrets:
       TWINE_USERNAME: ${{secrets.TWINE_USERNAME}}
-      TWINE_PASSWORD: ${{secrets.TWINE_PASSWORD}}
+      TWINE_PASSWORD: ${{secrets.TWINE_PASSWORD}}

CHANGELOG.md

@@ -3,6 +3,11 @@
 Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
 [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
 
+## [0.3.0] - 2023-05-15
+
++ Add - Quality metrics
++ Update - README and docs to reflect new structure across all Elements
+
 ## [0.2.1] - 2023-05-11
 
 + Fix - `.ipynb` dark mode output for all notebooks.
@@ -37,6 +42,7 @@ Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and
 + Add - Load data acquired with Miniscope-DAQ-V3
 + Add - Load data analyzed with MiniscopeAnalysis
 
+[0.3.0]: https://github.com/datajoint/element-miniscope/releases/tag/0.3.0
 [0.2.1]: https://github.com/datajoint/element-miniscope/releases/tag/0.2.1
 [0.2.0]: https://github.com/datajoint/element-miniscope/releases/tag/0.2.0
 [0.1.4]: https://github.com/datajoint/element-miniscope/releases/tag/0.1.4

README.md

@@ -2,14 +2,33 @@
 
 DataJoint Element for functional calcium imaging data acquired with the
 [UCLA Miniscope](https://github.com/Aharoni-Lab/Miniscope-v4) and
-[Miniscope DAQ](https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software)
-acquisition system, and analyzed with `CaImAn`. DataJoint Elements collectively
-standardize and automate data collection and analysis for neuroscience experiments.
-Each Element is a modular pipeline for data storage and processing with corresponding
-database tables that can be combined with other Elements to assemble a fully functional
-pipeline.
+[Miniscope DAQ](https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software) acquisition 
+system, and analyzed with [CaImAn](https://github.com/flatironinstitute/CaImAn). 
+DataJoint Elements collectively standardize and automate data collection and analysis 
+for neuroscience experiments. Each Element is a modular pipeline for data storage and 
+processing with corresponding database tables that can be combined with other Elements 
+to assemble a fully functional pipeline.
 
-![diagram](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/diagram_flowchart.svg)
+## Experiment Flowchart
 
-Installation and usage instructions can be found at the
-[Element documentation](https://datajoint.com/docs/elements/element-miniscope).
+![flowchart](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/flowchart.svg)
+
+## Data Pipeline Diagram
+
+![pipeline](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/pipeline.svg)
+
+## Getting Started
+
++ Install from PyPI
+
+     ```bash
+     pip install element-miniscope
+     ```
+
++ [Interactive tutorial on GitHub Codespaces](https://github.com/datajoint/workflow-miniscope#interactive-tutorial)
+
++ [Documentation](https://datajoint.com/docs/elements/element-miniscope)
+
+## Support
+
++ If you need help getting started or run into any errors, please contact our team by email at [email protected].

docs/.docker/pip_requirements.txt

@@ -8,4 +8,5 @@ mkdocs-gen-files
 mkdocs-literate-nav
 mkdocs-exclude-search
 mkdocs-markdownextradata-plugin
-mkdocs-jupyter
+mkdocs-jupyter
+mkdocs-section-index

docs/docker-compose.yaml

@@ -14,7 +14,6 @@ services:
       - PACKAGE
       - UPSTREAM_REPO
       - MODE
-      - GOOGLE_ANALYTICS_KEY
       - PATCH_VERSION 
     volumes:
       - ../docs:/main/docs

docs/mkdocs.yaml

@@ -6,18 +6,13 @@ repo_url: https://github.com/datajoint/element-miniscope
 repo_name: datajoint/element-miniscope
 nav:
   - Element Miniscope: index.md
-  - Concepts: concepts.md
+  - Data Pipeline: pipeline.md
   - Tutorials: 
-    - Overview: tutorials/index.md
-    - Data Download: tutorials/00-DataDownload_Optional.ipynb
-    - Configure: tutorials/01-Configure.ipynb
-    - Workflow Structure: tutorials/02-WorkflowStructure_Optional.ipynb
-    - Process: tutorials/03-Process.ipynb
-    - Automate: tutorials/04-Automate_Optional.ipynb
-    - Explore: tutorials/05-Explore.ipynb
-    - Drop: tutorials/06-Drop_Optional.ipynb
-    - Downstream Analysis: tutorials/07-DownstreamAnalysis_Optional.ipynb
-    - Visualizations: tutorials/08-Visualizations.ipynb
+    - tutorials/index.md
+    - Quality Metrics Notebook: tutorials/quality_metrics.ipynb
+  - Concepts: concepts.md
+  - Key Partnerships: partnerships.md
+  - Roadmap: roadmap.md
   - Citation: citation.md
   - API: api/ # defer to gen-files + literate-nav
   - Changelog: changelog.md
@@ -52,17 +47,14 @@ nav:
 #     UPSTREAM_REPO=https://github.com/datajoint/element-{ELEMENT}.git \
 #     HOST_UID=$(id -u) docker compose -f docs/docker-compose.yaml up --build
 #     ```
-# 02. Site analytics depend on a local environment variable GOOGLE_ANALYTICS_KEY
-#     You can find this in LastPass or declare with any string to suppress errors
-# 03. The API section will pull docstrings.
-#     A. Follow google style guide e.g.,
+# 02. The API section will pull docstrings.
+#     A. Follow google styleguide e.g.,
 #        https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html
 #        With typing suggestions: https://docs.python.org/3/library/typing.html
 #     B. To pull a specific workflow fork, change ./docs/src/api/make_pages.py#L19
-# 04. To see your fork of the workflow-{element} in this render, change the
+# 03. To see your fork of the workflow-{element} in this render, change the
 #     URL in ./docs/src/api/make_pages.py#L19 to your fork.
-# 05. For redirecting options For redirect options, see 'redirects' below.
-# 06. To deploy this site on your fork,
+# 04. To deploy this site on your fork,
 #     A. declare a branch called gh-pages
 #     B. go to the your fork > settings > pages
 #     C. direct pages to render from the gh-pages branch at root
@@ -97,9 +89,6 @@ theme:
 plugins:
   - markdownextradata: {}
   - search
-  # - redirects: # OPTIONAL REDIRECTS
-  #     redirect_maps:
-  #       "index.md": "getting_started.md"
   - mkdocstrings:
       default_handler: python
       handlers:
@@ -118,7 +107,8 @@ plugins:
         - "*/navigation.md"
   - mkdocs-jupyter:
       ignore_h1_titles: True
-      
+      include: ["*.ipynb"]
+  - section-index
 markdown_extensions:
   - attr_list
   - toc:
@@ -139,14 +129,13 @@ markdown_extensions:
       linenums: true
   - pymdownx.inlinehilite
   - pymdownx.snippets
+  - pymdownx.magiclink # Displays bare URLs as links
+  - pymdownx.tasklist: # Renders check boxes in tasks lists
+      custom_checkbox: true
   - footnotes
-
 extra:
   PATCH_VERSION: !ENV PATCH_VERSION
   generator: false # Disable watermark
-  analytics:
-    provider: google
-    property: !ENV GOOGLE_ANALYTICS_KEY
   version:
     provider: mike
   social:

docs/src/api/make_pages.py

@@ -18,6 +18,7 @@
 ):
     if (
         path.stem == "__init__"
+        or path.stem == "version"
         or "plotting" in path.parts  # Workaround for mkdocstring-python subfolder error
     ):
         continue

docs/src/citation.md

@@ -7,5 +7,4 @@ Resource Identifier (RRID).
   Reimer J, Walker EY, Tolias AS. DataJoint Elements: Data Workflows for
   Neurophysiology. bioRxiv. 2021 Jan 1. doi: https://doi.org/10.1101/2021.03.30.437358
 
-+ DataJoint Elements ([RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894)) -
-  Element Miniscope (version {{ PATCH_VERSION }})
++ DataJoint Element Miniscope - [RRID:SCR_021894](https://scicrunch.org/resolver/SCR_021894) - Version {{ PATCH_VERSION }}

docs/src/concepts.md

@@ -97,106 +97,3 @@ are open source, they are often developed by an individual during their training
 and lack funding for long term maintenance. These factors result in a lack of
 standardization for miniscope preprocessing tools, which is a major obstacle to adoption
 for new labs.
-
-## Key Partnerships
-
-The DataJoint team have been in contact with the following teams who are eager to engage
-and adopt DataJoint-based workflows in their labs.
-
-- Adrien Peyrache Lab, McGill University
-- Peyman Golshani Lab, UCLA
-- Daniel Aharoni Lab, UCLA
-- Anne Churchland Lab, UCLA
-- Fan Wang Lab, MIT
-- Antoine Adamantidis Lab, University of Bern
-- Manolis Froudaraki Lab, FORTH
-- Allan Basbaum Lab, UCSF
-
-## Element Architecture
-
-Each of the DataJoint Elements are a set of tables for common neuroinformatics
-modalities to organize, preprocess, and analyze data. Each node in the following diagram
-is either a table in the Element itself or a table that would be connected to the
-Element.
-
-![element-miniscope diagram](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/attached_miniscope_element.svg)
-
-### `subject` schema ([API docs](https://datajoint.com/docs/elements/element-animal/api/element_animal/subject))
-
-Although not required, most choose to connect the `Session` table to a `Subject` table.
-
-| Table | Description |
-| --- | --- |
-| Subject | Basic information of the research subject. |
-
-### `session` schema ([API docs](https://datajoint.com/docs/elements/element-session/api/element_session/session_with_datetime))
-
-| Table | Description |
-| --- | --- |
-| Session | Unique experimental session identifier. |
-
-### `miniscope` schema ([API docs](../api/element_miniscope/miniscope))
-
-Tables related to importing, analyzing, and exporting miniscope data.
-
-| Table | Description |
-| --- | --- |
-| Recording | A table containing information about the equipment used (e.g. the acquisition hardware information). |
-| RecordingInfo |  The metadata about this recording from the Miniscope DAQ software (e.g. frame rate, number of channels, frames, etc.). |
-| MotionCorrection | A table with information about motion correction performed on a recording. |
-| MotionCorrection.RigidMotionCorrection | A table with details of rigid motion correction (e.g. shifting in x, y). |
-| MotionCorrection.NonRigidMotionCorrection and MotionCorrection.Block | These tables describe the non-rigid motion correction. |
-| MotionCorrection.Summary | A table containing summary images after motion correction. |
-| Segmentation | This table specifies the segmentation step and its outputs, following the motion correction step. |
-| Segmentation.Mask | This table contains the image mask for the segmented region of interest. |
-| MaskClassification | This table contains information about the classification of `Segmentation.Mask` into a type (e.g. soma, axon, dendrite, artifact, etc.). |
-| Fluorescence | This table contains the fluorescence traces extracted from each `Segmentation.Mask`. |
-| ActivityExtractionMethod | A table with information about the activity extraction method (e.g. deconvolution) applied on the fluorescence trace. |
-| Activity | A table with neuronal activity traces from fluorescence trace (e.g. spikes). |
-
-### `miniscope_report` schema ([API docs](../api/element_miniscope/miniscope_report))
-
-Tables related to summary reports of miniscope data.
-
-| Table | Description |
-| --- | --- |
-| QualityMetrics | A table containing information about CaImAn estimates. |
-
-The above `QualityMetrics` table includes the following for each component in the CaImAn
-analysis:
-
-- `r_values`: Space correlation.
-- `snr`: Trace SNR.
-- `cnn_preds`: CNN predictions.
-
-## Pipeline Development
-
-With assistance from Dr. Peyman Golshani's Lab (UCLA) we have added support for the UCLA
-Miniscope DAQ V3 acquisition tool and MiniscopeAnalysis preprocessing tool in
-`element-miniscope` and `workflow-miniscope`. They have provided example data for
-development.
-
-Based on interviews, we are considering adding support for the tools listed below. The
-deciding factors include the number of users, long term support, quality controls, and
-python programming language (so that the preprocessing tool can be triggered within the
-element).
-
-- Acquisition tools + Miniscope DAQ V4 + Inscopix Data Acquisition Software (IDAS)
-- Preprocessing tools + Inscopix Data Processing Software (IDPS) + Inscopix Multimodal Image
-  Registration and Analysis (MIRA) + MiniAn + CaImAn + CNMF-E + CellReg
-
-## Roadmap
-
-Further development of this Element is community driven. Upon user requests and based on
-guidance from the Scientific Steering Group we will add features to this Element, such
-as:
-
-- Acquisition & Preprocessing tools
-- Inscopix
-  - Data Acquisition Software (IDAS)
-  - Data Processing Software (IDPS)
-  - Multimodal Image Registration and Analysis (MIRA)
-- MiniAn
-- CaImAn
-- CNMF-E
-- CellReg

docs/src/index.md

@@ -1,15 +1,36 @@
 # Element Miniscope Calcium Imaging
 
-DataJoint Element for functional calcium imaging data acquired with the [UCLA Miniscope](https://github.com/Aharoni-Lab/Miniscope-v4)
-and [Miniscope DAQ](https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software)
-acquisition system, and
-analyzed with `CaImAn`. DataJoint Elements collectively standardize and automate data
-collection and analysis for neuroscience experiments. Each Element is a modular
-pipeline for data storage and processing with corresponding database tables that can be
-combined with other Elements to assemble a fully functional pipeline.
-
-![diagram](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/diagram_flowchart.svg)
-
-Visit the [Concepts page](./concepts.md) for more information about miniscope calcium
-imaging and Element Miniscope. To get started with building your data pipeline visit
-the [Tutorials page](./tutorials/).
+DataJoint Element for functional calcium imaging data acquired with the
+[UCLA Miniscope](https://github.com/Aharoni-Lab/Miniscope-v4) and
+[Miniscope DAQ](https://github.com/Aharoni-Lab/Miniscope-DAQ-QT-Software) acquisition 
+system, and analyzed with [CaImAn](https://github.com/flatironinstitute/CaImAn). 
+DataJoint Elements collectively standardize and automate data collection and analysis 
+for neuroscience experiments. Each Element is a modular pipeline for data storage and 
+processing with corresponding database tables that can be combined with other Elements 
+to assemble a fully functional pipeline.
+
+## Experiment Flowchart
+
+![flowchart](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/flowchart.svg)
+
+## Data Pipeline Diagram
+
+![pipeline](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/pipeline.svg)
+
+## Getting Started
+
++ Install from PyPI
+
+     ```bash
+     pip install element-miniscope
+     ```
+
++ [Data Pipeline](./pipeline.md) - Pipeline and table descriptions
+
++ [Tutorials](./tutorials/index.md) - Start building your data pipeline
+
++ [Code Repository](https://github.com/datajoint/element-miniscope/){:target="_blank"}
+
+## Support
+
++ If you need help getting started or run into any errors, please contact our team by email at [email protected].