Merge pull request #22 from ACRLab/dataset

Paper prep

Author: cdpaiva

.github/workflows/draft-pdf.yaml

@@ -0,0 +1,24 @@
+name: Draft PDF
+on:
+  push:
+    paths:
+      - paper/**
+      - .github/workflows/draft-pdf.yaml
+
+jobs:
+  paper:
+    runs-on: ubuntu-latest
+    name: Paper Draft
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Build draft PDF
+        uses: openjournals/openjournals-draft-action@master
+        with:
+          journal: joss
+          paper-path: paper/paper.md
+      - name: Upload
+        uses: actions/upload-artifact@v4
+        with:
+          name: paper
+          path: paper/paper.pdf

.gitignore

@@ -1,5 +1,13 @@
 # dir with input files
-*/data/
+snazzy_processing/data/*
+snazzy_analysis/data/*
+
+# add annotated data for ROI length evaluation
+!snazzy_processing/data/20240611/
+snazzy_processing/data/20240611/*
+!snazzy_processing/data/20240611/annotated
+!snazzy_processing/data/20240611/emb_sizes
+
 
 # dir with output files
 */results/

README.md

@@ -1,7 +1,12 @@
 # SNAzzy: an image processing pipeline for investigating global Synchronous Network Activity
 
+SNAzzy is a Python package for studying synchronous network activity (SNA) in Drosophia embryos via high-thoughput microscopy.
+The software includes processing raw data into individual `.tif` files, quantification of fluorescence and changes in morphology, a custom peak detection algorithm, and a GUI for data visualization and curation.
+
 ## Getting Started
 
+Refer to the README files inside the `snazzy_processing` or `snazzy_analysis` packages for details on running the code.
+
 ### Installation
 
 The project uses [conda](https://docs.conda.io) to manage dependencies.
@@ -19,6 +24,18 @@ Activate the environment:
 ```
     conda activate snazzy-env
 ```
+
+## Testing
+
+Tests can be run with pytest.
+
+You can run the test suite from the project’s root directory to test everything at once.
+Make sure the environment is active, and then run:
+
+```
+    pytest
+```
+
 ## Contributing
 
 Thank you for being interested in `snazzy`!

docs/source/Data_analysis/Graphical_User_Interface.rst

@@ -98,9 +98,13 @@ From this window it's possible to set:
 * To_remove: embryo numbers that will be analyzed, but will appear in the 'Removed' group.
 * Embryos that have it's first peak before the first peak threshold or that were marked by the user as removed will also be at the to_remove category.
 * Has_transients: if selected the code will try to identify and skip the first peak if it's likely just a transient.
-* Has_dsna: if selected the code will try to determine dSNA and ignore all peaks that happen after dSNA start.
 * Dff_strategy: Combo box with the baseline strategy methods. ``local_minima`` will pick the bottom 11 points out of the ``baseline_window_size`` and use that average as the baseline. ``baseline`` will split the DFF values into bins and use the average of the most frequent bin as the baseline.  This method assumes that the bursts of activity are sparse, so that for all windows the most frequent bin falls into the baseline values.
 
+The embryos listed in ``to_remove`` are not used for plotting and comparisons between Groups.
+There are different reasons for marking an embryo as removed.
+One example would be to remove embryos that already are in later stages of development when the imaging session starts. 
+Another would be to remove unhealthy embryos, or embryos that don't hatch, if that is a requirement for the experiment.
+
 Inside the File menu there is an option to open the ``json`` file and change any of its parameters.
 Updating the file causes the entire Dataset to be recreated with the new configuration data.
 

docs/source/Data_analysis/Hatching_Point.rst

@@ -18,7 +18,7 @@ It can also happen that an embryo is still inside the egg and a larva that alrea
 It's very rare that this scenario affects the ROI calculation, because we only consider the largest connected area for calculating the ROI, which is usually the VNC.
 If it does, then the only option is to remove that embryo.
 
-When loading an experiment for the first time, it's worth it to visualize the structural channel signal of each embryo.
+When loading a dataset for the first time, it's worth it to visualize the structural channel signal of each embryo.
 On a few occasions, mostly due to very abrupt motion, the ROI is underestimated and the hatching point is determined earlier.
 In these cases, you can drag the line that indicates the hatching to a more accurate position or remove that embryo.
 

docs/source/Data_analysis/Peak_Detection.rst

@@ -39,6 +39,9 @@ Since the leftmost peak can have an amplitude very different than the local maxi
 3. Filter peaks by local threshold
 ----------------------------------
 
+This step is **not enabled** by default.
+To enable it the ``dff_peak_indices`` in ``trace.calculate_peaks`` must be passed to ``trace.filter_peaks_by_local_threshold``.
+
 As the embryos develop, there is a global trend of peak amplitude to rise and then stabilize before hatching.
 We use this fact to perform an extra validation step for the calculated peaks.
 Each peak is compared against its neighboring peaks, and peaks that are too high or too low are discarted.

…cs/source/Data_processing/ROI_length.rst …ssing/Neurodevelopmental_progression.rstdocs/source/Data_processing/ROI_length.rst renamed to docs/source/Data_processing/Neurodevelopmental_progression.rst docs/source/Data_processing/ROI_length.rst renamed to docs/source/Data_processing/Neurodevelopmental_progression.rst

@@ -1,15 +1,14 @@
-Neurodevelopmental Time
-----------------
-Together the ROI length and the full embryo size are used as a proxy to measure the embryonic neurodevelopmental progression:
-
-developmental_progression = embryo_length / ROI_length
-
-ROI length
-==========
+Neurodevelopmental Progression
+==============================
 
 The ROI length is calculated by center line estimation.
 The general approach is to measure the line that will pass through the center of the ROI; this will correspond to the ventral nerve cord length.
 
+Together the ROI length and the full embryo size are used as a proxy to measure the embryonic neurodevelopmental progression:
+
+.. math::
+    neurodevelopmental\ progression = \frac{embryo\ length}{ROI\ length}
+
 To determine the ROI length, the following steps are used:
 
 1. Binarize the image
@@ -26,6 +25,7 @@ Embryo Full size
 The full specimen size is calculated by approximating the entire sample shape as an ellipse, and measuring this ellipse's diameter.
 
 The steps to calculate the embryo's size are:
+
 1. Equalize the image histogram
 2. Automatic threshold (Triangle method)
 3. Binarize the image

docs/source/Data_processing/Overview.rst

@@ -34,4 +34,4 @@ Refer to the other documentation pages for a description of the pipeline steps:
 
 * `Process raw data <Process_raw_data.html>`__
 * `ROI and signal intensity <ROIs_and_signal_intensity.html>`__
-* `ROI length <ROI_length.html>`__
+* `Neurodevelopmental_progression <Neurodevelopmental_progression.html>`__

docs/source/Data_processing/ROIs_and_signal_intensity.rst

@@ -36,5 +36,5 @@ To display it, ``cd`` into the ``snazzy_processing`` directory, and run the file
 
     python3 scripts/plot_contours.py
 
-It will look for any experiment directories you have inside the ``./data`` directory and present the available options in the terminal.
+It will look for any dataset directories you have inside the ``./data`` directory and present the available options in the terminal.
 Animations can be paused by pressing any key.

docs/source/Data_processing/index.rst

@@ -8,4 +8,4 @@ Data Processing
    Overview
    Process_raw_data
    ROIs_and_signal_intensity
-   ROI_length
+   Neurodevelopmental_progression