You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/source/Data_processing/Overview.rst
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ Overview
4
4
``snazzy_processing`` is a Python package to automate the extraction of primary data: Ventral Nerve Cord (VNC) length and signal intensity from fluorescence imaging of Drosophila embryos.
5
5
The package is an image processing pipeline, that can be divided intro three main stages:
6
6
7
-
* Crop movies for individual specimens from raw data
7
+
* Crop raw data containing many samples into smaller movies (one per sample)
8
8
* Calculate signal intensity inside ROIs
9
9
* Measure the VNC length
10
10
@@ -16,16 +16,16 @@ This means that you can feed either ``.tif`` or ``.nd2`` files into the pipeline
16
16
If your raw data is in another format, you must first convert it to ``.tif``.
17
17
ImageJ for example provides several plugins to convert files to tif, including the excellent `BioFormats extension <https://imagej.net/formats/bio-formats>`__.
18
18
19
-
Before running the pipeline, which is the last cell of the jupyter notebook, we must determine from where to crop each movie in the raw data.
19
+
Before running the pipeline, which is the last cell of the jupyter notebook, we must determine where to crop the raw data to separate samples.
20
20
21
-
The bounding boxes for each individual sample are determined via thresholding.
22
-
Because the image histogramsmight vary, some manual adjustment might be necessary.
21
+
Thresholding is used to distinguish each sample from its background and create bounding boxes.
22
+
Because thresholding depends on image histograms, and these histograms might vary between movies, some manual adjustment may be necessary.
23
23
Inspect where the bounding boxes will be created in the jupyter notebook.
24
24
They should cover the entire sample.
25
-
If there is a bounding box that covers more than one specimen, because maybe they were touching each other, those specimens must be ignored or further manually processed.
26
-
We can't use it directly because the resulting ROI will not match a single sample and the length and activity data will be wrong.
25
+
If there is a bounding box that covers more than one specimen (e.g, samples are touching each other), those specimens must be ignored or further manually processed.
26
+
Boxes that contain several samples cannot be used directly because the resulting ROI will not match a single sample and the length & activity data will be wrong.
27
27
28
-
After the bounding boxes are determined, you can run the pipeline.
28
+
After the bounding boxes are determined, you can select which embryos to include in the analysis and run the pipeline.
29
29
By default, the methods in the pipeline will not overwrite any data.
30
30
If data for a given sample is found in the output directory, it will simply skip that sample.
31
31
If you want to recalculate any data, first remove or rename the current files.
Copy file name to clipboardExpand all lines: docs/source/Data_processing/Process_raw_data.rst
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,12 @@
1
1
Process raw images
2
2
==================
3
3
4
-
Since the imaging is done with a large Field of View microscope, usually during 6 hours or more, the raw images tend to be in the range of 50 ~ 200 GiB.
5
-
The simplest way to handle the raw data is to crop it in individual movies.
6
-
There is a considerable amount of background pixels that can be ignored in the raw data. After cropping the embryos, all individual movies combined take about 40% of the original memory space.
7
-
This already saves considerable ROM memory but most importantly, it means we can easily load individual movies in the RAM of a regular computer (16-32 GB RAM), without needing to use memory mapped files.
4
+
Since the imaging is performed with a large field of view microscope and typically time-lapsed for 6 hours or more, the raw images tend to be in the range of 50 ~ 200 GiB.
5
+
The simplest way to handle the raw data is to first crop it into individual movies for each sample, as there is a considerable amount of background pixels that can be ignored in the raw data.
6
+
After cropping the embryos, all individual movies combined take about 40% of the original memory space.
7
+
This already saves considerable ROM memory but, most importantly, it means individual movies can be easily loaded in the RAM of a regular computer (16-32 GB RAM), without needing to use memory mapped files.
8
8
9
-
The algorithm to process the raw image can be resumed as:
9
+
The algorithm to process the raw image can be summarized as:
10
10
11
11
1. Get the maximum projection of each pixel for the first 10 frames
12
12
2. Automatic threshold (Triangle method)
@@ -19,11 +19,11 @@ The algorithm to process the raw image can be resumed as:
19
19
To calculate the bounding boxes of each embryo, we first take the maximum projection of each pixel for the first 10 frames, and then use the Triangle threshold method to binarize the image.
20
20
The Triangle threshold is a good choice here because the image has a lot of background pixels, resulting in an unimodal histogram that is centered around the background pixels average value.
21
21
22
-
Once we have the binary image, we traverse it to identify each embryo.
23
-
Whenever a foreground pixel is found, we mark all connecting foreground pixels, and also keep track ofthe amount of pixels marked and the extreme points (minimum and maximum coordinates in both dimensions).
24
-
The pixel count is used to determine if the marked area really corresponds to an embryo, or just a smaller artifact that was erroneously considered a foreground.
22
+
Once the binary image is created, it is traversed to identify each embryo.
23
+
Whenever a foreground pixel is found, we mark all connecting foreground pixels and keep track of: (1) the amount of pixels marked and (2) the extreme points (minimum and maximum coordinates in both dimensions).
24
+
The pixel count is used to determine if the marked area accurately corresponds to an embryo, or just a smaller artifact that was erroneously considered a foreground.
25
25
The minimum pixel count might change depending on the type of sample being processed, and can be adjusted in ``slice_img.get_bbox_boundaries``.
26
-
Regions with high signal intensity, for example corresponding to fly embryo's eyes or gut are examples of smaller artifacts that sometimes are included in the binary image, but can easily be removed due to its size.
26
+
Regions of high signal intensity that represent small artifacts (e.g., fly embryo eyes or gut) can be included in the binary image but easily distingushed from the embryo due to their size.
27
27
The extreme points are then used to generate the bounding boxes, which will determine the positions where the image will be cropped.
28
28
29
29
The raw image is opened as a memory map using ``numpy``, and the individual embryos are cropped and saved as tif files.
Copy file name to clipboardExpand all lines: docs/source/Data_processing/ROIs_and_signal_intensity.rst
+9-8Lines changed: 9 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,28 +1,29 @@
1
1
ROIs and signal intensity
2
2
=========================
3
3
4
-
When running the pipeline from the jupyter notebook, the ROIs are calculated for each frame.
5
-
The processing can be sped up by calculating a single ROI for groups of 5 or 10 frames, based on the fact that the sample signal won't change considerably within this interval.
6
-
This is a good approximation and useful for quick analyses, at the cost of the eventual errors in readings caused by movement (see ``activity.ipynb`` for details about the error in activity caused by downsampling).
4
+
When running the pipeline from the jupyter notebook, a single ROI is calculated for each frame.
5
+
The processing can be sped up by calculating a single ROI for groups of 5 or 10 frames, instead of a single ROI per frame.
6
+
This is possible if the sample signal doesn't change considerably within this interval.
7
+
Therefore, calculating one ROI per group of frames is a good approximation, and it can be useful for quick analyses, at the cost of the eventual errors in readings caused by movement (see ``activity.ipynb`` for details about the error in activity caused by downsampling).
7
8
8
-
The ROI algorithm can be resumed as:
9
+
The ROI algorithm can be summarized as:
9
10
10
11
1. Average the group of frames into a single 2D matrix
11
12
2. Automatic threshold (Otsu's method)
12
13
3. Binarize the image
13
-
4. Remove small holes inside the VNC
14
+
4. Seal small holes inside the VNC
14
15
5. Select the largest group of connected foreground pixels
15
16
6. Return a mask that matches the largest label
16
17
17
18
For some datasets with lower VNC signal, a higher threshold value tends to provide better results.
18
19
To change the threshold method, change the ``threshold_method`` parameter in the ``pipeline.measure_vnc_length`` function to ``otsu``.
19
20
20
21
Even though the signal from pixels inside the VNC is stable, the selected threshold value is not always perfect.
21
-
After binarizing the image, the VNC might contain small regions that were lower than the calculated threshold.
22
+
After binarizing the image, the VNC might contain small regions that were lower than the calculated threshold (i.e., holes).
22
23
These regions are merged back into the VNC if they are completely contained inside the VNC binary component.
23
24
24
-
To calculate the signal intensity, we apply the mask to the embryo and calculate the mean pixel value.
25
-
The dynamic and structural channel measurements are exported as a ``.csv`` file and further processed using the code from ``snazzy_analysis``.
25
+
To calculate the signal intensity, a mask is applied to the embryo and the mean pixel value is calculated.
26
+
The dynamic and structural channel measurements are exported as a ``.csv`` file and further processed using ``snazzy_analysis``.
0 commit comments