Merge pull request #1008 from MouseLand/dev

Addresses doc issues and package issues

Author: chriski777

README.md

@@ -52,7 +52,7 @@ Pachitariu, M., Stringer, C., Schröder, S., Dipoppa, M., Rossi, L. F., Carandin
 ### <a name="installation_section"></a> Installation for Linux, Windows, and MacOS (intel processors) machines
 1. Install an [Anaconda](https://www.anaconda.com/download/) distribution of Python -- Choose **Python 3.8** and your operating system. Note you might need to use an anaconda prompt if you did not add anaconda to the path.
 2. Open an anaconda prompt / command prompt with `conda` for **python 3** in the path
-3. Create a new environment with `conda create --name suite2p python=3.8`.
+3. Create a new environment with `conda create --name suite2p python=3.9`.
 4. To activate this new environment, run `conda activate suite2p`
 5. (Option 1) You can install the minimal version of suite2p, run `python -m pip install suite2p`.  
 6. (Option 2) You can install the GUI version with `python -m pip install suite2p[gui]`. If you're on a zsh server, you may need to use `' '` around the suite2p[gui] call: `python -m pip install 'suite2p[gui]'`. This also installs the NWB dependencies.
@@ -76,7 +76,7 @@ pip install --upgrade suite2p
 1. Download an iTerm2 terminal from this [link](https://iterm2.com/). Install it into your /Applications folder. If you already have downloaded iTerm, duplicate it and give it whatever name you'd like (e.g., "iterm2Rosetta"). 
 2. Navigate to the iTerm app you will use, right click it, and then select "Get Info". Check "Open using Rosetta". 
 3. Open up this iTerm app and follow steps 1 & 2 in the installation section [above](#installation_section) to install anaconda.
-4. Use the following command `CONDA_SUBDIR=osx-64 conda create --name suite2p python=3.8`
+4. Use the following command `CONDA_SUBDIR=osx-64 conda create --name suite2p python=3.9`
 5. Follow steps 4-7 in the installation section [above](#installation_section) to install the `suite2p` package. 
 
 
@@ -102,7 +102,7 @@ The software has been heavily tested on Windows 10 and Ubuntu 18.04, and less we
 1. Clone the repository and `cd suite2p` in an anaconda prompt / command prompt with `conda` for **python 3** in the path
 2. Run `conda env create --name suite2p`
 3. To activate this new environment, run `conda activate suite2p` (you will have to activate every time you want to run suite2p)
-4. Install the local version of suite2p into this environment in develop mode with the command `pip install -e .`
+4. Install the local version of suite2p into this environment in develop mode with the command `pip install -e .[all]`
 5. Run tests: `python setup.py test` or `pytest -vs`, this will automatically download the test data into your `suite2p` folder. The test data is split into two parts: test inputs and expected test outputs which will be downloaded in `data/test_inputs` and `data/test_outputs` respectively. The .zip files for these two parts can be downloaded from these links: [test_inputs](https://www.suite2p.org/static/test_data/test_inputs.zip) and [test_outputs](https://www.suite2p.org/static/test_data/test_outputs.zip).
 
 ## Examples

docs/inputs.rst

@@ -156,21 +156,21 @@ Suite2p reads nd2 files using the nd2 package and returns a numpy array represen
 
 
 
-BinaryRWFile
+BinaryFile
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-The ``BinaryRWFile`` is a special class in suite2p that is used to read/write imaging data and acts like a Numpy Array. Inputs of any format listed above will be converted into a ``BinaryRWFile`` before being passed in through the suite2p pipeline. An input file can easily be changed to a ``BinaryRWFile`` in the following way: 
+The ``BinaryFile`` is a special class in suite2p that is used to read/write imaging data and acts like a Numpy Array. Inputs of any format listed above will be converted into a ``BinaryFile`` before being passed in through the suite2p pipeline. An input file can easily be changed to a ``BinaryFile`` in the following way: 
 
 ::
 
    import suite2p
 
    fname = "gt1.tif" # Let's say input is of shape (4200, 325, 556)
    Lx, Ly = 556, 326 # Lx and Ly are the x and y dimensions of the imaging input
-   # Read in our input tif and convert it to a BinaryRWFile
-   f_input = suite2p.io.BinaryRWFile(Ly=Ly, Lx=Lx, filename=fname)
+   # Read in our input tif and convert it to a BinaryFile
+   f_input = suite2p.io.BinaryFile(Ly=Ly, Lx=Lx, filename=fname)
 
-``BinaryRWFile`` can work with any of the input formats above. For instance, if you'd like to convert an input binary file, you can do the following:
+``BinaryFile`` can work with any of the input formats above. For instance, if you'd like to convert an input binary file, you can do the following:
 
 ::
 

docs/settings.rst

@@ -77,7 +77,7 @@ Suite2p can accomodate many different file formats. Refer to this
 
 - **nwb_series** (*str, default: ''*) Name of TwoPhotonSeries values you wish to retrieve from your NWB file. 
 
-- **save_path0** (*list[str], default: empty list*) List containing pathname of where you'd like to save your pipeline results. If list is empty, the first element of ``ops['data_path']`` is used. 
+- **save_path0** (*str, default: ''*) String containing pathname of where you'd like to save your pipeline results. If no pathname is provided, the first element of ``ops['data_path']`` is used. 
 
 - **save_folder** (*list[str], default: empty list*) List containing directory name you'd like results to be saved under. Defaults to ``"suite2p"``. 
 
@@ -247,7 +247,7 @@ ROI detection settings
   1.0.
 
 - **high_pass**: (*int, default: 100*) running mean subtraction across
-  time with window of size 'high_pass'. Values of less than 10 are
+  bins of frames with window of size 'high_pass'. Values of less than 10 are
   recommended for 1P data where there are often large full-field
   changes in brightness.
 
@@ -355,4 +355,4 @@ Channel 2 specific settings
 Miscellaneous settings
 ~~~~~~~~~~~~~~~~~~~~~~
 
-- **suite2p_version**: specifies version of suite2p pipeline that was run with these settings. Changing this parameter will NOT change the version of suite2p used. 
+- **suite2p_version**: specifies version of suite2p pipeline that was run with these settings. Changing this parameter will NOT change the version of suite2p used. 

scripts/generate_test_data.py

@@ -70,7 +70,7 @@ def generate_detection_1plane1chan_test_data(ops):
 			[[Path(ops['data_path'][0]).joinpath('detection/pre_registered.npy')]],
 			(404, 360)
 		)
-		with suite2p.io.BinaryRWFile(Ly = ops[0]['Ly'], Lx = ops[0]['Lx'], filename=ops[0]['reg_file']) as f_reg:
+		with suite2p.io.BinaryFile(Ly = ops[0]['Ly'], Lx = ops[0]['Lx'], filename=ops[0]['reg_file']) as f_reg:
 			ops, stat = suite2p.detection.detection_wrapper(f_reg, ops=ops[0])
 			ops['neuropil_extract'] = True
 			cell_masks, neuropil_masks = masks.create_masks(stat, ops['Ly'], ops['Lx'], ops=ops)
@@ -107,7 +107,7 @@ def generate_detection_2plane2chan_test_data(ops):
 		two_plane_ops[1]['meanImg_chan2'] = np.load(detection_dir.joinpath('meanImg_chan2p1.npy'))
 		for i in range(len(two_plane_ops)):
 			op = two_plane_ops[i]
-			with suite2p.io.BinaryRWFile(Ly = op['Ly'], Lx = op['Lx'], filename=op['reg_file']) as f_reg:
+			with suite2p.io.BinaryFile(Ly = op['Ly'], Lx = op['Lx'], filename=op['reg_file']) as f_reg:
 				# Neuropil_masks are later needed for extraction test data step
 				op['neuropil_extract'] = True
 				op, stat = suite2p.detection.detection_wrapper(f_reg, ops=op)

setup.py

@@ -36,8 +36,9 @@
       "pytest",
       "tenacity",
       "tqdm",
+      "pynwb>=2.3.2", #this is needed as test_io contains a test with nwb
       "pytest-qt>3.3.0",
-    ]
+]
 
 all_deps = gui_deps + nwb_deps + test_deps + io_deps
 

suite2p/default_ops.py

@@ -25,7 +25,7 @@ def default_ops():
         "nwb_driver": "",  # driver for nwb file (nothing if file is local)
         "nwb_series":
             "",  # TwoPhotonSeries name, defaults to first TwoPhotonSeries in nwb file
-        "save_path0": [],  # stores results, defaults to first item in data_path
+        "save_path0": '',  # pathname where you'd like to store results, defaults to first item in data_path
         "save_folder": [],  # directory you"d like suite2p results to be saved to
         "subfolders": [
         ],  # subfolders you"d like to search through when look_one_level_down is set to True
@@ -120,7 +120,7 @@ def default_ops():
         "max_overlap":
             0.75,  # cells with more overlap than this get removed during triage, before refinement
         "high_pass":
-            100,  # running mean subtraction with window of size "high_pass" (use low values for 1P)
+            100,  # running mean subtraction across bins with a window of size "high_pass" (use low values for 1P)
         "spatial_hp_detect":
             25,  # window for spatial high-pass filtering for neuropil subtraction before detection
         "denoise": False,  # denoise binned movie for cell detection in sparse_mode

suite2p/gui/visualize.py

@@ -10,7 +10,7 @@
 from PyQt5.QtWidgets import QStyle
 from PyQt5.QtWidgets import QWidget, QSlider, QMainWindow, QGridLayout, QStyleOptionSlider, QApplication, QLabel, QLineEdit, QPushButton, QComboBox, QCheckBox
 from matplotlib import cm
-from rastermap.mapping import Rastermap
+from rastermap.rastermap import Rastermap
 from scipy.ndimage import gaussian_filter1d
 from scipy.stats import zscore
 

suite2p/io/binary.py

@@ -3,6 +3,8 @@
 """
 from typing import Optional, Tuple, Sequence
 from contextlib import contextmanager
+from tifffile import TiffWriter
+
 import os
 
 import numpy as np
@@ -191,6 +193,24 @@ def bin_movie(self, bin_size: int, x_range: Optional[Tuple[int, int]] = None,
         mov = np.stack(batches)
         return mov
 
+    def write_tiff(self, fname, range_dict={}):
+        "Writes BinaryFile's contents using selected ranges from range_dict into a tiff file."
+        n_frames, Ly, Lx = self.shape
+        frame_range, y_range, x_range = (0,n_frames), (0, Ly), (0, Lx)
+        with TiffWriter(fname, bigtiff=True) as f:
+            # Iterate through current data and write each frame to a tiff
+            # All ranges should be Tuples(int,int)
+            if 'frame_range' in range_dict:
+                frame_range = range_dict['frame_range']
+            if 'x_range' in range_dict:
+                x_range = range_dict['x_range']
+            if 'y_range' in range_dict:
+                y_range = range_dict['y_range']
+            print('Frame Range: {}, y_range: {}, x_range{}'.format(frame_range, y_range, x_range))
+            for i in range(frame_range[0], frame_range[1]):
+                curr_frame = np.floor(self.file[i, y_range[0]:y_range[1], x_range[0]:x_range[1]]).astype(np.int16)
+                f.write(curr_frame)
+        print('Tiff has been saved to {}'.format(fname))
 
 def from_slice(s: slice) -> Optional[np.ndarray]:
     """Creates an np.arange() array from a Python slice object.  Helps provide numpy-like slicing interfaces."""

suite2p/io/tiff.py

@@ -60,7 +60,7 @@ def generate_tiff_filename(functional_chan: int, align_by_chan: int, save_path:
             wchan = 0
     if not os.path.isdir(tifroot):
         os.makedirs(tifroot)
-    fname = "file%0.3d_chan%d.tif" % (k, wchan)
+    fname = "file00%0.3d_chan%d.tif" % (k, wchan)
     fname = os.path.join(tifroot, fname)
     return fname
 

tests/instructions.md

@@ -0,0 +1,4 @@
+# Instructions on generating new test data
+1. Make sure you are in the `suite2p/scripts` directory instead of `suite2p/tests` where this `instructions.md` file is located. Run `python generate_test_data.py`. 
+2. All the generated test data will be placed in the directory 
+`suite2p/scripts/test_data`. These directories will correspond to the expected outputs for our tests.