BUG: Manual Labelling failing when loading data via GUI and corrupting stat.npy

[Vadenis]

Describe the issue:

The issue arises when trying to add manual labelling data. The plane has been registered and saved as a .mat the day before and I reload it in the GUI with File>Load processed data.

Then I enter the Manual Labelling GUI, add the ROI and extract them. When pressing Save and Quit, the GUI stays open and the following error appears in the Run GUI:

Saving old stat
Num cells 136
Saving new stat
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\drawroi.py", line 212, in <lambda>
    self.closeGUI.clicked.connect(lambda: self.close_GUI())
                                          ^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\drawroi.py", line 294, in close_GUI
    Fcell = np.concatenate((self.Fcell, self.parent.Fcell), axis=0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 194044 and the array at index 1 has size 245954
closing GUI

The only thing I can do is to close the Manual Labelling GUI with the red cross and click no to the prompt. The manual labels are then discarded and the stat.npy corrupted.

Trying to re-load that stat.npy another day, after manual labelling failure, the GUI errors as follow:

.../suite2pFOV1_Cellpo1_maxprojbyMeanImg/plane0/stat.npy
no channel 2 labels found (redcell.npy)
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\menus.py", line 25, in <lambda>
    loadProc.triggered.connect(lambda: io.load_dialog(parent))
                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\io.py", line 178, in load_dialog
    load_proc(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 327, in load_proc
    load_to_GUI(parent, basename, output)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 366, in load_to_GUI
    parent.stat[n]["snr"] = snr[n]
                            ~~~^^^
IndexError: index 268 is out of bounds for axis 0 with size 268

Trying to restore the stat.npy with the stat_orig,npy and reloading it in the GUI prompts the following error but the GUI doesn't crash:

.../suite2pFOV1_Cellpo1_maxprojbyMeanImg/plane0/stat.npy
no channel 2 labels found (redcell.npy)
time to draw and plot masks:  0.2659 sec
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\menus.py", line 25, in <lambda>
    loadProc.triggered.connect(lambda: io.load_dialog(parent))
                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\io.py", line 178, in load_dialog
    load_proc(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 327, in load_proc
    load_to_GUI(parent, basename, output)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 369, in load_to_GUI
    make_masks_and_enable_buttons(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 106, in make_masks_and_enable_buttons
    classgui.activate(parent, False)
  File "D:\Github\suite2p\suite2p\gui\classgui.py", line 235, in activate
    class_masks(parent)
  File "D:\Github\suite2p\suite2p\gui\classgui.py", line 364, in class_masks
    parent.colors['cols'][c] = col
    ~~~~~~~~~~~~~~~~~~~~~^^^
ValueError: could not broadcast input array from shape (404,3) into shape (268,3)

If I run the plane, register it, and do manual labelling within the same session, it works. I only encounter this if I save the data as a .mat and attempted manual labelling another day.

Reproduce the code example:

""" 1 - Lauch the GUI """
python -m suite2p

""" 2 - Do a run, allow overlap """
""" 3 - Save plane as a .mat """
""" 4 - Close Suite2p and env """
""" 5 - Re-start env and suite2P """
""" 6 - Load processed data """
""" 7 - Open manual labelling and extract manual ROI """
""" 8 - Click Save and Quit """

Example input dataset used to reproduce the issue:

No response

Error message:

Error 1:

Saving old stat
Num cells 136
Saving new stat
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\drawroi.py", line 212, in <lambda>
    self.closeGUI.clicked.connect(lambda: self.close_GUI())
                                          ^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\drawroi.py", line 294, in close_GUI
    Fcell = np.concatenate((self.Fcell, self.parent.Fcell), axis=0)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: all the input array dimensions except for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 194044 and the array at index 1 has size 245954
closing GUI

Error 2:

.../suite2pFOV1_Cellpo1_maxprojbyMeanImg/plane0/stat.npy
no channel 2 labels found (redcell.npy)
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\menus.py", line 25, in <lambda>
    loadProc.triggered.connect(lambda: io.load_dialog(parent))
                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\io.py", line 178, in load_dialog
    load_proc(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 327, in load_proc
    load_to_GUI(parent, basename, output)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 366, in load_to_GUI
    parent.stat[n]["snr"] = snr[n]
                            ~~~^^^
IndexError: index 268 is out of bounds for axis 0 with size 268

Error 3: 
.../suite2pFOV1_Cellpo1_maxprojbyMeanImg/plane0/stat.npy
no channel 2 labels found (redcell.npy)
time to draw and plot masks:  0.2659 sec
Traceback (most recent call last):
  File "D:\Github\suite2p\suite2p\gui\menus.py", line 25, in <lambda>
    loadProc.triggered.connect(lambda: io.load_dialog(parent))
                                       ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Github\suite2p\suite2p\gui\io.py", line 178, in load_dialog
    load_proc(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 327, in load_proc
    load_to_GUI(parent, basename, output)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 369, in load_to_GUI
    make_masks_and_enable_buttons(parent)
  File "D:\Github\suite2p\suite2p\gui\io.py", line 106, in make_masks_and_enable_buttons
    classgui.activate(parent, False)
  File "D:\Github\suite2p\suite2p\gui\classgui.py", line 235, in activate
    class_masks(parent)
  File "D:\Github\suite2p\suite2p\gui\classgui.py", line 364, in class_masks
    parent.colors['cols'][c] = col
    ~~~~~~~~~~~~~~~~~~~~~^^^
ValueError: could not broadcast input array from shape (404,3) into shape (268,3)

Version information:

suite2p v1.1.0
Windows 11

Context for the issue:

No response

[chriski777]

Hi @Vadenis, thanks for letting us know about this issue. I've tried retracing the steps you've provided on an example dataset and am unable to reproduce the error. Are you re-running only registration separately after running the full suite2p once? This error suggests that there is a difference in extraction output (self.Parent.Fcell loads from F.npy) and the registered binary (self.Fcell reads from the registered binary) have different shapes.

[Vadenis]

Hi @Vadenis, thanks for letting us know about this issue. I've tried retracing the steps you've provided on an example dataset and am unable to reproduce the error. Are you re-running only registration separately after running the full suite2p once? This error suggests that there is a difference in extraction output (self.Parent.Fcell loads from F.npy) and the registered binary (self.Fcell reads from the registered binary) have different shapes.

Hi @chriski777 , I've re-run another sub-sample of the data in the meantime with the same steps and couldn't reproduce the errors either. I'm currently re-running the full data set that triggered the errors in the first place. I will let you know if they arise again and update the Issue accordingly.

To answer your question, no. I was running the full suite2p once, would do a bit of curation (cell / not cell and merge appending with the GUI) and save as a .mat. I would close everything and reload the data the next day to do the manual labelling which would end up with Error 1. Errors 2 and 3 would arise only if I try to re-load the data once more after encountering Error1.