Spikesorting issue - IncrementalPCA instance is not fitted yet

[sarahthon]

Hi again, I have also run into another problem with spikesorting, which do not occur with all actions but a few. Is this relatable to the resolved issue #82

---

Processing 022-200322-6

Cleaning up existing NWB file

Preprocessing recording:
Num channels: 32
Duration: 927.56 s
Detected bad channels: ['CH1' 'CH4' 'CH25' 'CH26' 'CH27' 'CH29' 'CH32']
Active channels: 25
Saving preprocessed recording

Spike sorting with mountainsort4 using installed sorter
Found 46 units!
Removed 0 units with less than 3 spikes

Postprocessing
Extracting waveforms
Computing QC metrics
Exporting to phy
ERROR: unable to process 022-200322-6
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/process.py", line 261, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/process.py", line 210, in _process_chunk
return [fn(*args) for args in chunk]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/process.py", line 210, in
return [fn(*args) for args in chunk]
^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/spikeinterface/core/job_tools.py", line 439, in function_wrapper
return _func(segment_index, start_frame, end_frame, _worker_ctx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/spikeinterface/postprocessing/principal_component.py", line 661, in _all_pc_extractor_chunk
all_pcs[i, :, c] = pca_model[chan_ind].transform(w)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 295, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/sklearn/decomposition/_incremental_pca.py", line 409, in transform
return super().transform(X)
^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/sklearn/utils/_set_output.py", line 295, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/sklearn/decomposition/_base.py", line 143, in transform
check_is_fitted(self)
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/sklearn/utils/validation.py", line 1622, in check_is_fitted
raise NotFittedError(msg % {"name": type(estimator).name})
sklearn.exceptions.NotFittedError: This IncrementalPCA instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/fp/projects01/ec109/software/expipe-plugin-cinpla/src/expipe_plugin_cinpla/widgets/process.py", line 300, in on_run
process.process_ecephys(
File "/fp/projects01/ec109/software/expipe-plugin-cinpla/src/expipe_plugin_cinpla/scripts/process.py", line 281, in process_ecephys
sexp.export_to_phy(
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/spikeinterface/exporters/to_phy.py", line 245, in export_to_phy
pc.run_for_all_spikes(output_folder / "pc_features.npy", **job_kwargs)
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/spikeinterface/postprocessing/principal_component.py", line 373, in run_for_all_spikes
processor.run()
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/site-packages/spikeinterface/core/job_tools.py", line 401, in run
for res in results:
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/process.py", line 620, in _chain_from_iterable_of_lists
for element in iterable:
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/projects/ec109/conda-envs/cinpla/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
sklearn.exceptions.NotFittedError: This IncrementalPCA instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

[sarahthon]

I adjusted the autothreshold level to 10 to include all channels, and now it works. But I don't think including more noise in a few recordings is a good solution

[nicolossus]

@alejoe91 Are channel IDs counted starting from 0 or 1? If they start from 1, this recording has three bad channels on one tetrode (CH25, CH26, CH27). Would having just one functional channel pose issues for the PCA? Additionally, does the sparse=True keyword argument matter in this context?

[alejoe91]

Hi @nicolossus

Yeah they should start from 0, but you can double check from the settings.xml. If only one channel is left after bad channel removal, the first issue will be spike sorting, since it will be tricky to isolate units.

In another comment (that I'm not finding) you asked about switching to 0.101.1. This would be highly recommended from my side and it shouldn't take too long. I believe this specific issue is also solved with additional protections :)
If you have time to spearhead this, I can help out in case of need or with a couple of meetings when needed!

The main change is in the postprocessing module, and you can find a guide to upgrade here.

[nicolossus]

@alejoe91 Thanks! I have some time the coming weeks, so I can spearhead the transition. I'll let you know if there are problems :)