FEATURE: <Use spike templates from one sorting run on all subsequent runs>

[mikemanookin]

Feature you'd like to see:

Our data is typically collected over >10 hours of recording, meaning that we need to separate spike sorting for the same tissue/cells across several sorting chunks. This makes it difficult to find the same cells across different sorting chunks/runs. Ideally, I would like to sort one chunk and then use the spike templates, etc from that sorting run on all of the subsequent chunks. This would save us a lot of extra work. My understanding is that currently the only way to use custom templates is to replace the wTEMP.npz file in the git repository. It would be ideal to be able to point to a directory/file containing the first run (e.g., by passing an argument to spike_detect.get_waves -> template_path). Related, it would be helpful to have some documentation about how this is done and the required format for the wTEMP.npz file. Perhaps I missed it. Thank you!

Additional Context

No response

[rdarie]

I'd be interested in something like this as well, for a different use case. I am working with recordings that have a lot of electrical stimulation artifact, which despite my best efforts to remove via preprocessing, still seem to cause problems during sorting. Ideally, I'd like to be able to identify templates based on snippets of the recording where I know there are no artifacts, and then apply them to the entire dataset.

I'm curious, @jacobpennington and @marius10p, if you think this approach would be feasible. If I understand the algorithm right, the templates are used to detect candidate spike waveforms, which are then passed to the clustering algorithm for the final cluster assignment. Are the templates themselves used as an input to the clustering algorithm? Or, more to the point, is the algorithm deterministic in that candidate waveforms from different files, identified with the same templates, would produce equivalent clusters?

[jacobpennington]

I will talk with Marius about the feasibility of this in geneneral. As to your other questions @rdarie, the clustering algorithm is not deterministic and the templates are not used as an input for clustering. They are only used for detecting spikes.

[alexandrath]

@jacobpennington , @marius10p - this feature (re-using templates for sorting) is something we've also been talking about for years now (across several labs) and it would be absolutely invaluable. Sometimes I find a cell using kilosort in some chunks of data but not others that are in-between, so we know the cell is alive and responding... sometimes joint chunk sorting, or using different spike sorters helps but it takes a lot of time and sometimes fails altogether. Some of these cells come from extremely precious, hard-to-obtain human tissue, so having the option to re-use templates would immediately boost the value we can extract from these recordings. Please let me know if it can happen.

[mikemanookin]

@jacobpennington , @marius10p I agree with alexandrath's comment. This would be a very valuable tool for my lab and I know of several other labs that would be very grateful for this feature.

[jacobpennington]

For all of you, can you please provide more information about the situations you think this would be useful for, and clarify if you're referring to the universal templates (these are in wPCA.npz, or extracted from data by default if templates_from_data = True), learned templates (built by clustering the spikes detected by universal templates), or the final clusters (referred to as templates in Phy). Screenshots from Phy or other software showing examples of what you're referring to would be helpful.

@mikemanookin are there specific problems with your results that lead you to believe this would work better than generating the templates from your data (i.e. setting templates_from_data = True)? This is already done automatically by default.

@rdarie it's unlikely that re-using templates this way will help sort data with a lot of artifacts in it. Those artifacts will still interfere with spike detection and cluster assignment.

[mikemanookin]

@jacobpennington Hi Jacob. We would ideally use the final clusters to sort the other blocks of data from the same experiment.

[jacobpennington]

@mikemanookin Unfortunately that would not work without substantial changes to the algorithm. Unlike past versions of Kilosort, the templates used for spike detection and the final clusters that are built from detected spikes are completely separate. In short, you would have to run the full pipeline again anyway.

[mikemanookin]

@jacobpennington Would it be possible to use the learned templates instead (as opposed to the final templates)?

[jacobpennington]

@mikemanookin In principle yes, but in practice it's not as simple as just re-using the templates. Using those templates (and running the subsequent steps) relies on a lot of other variables being computed by earlier steps in the pipeline. You would still need to re-run the preprocessing steps (including drift correction) for each recording, and it's not clear how well the templates' positions would translate if there's a lot of drift.