Drift correction makes things even worse

[ZhongRenNeuroscience]

Describe the issue:

Hi,

I'm using kilosort4 GUI to sort Neuropixel2.0 recordings. The session can last for 1 to 2 hours. These days I'm sorting some data probably with big drift because I do see some flat lines at certain period of edge channels, and clusters always break at a consistant moment. However, I'm thinking whether KS4 overestimates the drift. For example this session, the drift correction algorithm moves in general 4 channels up. But I can always find the matched cluster 10-12 channels above. Which means, the drift correction makes a wrong estimation. Even the direction is totally wrong. It should correct the drift by moving 6-8 channels down but not 4 channels up. It makes the manual curation more difficult. Could you help me to check what is the reason and how to fix it? The majority of the sortings are still good, but this is not the only session with this issue as well. Thank you!

kilosort4.log

Reproduce the bug:

Error message:

Version information:

Python3.11
KS 4.0.11

[ZhongRenNeuroscience]

Sorry, I don't know whether this screenshot can be better.

[jacobpennington]

The recording looks pretty unstable, especially after the first ~3000s. The data after that point is not likely to be sorted reliably. I would recommend you try sorting just the first portion of the recording (tmax = 3000). You should also try setting nblocks = 1 and batch_size = 30000. That will make the drift estimate a bit less noisy, and using a smaller batch size will help track the fast changes more accurately.

[ZhongRenNeuroscience]

Thanks for your reply. I ran the same but still the entire session with the parameters you suggested. It looks better. But there is still a big and systematic mismatch at time point 5225s (but not 5000s last time).

Regarding the unstable recording quality, I agree that there can be drifts. But I don't think the recording quality is as bad as KS4 reported. There shouldn't be that many sudden and big jumps. If I understand it correctly, KS4 thinks there is a big drift and reports it in those figures, also makes correction based on this drift correction results. But my point is KS4 may detect the drift incorrectly, and then makes a wrong drift correction. I would like to ask, is there any way to limit the function of drift detection and correction? In this case, for example, if the 'detected' drifts happen quite frequently and always jump to 10 channels away, then KS4 should question about this drift detection result, even don't make the correction.

Thank you!

[jacobpennington]

Correct, KS4 is reported huge drift in the latter portion of the recording. I am not claiming that there is in fact that much drift in the recording. The issue is that something is happening after ~3000s that is causing most/all units to drop in and out, resulting in the vertical white bands in the spike raster. The drift traces are just a visible side-effect of that.