error after "read and addd to neighborhood"

[gmbcrazy]

Hello Magland,

I tried to sort one of the file (final mda file: 26G), 32 channels of 2 hours recording, there's an error as bellow. I doubled if it is because the file is too big. When I sort part of the same datasets (final mda file: 15G), it's fine.

I'm using my own multi-pieline modifed from your mountainsort3.mlp. The .mlp file I used is uploaded.

The command line is

mlp-run LU.mlp sort --raw=Raw.mda --geom=geom.csv --firings_out=Step1firings.mda --_params=params.json --curate=false --metrics_out=Step1cluster_metrics.json --pre_out=pre.mda --templates_out=Step1templates.mda

|mountainsortalg.ms3| Processed 62.564 MB in 0.448 sec (139.652 MB/sec): Total 26402 MB [Read data and add to neighborhood sorters] (98.7563%)...
|mountainsortalg.ms3| Processed 62.564 MB in 0.458 sec (136.603 MB/sec): Total 26464.6 MB [Read data and add to neighborhood sorters] (98.9903%)...
|mountainsortalg.ms3| 2017-12-15:12-57-05-032 mproc.main warning
|mountainsortalg.ms3| 2017-12-15:12-57-05-034 mproc.main warning Process returned with non-zero exit code (9): mountainsortalg.ms3 (Elapsed: 211.434 sec)
|mountainsortalg.ms3| making prv file: /home/singerlab/TempSorting/mountainlab/prvbucket/_mountainprocess/output_console_out_18ce164a78.prv
Process error:
Process error:
Error in job (processor: mountainsortalg.ms3): Process error:
Could not find input in running or pending jobs: firings_out
Error: Error in job (processor: mountainsortalg.ms3): Process error:
Error: Error in job (processor: mountainsortalg.ms3): Process error:
Stopped

LU.mlp.txt

[magland]

Could you be running out of memory? How much memory to you have on your system?

[gmbcrazy]

Hi Magland, Thanks for your reply. Yes, it's memory problem. I have a 64G memory machine. I checked the exact data, which is 32 channels and 2.9 hours recording. And I didn't define the "clip_size" (default is 50), after changing to 32, it's fine. I watched the compute resources occupied during the whole process, it only when "Read data and adding to neighborhood sorter" would cause extremely high RAM used.

I wonder the possibility to optimize that process? For example save part of those data from RAM to hard disk as temporary file and re-imported later? I also tried reduce cores as an closed issue suggested, it seems help very little.

BTW, I wonder if it would help that I re-set other parameters? Such as make neighborhood radius smaller? I tried, but the improvement is not very obvious. I was quite worried with much more channels of data in future.