Parallelize coherence-based bad channel detection - #4750
Open
JESUSROYETH wants to merge 1 commit into
Open
Conversation
alejoe91
reviewed
Sep 3, 2026
Comment on lines
+291
to
+297
| if job_kwargs is None: | ||
| job_kwargs = {"progress_bar": False} | ||
| job_kwargs = fix_job_kwargs(job_kwargs) | ||
| executor_job_kwargs = { | ||
| key: job_kwargs[key] | ||
| for key in ("pool_engine", "n_jobs", "progress_bar", "mp_context", "max_threads_per_worker") | ||
| } |
Member
There was a problem hiding this comment.
Suggested change
| if job_kwargs is None: | |
| job_kwargs = {"progress_bar": False} | |
| job_kwargs = fix_job_kwargs(job_kwargs) | |
| executor_job_kwargs = { | |
| key: job_kwargs[key] | |
| for key in ("pool_engine", "n_jobs", "progress_bar", "mp_context", "max_threads_per_worker") | |
| } | |
| job_kwargs = {} if job_kwargs is None else job_kwargs | |
| job_kwargs = fix_job_kwargs(job_kwargs) |
@JESUSROYETH why isn't this enough to propagate to the TimeSeriesChunkExecutor?
Member
There was a problem hiding this comment.
from spikeinterface.core.job_tools import fix_job_kwargs
job_kwargs = dict(n_jobs=4)
print(fix_job_kwargs(job_kwargs)
>>> {'pool_engine': 'process',
'n_jobs': 4,
'chunk_duration': '1s',
'progress_bar': True,
'mp_context': None,
'max_threads_per_worker': 1}
alejoe91
reviewed
Sep 3, 2026
alejoe91
left a comment
Member
There was a problem hiding this comment.
Thanks @JESUSROYETH
I just have one comment but this looks great! :)
alejoe91
reviewed
Sep 3, 2026
| handle_returns=True, | ||
| chunk_size=random_chunk_kwargs["chunk_size"], | ||
| job_name="detect_bad_channels", | ||
| **executor_job_kwargs, |
Member
There was a problem hiding this comment.
Suggested change
| **executor_job_kwargs, | |
| **job_kwargs, |
See https://github.com/SpikeInterface/spikeinterface/pull/4750/changes#r3923673650
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The default
coherence+psdmethod first materialises 100 scaled chunks and then processes them in serial. On a 384-channel recording, profiling shows that the Welch and median computations dominate the call, while extracting the random traces takes a small part of the total time.This change sends the same random slices through
TimeSeriesChunkExecutor. The default stays atn_jobs=1, but the chunks are streamed instead of kept together in memory. A finaljob_kwargsparameter lets callers select a thread or process pool without changing positional arguments.I measured the default 100 × 0.3 s chunk path on an Intel Core i9-13900HX, with numerical thread pools pinned to one thread. The generated row uses the 30 s / 384-channel shape reported in the issue. The second row uses real 384-channel SpikeGLX traces from the public Noise4Sam fixture; the public file is short, it was repeated in memory to cover the same chunk path.
n_jobs=8, processes)The channel ids and labels matched exactly between both implementations in each run. On the serial real-trace path, three fresh-process peak RSS measurements gave a median of 1,688,604 KiB before (range 1,688,500-1,689,352) and 353,452 KiB after (range 352,984-354,896), around 79% less. This comes from processing one random slice at a time instead of retaining the scaled chunks.
Validation:
pytest -q src/spikeinterface/preprocessing/tests/test_detect_bad_channels.py: 11 passed.highpass_filterwrapperdetect_bad_channelsbuilds internally also gets exercised across a process boundary; it matches the serial result on the same data.mainrevision and pass with this patch.blackand the trailing-whitespace/end-of-file checks are clean.Fixes #2869.