Tool-agnostic spec vs streamlined SpikeInterface loading #4
Description
An essential tension on this project.
It would be great if ndx-spikesorting were to be tool-agnostic without assuming SpikeInterface as the consumer. At the same time, we want a streamlined NWB to SortingAnalyzer to GUI loading path. This creates a tension because SpikeInterface extensions require both data and params to initialize, and some of those params are SpikeInterface-specific concepts that don't belong in a tool-agnostic spec. The key question for each gap is whether the SpikeInterface loader can derive what it needs from what NWB already stores, or whether the spec itself must change.
For both currently supported extensions (random_spikes and templates), all SpikeInterface parameters turn out to be recoverable by the loader (see #3). method and max_spikes_per_unit for random_spikes can be inferred from the per-unit spike counts in the ragged array; margin_size (I think this is only used at calculation time to exclude spikes near segment borders, right @alejoe91?) and seed (consumed during random selection) are never read back by any downstream consumer, so they safely default to None for precomputed results. For templates, ms_before and ms_after are derived exactly from peak_sample_index and sampling_frequency. No SpikeInterface-specific metadata needs to leak into the NWB spec for parameter recovery in these cases.
However, as new extensions are added to the spec, this same question will come up for each one: can the loader derive what SpikeInterface needs, or does the spec need to change? We can use this issue to document and discuss here or at least to link.