Skip to content

Avoid in-memory copies when filtering memmap signals in SequenceInterpolator - #1

Draft
reneburghardt with Copilot wants to merge 1 commit into
mainfrom
copilot/optimize-memmap-filtering
Draft

Avoid in-memory copies when filtering memmap signals in SequenceInterpolator#1
reneburghardt with Copilot wants to merge 1 commit into
mainfrom
copilot/optimize-memmap-filtering

Conversation

Copilot AI commented Mar 26, 2026

Copy link
Copy Markdown

Fancy-indexing a np.memmap with non-contiguous column indices materialises the entire selected dataset into RAM at init time, defeating the point of memory-mapping. This adds a signal_ids parameter to SequenceInterpolator (and PhaseShiftedSequenceInterpolator) with a tiered strategy to avoid that:

Filtering strategies

  • Contiguous IDs (e.g. [3,4,5,6]) → strided slice view, zero-copy, still lazy
  • Non-contiguous + uncached memmap_data stays as the full np.memmap; sorted indices stored in _signal_ids; column filter applied inside interpolate() after the per-query row read — the full array is never materialised
  • Cached or .npy → subset applied once at init (smaller allocation)

Usage

# Only signals [0, 7, 42] are ever loaded from disk per query
interp = SequenceInterpolator("path/to/data", signal_ids=[0, 7, 42])
assert isinstance(interp._data, np.memmap)  # still lazy
assert interp.n_signals == 3

Affected components

  • _is_contiguous_range() helper added
  • SequenceInterpolator.__init__: routing logic + _sorted_signal_ids attribute for subclasses to reuse without re-sorting
  • SequenceInterpolator.normalize_init: subsets mean/std for the lazy path
  • SequenceInterpolator.interpolate: applies _signal_ids after row read; fixes empty-array shape to use self.n_signals
  • PhaseShiftedSequenceInterpolator: subsets _phase_shifts via _sorted_signal_ids; interpolate() uses original column indices when _signal_ids is set (required because np.take_along_axis demands matching shapes, which don't hold in the lazy case)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants