Skip to content

Create NDCoord Index + Big Doc improvemntes#11

Merged
ianhi merged 12 commits intomainfrom
feature/absolute-relative-index
Dec 19, 2025
Merged

Create NDCoord Index + Big Doc improvemntes#11
ianhi merged 12 commits intomainfrom
feature/absolute-relative-index

Conversation

@ianhi
Copy link
Copy Markdown
Owner

@ianhi ianhi commented Dec 17, 2025

Create a new index for ND coords. e.g. abs_time in this dataset

<xarray.Dataset> Size: 28kB
Dimensions:   (trial: 3, rel_time: 500)
Coordinates:
  * trial     (trial) <U8 96B 'cosine' 'square' 'sawtooth'
  * rel_time  (rel_time) float64 4kB 0.0 0.01 0.02 0.03 ... 4.96 4.97 4.98 4.99
    abs_time  (trial, rel_time) float64 12kB 0.0 0.01 0.02 ... 14.97 14.98 14.99
Data variables:
    data      (trial, rel_time) float64 12kB 1.0 0.9995 0.998 ... -0.02 -0.01

Claude generated summary:

Add NDIndex for N-D Coordinate Selection

Summary

This PR introduces NDIndex, a new xarray Index that enables label-based selection on N-dimensional derived coordinates. The primary use case is trial-based data where you have a 2D abs_time coordinate (shape: trial × rel_time) computed from trial offsets plus relative time.

New Features

NDIndex (src/linked_indices/nd_index.py)

A simple xarray Index for N-D derived coordinates that:

  • Manages N-D coordinates (ndim ≥ 2) for label-based selection
  • Scalar selection: ds.sel(abs_time=7.5, method='nearest') finds the (trial, rel_time) indices nearest to the value
  • Slice selection: ds.sel(abs_time=slice(2, 8)) returns a bounding box containing all matching cells
  • Slice methods:
    • "bounding_box" (default): smallest rectangular region containing all matches
    • "trim_outer": like bounding_box but trims outer dimensions more aggressively
  • Step support: slice(0, 10, 2) applies step to the innermost dimension
  • Properly handles isel() with dimension reduction
  • Works correctly with xarray plotting

Example Data (src/linked_indices/example_data.py)

  • Added trial_based_dataset() function for generating test data with configurable waveforms (sine, cosine, square, sawtooth)

Documentation

New Notebooks

Notebook Description
nd_index_demo.ipynb Comprehensive tutorial covering basic usage, bounding box behavior, time-locking to events, and 3D coordinates. Includes dual-axis plots showing both rel_time and abs_time.
ndindex_slicing_gallery.ipynb Visual examples of slicing behavior using images. Demonstrates bounding box selection with diagonal gradients and radial coordinates.
ndindex_performance.ipynb Benchmarks for index creation (O(1)), scalar selection, slice selection, and isel overhead. Shows NDIndex is fast even for 10M+ cells.

Updated Documentation

  • index.md: Added NDIndex section with note that time-locking/epoching utilities are not yet implemented
  • myst.yml: Added new notebooks to table of contents

Diagram Images

  • abs-rel.png.excalidraw.png: Absolute vs relative time concept
  • event-locking.png.excalidraw.png: Event-locking/epoching concept
  • generic-intervals.png.excalidraw.png: DimensionInterval diagram

Testing

  • 68 tests in tests/test_nd_index.py with 100% code coverage
  • Test classes:
    • TestNDIndexCreation: Index creation and validation
    • TestNDIndexSelFollower: Scalar selection on N-D coordinates
    • TestNDIndexSelSlice: Slice selection
    • TestNDIndexIsel: Integer-based selection and dimension reduction
    • TestNDIndexSliceMethods: Slice method options (bounding_box, trim_outer)
    • TestNDIndexSliceMethods3D: 3D coordinate handling
    • TestNDIndexBoundingBoxBehavior: Comprehensive bounding box behavior tests

Performance Characteristics

Operation Complexity Notes
Index creation O(1) Just stores reference, no preprocessing
Scalar selection O(n) NumPy-optimized, ~25ms for 10M cells
Slice selection O(n) Boolean masking + bounding box computation
isel() O(1) ~1.3x overhead vs no index
Memory Zero copy NDIndex stores references, not copies

Breaking Changes

None - this is additive functionality.

How to Test

from linked_indices import NDIndex
from linked_indices.example_data import trial_based_dataset

# Create dataset with 2D abs_time coordinate
ds = trial_based_dataset(mode="stacked")
ds = ds.set_xindex(["abs_time"], NDIndex)

# Scalar selection - find where abs_time ≈ 7.5
ds.sel(abs_time=7.5, method="nearest")

# Slice selection - get bounding box for abs_time in [2, 8]
ds.sel(abs_time=slice(2, 8))

# Works with time-locked coordinates too
ds = ds.assign_coords(speech_locked=ds["rel_time"] - ds["speech_onset"])
ds = ds.drop_indexes("abs_time").set_xindex(["abs_time", "speech_locked"], NDIndex)
ds.sel(speech_locked=slice(-0.5, 1.0))  # Window around speech onset

@netlify
Copy link
Copy Markdown

netlify bot commented Dec 17, 2025

Deploy Preview for xarray-linked-indexes ready!

Name Link
🔨 Latest commit 29aa2db
🔍 Latest deploy log https://app.netlify.com/projects/xarray-linked-indexes/deploys/6945def89f88920008887d96
😎 Deploy Preview https://deploy-preview-11--xarray-linked-indexes.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov
Copy link
Copy Markdown

codecov bot commented Dec 17, 2025

Codecov Report

❌ Patch coverage is 97.41935% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.90%. Comparing base (07da9c1) to head (29aa2db).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
src/linked_indices/example_data.py 65.71% 8 Missing and 4 partials ⚠️
src/linked_indices/nd_index.py 98.48% 0 Missing and 2 partials ⚠️
tests/test_nd_index.py 99.55% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #11      +/-   ##
==========================================
+ Coverage   95.02%   95.90%   +0.87%     
==========================================
  Files           5        7       +2     
  Lines        1066     1685     +619     
  Branches       74      113      +39     
==========================================
+ Hits         1013     1616     +603     
- Misses         28       36       +8     
- Partials       25       33       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ianhi ianhi changed the title initial absrel index Create NDCoord Index Dec 19, 2025
@ianhi ianhi changed the title Create NDCoord Index Create NDCoord Index + Big Doc improvemntes Dec 19, 2025
@ianhi ianhi merged commit d812baa into main Dec 19, 2025
8 checks passed
@ianhi ianhi deleted the feature/absolute-relative-index branch December 19, 2025 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant