Skip to content

Commit 961d9cb

Browse files
authored
Merge pull request #56 from DigitalHolography/spectrum_analysis
Correct spectrogram analysis and band-ratio edge cases
2 parents 2331ef3 + 8220d0c commit 961d9cb

26 files changed

Lines changed: 1022 additions & 341 deletions

‎AGENTS.md‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,14 @@ The source layout requires `src` on `sys.path`; an editable install provides
9999
that. If using an existing interpreter without an editable install, set
100100
`PYTHONPATH` to the repository's `src` directory for the command.
101101

102-
Before changing a contract, search both its producer and consumers. Before
103-
finishing, run the focused tests from [the test map](test/AGENTS.md), then the
104-
full suite when the change crosses subsystem boundaries.
102+
Before changing a contract, search both its producer and consumers. Keep
103+
verification proportional to the change, using [the test map](test/AGENTS.md)
104+
to find the smallest relevant test cases rather than running every mapped file.
105+
For simple output-path/name changes, run only a focused contract test; for
106+
documentation-only changes, do not run tests. Avoid expensive end-to-end
107+
workflow tests for path, naming, or presentation changes when focused coverage
108+
verifies the behavior. Crossing a directory or subsystem boundary alone does
109+
not require the full suite. Run broad integration tests or the full suite only
110+
when requested or when a concrete behavior risk cannot be covered by smaller
111+
checks. Do not repeat passing checks unless a relevant change, failure, or
112+
unresolved concern justifies it.

‎docs/data-contracts.md‎

Lines changed: 117 additions & 58 deletions
Large diffs are not rendered by default.

‎src/calculations/math/frequency.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
"""Full-spectrum frequency calibration independent of pipeline and output policy."""
2+
3+
from __future__ import annotations
4+
5+
import numpy as np
6+
7+
8+
def frequency_bin_spacing_khz(bin_count: int, sampling_frequency_hz: float) -> float:
9+
"""Return the positive full-spectrum bin spacing in kHz."""
10+
if bin_count < 1 or not np.isfinite(sampling_frequency_hz) or sampling_frequency_hz <= 0:
11+
raise ValueError("Frequency calibration requires bins and a positive finite sampling frequency.")
12+
return float(sampling_frequency_hz / (bin_count * 1000))
13+
14+
15+
def frequency_coordinates_khz(bin_count: int, sampling_frequency_hz: float) -> np.ndarray:
16+
"""Map positive-then-negative bins with spacing fs/N and a floor(N/2) split.
17+
18+
For 64 bins, indices 0..31 are positive/DC and index 32 is -fs/2.
19+
A single-bin spectrum is DC. Preserve source order, including odd domains.
20+
"""
21+
spacing = frequency_bin_spacing_khz(bin_count, sampling_frequency_hz)
22+
bins = np.arange(bin_count, dtype=np.float64)
23+
if bin_count > 1:
24+
bins[bin_count // 2:] -= bin_count
25+
return bins * spacing
26+
27+
28+
def positive_frequency_coordinates_khz(bin_count: int, sampling_frequency_hz: float) -> np.ndarray:
29+
"""Return the actual positive/DC half of the full-spectrum frequency grid."""
30+
return frequency_coordinates_khz(bin_count, sampling_frequency_hz)[:bin_count // 2]
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
"""Reusable closed phase windows, NaN-aware reductions, and late-minus-early deltas."""
2+
3+
from __future__ import annotations
4+
5+
import numpy as np
6+
7+
from .statistics import nanmedian
8+
9+
DEFAULT_EARLY_PHASE_INTERVAL = (0.0, 1 / 3)
10+
DEFAULT_LATE_PHASE_INTERVAL = (1 / 3, 1.0)
11+
12+
13+
def early_late_phase_windows(
14+
phase_samples, *, early_interval=DEFAULT_EARLY_PHASE_INTERVAL,
15+
late_interval=DEFAULT_LATE_PHASE_INTERVAL,
16+
) -> dict[str, tuple[np.ndarray, list[float]]]:
17+
"""Select closed early/late intervals expressed as fractions of a cardiac cycle.
18+
19+
Preserve source-dtype comparisons so float32 coordinates at a shared boundary
20+
belong to both windows. Phase one remains an excluded source endpoint.
21+
"""
22+
phase = np.asarray(phase_samples)
23+
if phase.dtype.kind != "f":
24+
phase = phase.astype(np.float64)
25+
if (
26+
phase.ndim != 1 or phase.size == 0 or not np.all(np.isfinite(phase))
27+
or np.any(phase < 0) or np.any(phase >= 1) or np.any(np.diff(phase) <= 0)
28+
):
29+
raise ValueError("Phase samples must be strictly increasing coordinates in [0, 1).")
30+
windows = {}
31+
for name, interval in (("early", early_interval), ("late", late_interval)):
32+
bounds = np.asarray(interval, dtype=np.float64)
33+
if (
34+
bounds.shape != (2,) or not np.all(np.isfinite(bounds))
35+
or not 0 <= bounds[0] <= bounds[1] <= 1
36+
):
37+
raise ValueError("Phase intervals must contain two ordered finite bounds in [0, 1].")
38+
lower, upper = (phase.dtype.type(bound) for bound in bounds)
39+
windows[name] = ((phase >= lower) & (phase <= upper), bounds.tolist())
40+
return windows
41+
42+
43+
def phase_window_medians(values, windows, *, axis: int = 0) -> dict[str, np.ndarray]:
44+
"""Reduce any named phase windows, ignoring NaNs and retaining missing cells."""
45+
values = np.asarray(values, dtype=np.float64)
46+
if values.ndim < 1:
47+
raise ValueError("Phase reductions require an array with a phase axis.")
48+
values = np.moveaxis(values, axis, 0)
49+
result = {}
50+
for name, (mask, _interval) in windows.items():
51+
mask = np.asarray(mask)
52+
if mask.dtype != np.bool_ or mask.shape != values.shape[:1]:
53+
raise ValueError("Phase masks must be Boolean and match the phase axis.")
54+
result[name] = nanmedian(values[mask], axis=0, dtype=np.float64)
55+
return result
56+
57+
58+
def late_minus_early(early, late):
59+
"""Return the signed late-minus-early difference, propagating missing values."""
60+
return np.asarray(late, dtype=np.float64) - np.asarray(early, dtype=np.float64)

‎src/input_output/schema/eyeflow_output.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,7 @@ class EyeFlowOutputPaths:
145145
meta_root: str
146146
sampling_frequency_hz: str
147147
blood_volume_rate: BloodVolumeRateOutputPaths
148+
spectrograms_frequency_bin_table: str
148149
spectrograms_spatial_average: str
149150
spectrograms_spatial_median: str
150151
spectrograms_masked: str
@@ -359,8 +360,9 @@ def _velocity_profile_paths(
359360
meta_root="Meta",
360361
sampling_frequency_hz="Meta/SamplingFrequencyHz/value",
361362
blood_volume_rate=_blood_volume_rate_paths("ProcessingMoments/BloodVolumeRate"),
362-
spectrograms_spatial_average="ProcessingSpectrograms/SpatialAverage",
363-
spectrograms_spatial_median="ProcessingSpectrograms/SpatialMedian",
363+
spectrograms_frequency_bin_table="ProcessingSpectrograms/FrequencyBinTable",
364+
spectrograms_spatial_average="ProcessingSpectrograms/Global/SpatialAverage",
365+
spectrograms_spatial_median="ProcessingSpectrograms/Global/SpatialMedian",
364366
spectrograms_masked="ProcessingSpectrograms/Gaps/SpatialMedian",
365367
spectrograms_masked_per_beat="ProcessingSpectrograms/Gaps/PerBeat/Raw",
366368
spectrograms_masked_time_aggregated_median=(

‎src/input_output/writers/h5.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -318,6 +318,17 @@ def _create_dataset(
318318
dtype=h5py.string_dtype(encoding="utf-8"),
319319
)
320320
options = dict(h5_options or {})
321+
# HDF5 filters require chunked arrays; scalars and strings stay unfiltered.
322+
if (
323+
isinstance(payload, np.ndarray)
324+
and payload.ndim > 0
325+
and payload.dtype.kind in {"b", "i", "u", "f", "c"}
326+
):
327+
options.setdefault("compression", "gzip")
328+
if options["compression"] == "gzip":
329+
options.setdefault("compression_opts", 6)
330+
if options["compression"] is not None:
331+
options.setdefault("shuffle", True)
321332
if isinstance(payload, np.ndarray) and payload.dtype.kind in {"U", "O"}:
322333
if all(isinstance(item, (str, bytes)) for item in payload.flat):
323334
payload = np.asarray(payload, dtype=object)

‎src/pipelines/AGENTS.md‎

Lines changed: 47 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -66,12 +66,14 @@ The velocity estimator is product-specific and therefore remains in
6666
`velocity/estimation.py` rather than `calculations`. It processes `(frame, y,
6767
x)` volumes in bounded temporal chunks, uses moments or calibrated frequency
6868
bands as its active source, applies the shared local-background difference, and
69-
returns physical velocity in `mm/s`. Exact-zero LF samples map to zero in band
70-
mode; finite/non-negative validation and calibration provenance are part of the
69+
returns physical velocity in `mm/s`. In band mode, finite negative samples in
70+
both HF and LF are replaced with zero during chunk validation, with a warning
71+
naming the dataset and frame interval. Zero LF denominators map to ratio zero
72+
without division. Both bands must remain finite; validation and calibration provenance are part of the
7173
pipeline contract.
7274

7375
`low_freq_spectra/spectrograms.py` reads HD frequency-bin spectra once per frame for
74-
full-frame `SpatialAverage`/`SpatialMedian` reductions and optional gap median
76+
full-frame `Global/SpatialAverage`/`Global/SpatialMedian` reductions and optional gap median
7577
power. `low_freq_spectra/preprocessing.py` interpolates Boolean gap support with
7678
nearest neighbors to the spectrogram video's last two dimensions. The low-frequency spectral
7779
runner publishes `/ProcessingSpectrograms/Gaps/SpatialMedian` with `(time, freq)`
@@ -101,28 +103,41 @@ frequency order, window endpoints, and selected/valid sample counts.
101103
over beats after each beat's whole/early/late phase median, reusing `PerBeat/Raw`.
102104
Ignore NaNs; one valid beat gives zero, none gives NaN. Persist raw power units,
103105
`ddof=0`, reduction order, windows, and valid beat counts per frequency.
104-
Positive-frequency PNG/EPS overlays use these STD envelopes and normalize each
105-
curve by its own maximum over bins 0 through 31. Their frequency grid is fixed
106-
at `linspace(0, fs/4, 33)` in Hz, independent of source length. All three figures
107-
report the same stored `Metrics/index_max1` through bin 31 inclusive. Keep power
108-
linear, scale STD by the same per-curve maximum, and do not save the grid to HDF5.
109-
Missing/invalid stored peaks or fewer than 32 bins omit these figures.
106+
Positive-frequency PNG/EPS overlays use the actual positive/DC half of the
107+
source: `P=floor(N/2)` bins at spacing `fs/N`. Normalize each curve by the
108+
Hv, the same NaN-aware central-quarter median of the full spectrum used by A0
109+
(64 source bins use bins 24 through 39), before cropping from stored `Metrics/index_max1`
110+
through bin `P-1` inclusive. Scale STD by the same per-curve divisor, retain
111+
linear power. The runner publishes the full signed frequency mapping once at
112+
`/ProcessingSpectrograms/FrequencyBinTable`, as a `(freq,)` array in `kHz` where
113+
each zero-based array index identifies the source bin. Use the
114+
shared conversions in `calculations/math/frequency.py` for tables, metrics, and
115+
plots. Table publication requires valid sampling calibration, independently of
116+
gap support, cardiac timing, and figure export. Missing/invalid stored peaks,
117+
all-NaN neighborhoods, or nonpositive divisors omit the affected figures.
110118
Median/early/late use solid/dashed/dotted signals and envelope patterns.
111-
The full-domain heatmap/median frequency axes use spacing `fs/(2*N)` with DC zero,
119+
The full-domain heatmap/median frequency axes use full-spectrum spacing `fs/N` with DC zero,
112120
positive-then-negative order, and kHz labels. Resolve `fs` from shared
113121
`Meta/SamplingFrequencyHz/value`; publish it from HD timing parameters/config
114122
before export if absent, since velocity analysis may run later. Missing/invalid
115123
sampling metadata skips frequency figures while keeping other products.
116124
The spectrum figure and pooled scalar metrics consume `MedianPhaseBeat`.
117125
All gap metrics live directly under `/ProcessingSpectrograms/Gaps/Metrics`.
118-
Phase curves `A0_t`, `A1_t`, `A2_t`, `R1_t`, and `R2_t` use `MedianBeat` with
119-
the pooled peak indices fixed at every phase. Retain the original pooled scalar
120-
names `A0`/`A1`/`A2`/`R1`/`R2`; remove output heights `h0`/`h1`/`hv` and `rho_h`.
126+
Phase curves `A0_t`, `A1_t`, `A2_t`, `R1_t`, `R2_t`, `Z1_t`, and `Z2_t` use
127+
`MedianBeat` with the pooled peak indices fixed at every phase. Pooled scalar
128+
metrics use the same seven names without `_t`; output heights `h0`/`h1`/`hv`
129+
and `rho_h` are absent.
121130
Each curve also has `_early` and `_late` scalar medians over closed phase
122-
intervals `[0,1/3]` and `[1/3,1]`, and a signed `delta_<metric>` early-minus-late
123-
scalar. Ignore NaNs within each window; missing windows/differences remain NaN.
131+
intervals (default `[0,1/3]` and `[1/3,1]`), and a signed `delta_<metric>` scalar.
132+
All seven deltas are late-minus-early. Generic helpers in
133+
`calculations/math/phase_windows.py` own closed interval masks, NaN-aware medians,
134+
and subtraction. `run_gap_analysis` and `pack_spectrum_metrics` accept
135+
`early_interval`/`late_interval` cycle-fraction bounds in `[0,1]`; use the same
136+
windows for early/late spectra, STD envelopes, and metric summaries/deltas.
137+
Keep subtraction order explicit in definitions, source paths, and calculation
138+
metadata. Ignore NaNs within each window; missing windows/differences remain NaN.
124139
Keep the source's excluded endpoint and include a one-third boundary sample in
125-
both windows when present. Curves retain phase/timing/peak provenance and also
140+
both default windows when present. Curves retain phase/timing/peak provenance and also
126141
produce shared `gaps_<metric>` PNG/EPS plots.
127142
The valley height `hv` uses a NaN-aware median over a neighborhood centered on
128143
the two central bins. Its default length is one quarter of the actual domain
@@ -132,8 +147,22 @@ with at least two bins is supported; an explicit even `valley_neighborhood_lengt
132147
overrides the default. Peak halves and central valley bins scale with the domain,
133148
splitting at `floor(bin_count/2)` for odd lengths. All-NaN peak halves retain NaN
134149
indexes, log a warning, and are identified by per-half valid-bin counts in metadata.
135-
`A2` uses the same inclusive peak-to-peak width as `A0`.
136-
`R1=A1/A0` replaces `rho_A`; `R2=A2/A0`. Metric metadata records the neighborhood.
150+
`A0` is valley-median height times peak-to-peak width in kHz; `A2` is the
151+
peak-height mean times the same width. These swap the former A0/A2 meanings.
152+
Use `df_khz=fs/(N*1000)` and width `(index_max2-index_max1)*df_khz`, with no
153+
extra bin. `A1` is the trapezoidal integral between those peak bins with positive
154+
`df_khz` spacing in stored HD order (unwrapped kHz, no signed-axis sorting).
155+
Areas and their phase summaries/deltas have source-power-times-kHz units;
156+
peak/valley heights retain power units. Preserve calibration, spacing, integration
157+
method, and units in metadata. Artifact axis labels abbreviate `a.u.*kHz` to
158+
`a.u.`. Resolve calibration from shared
159+
`Meta/SamplingFrequencyHz/value`; missing/invalid calibration warns and skips
160+
the Metrics family while retaining raw/aggregate spectra and filters.
161+
`R1=A1/A0`, `R2=A2/A0`,
162+
`Z1=(A1-A0)/A0`, and `Z2=(A2-A0)/A0` are dimensionless; zero/missing A0 gives
163+
NaN. All seven metrics have pooled scalars, phase curves, early/late medians,
164+
signed late-minus-early deltas, and matching PNG/EPS phase plots.
165+
Metric metadata records the neighborhood.
137166
Independent `PerBeat/BandLimited` keeps DC plus the configured non-DC harmonics
138167
(default 12) via the shared bandlimiting API in signed mode, using
139168
periodic phase boundaries. Original durations define each effective Hz cutoff.

0 commit comments

Comments
 (0)