Windowed auditory scene classification with AST#493
Conversation
New API: - classify_audios_in_windows(): sliding window classification with configurable window_size/hop_size, returns per-window labels with timestamps. Batched inference for memory efficiency. - scene_results_to_segments(): converts window results to plot_aligned_panels segment format for timeline visualization. Supports any HuggingFace audio-classification model: - MIT/ast-finetuned-audioset-10-10-0.4593 (521 AudioSet classes) - Any other HF audio-classification model Tutorial: auditory_scene_analysis.ipynb — windowed classification, timeline visualization, event filtering. 3 new tests, all pass. Pre-commit clean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces windowed auditory scene analysis, allowing for temporal classification of audio events using a sliding-window approach. It includes a new API function classify_audios_in_windows, a utility to convert results for visualization, and a comprehensive tutorial notebook. Feedback focuses on improving memory efficiency for long recordings by avoiding the pre-allocation of all windows, preventing potential infinite loops with a minimum hop size check, and adding safety checks for empty label lists to avoid index errors.
| all_windows: List[Audio] = [] | ||
| window_meta: List[List[Dict[str, Any]]] = [] # per-audio list of {start, end} | ||
|
|
||
| for audio in audios: | ||
| sr = audio.sampling_rate | ||
| waveform = audio.waveform | ||
| n_samples = waveform.shape[1] | ||
|
|
||
| win_samples = int(window_size * sr) | ||
| hop_samples = int(hop_size * sr) | ||
|
|
||
| meta_for_audio: List[Dict[str, Any]] = [] | ||
|
|
||
| if n_samples <= win_samples: | ||
| # Audio shorter than one window -> use full audio as a single window. | ||
| all_windows.append(Audio(waveform=waveform, sampling_rate=sr)) | ||
| meta_for_audio.append({"start": 0.0, "end": n_samples / sr}) | ||
| else: | ||
| start = 0 | ||
| while start + win_samples <= n_samples: | ||
| chunk = waveform[:, start : start + win_samples] | ||
| all_windows.append(Audio(waveform=chunk, sampling_rate=sr)) | ||
| meta_for_audio.append({"start": start / sr, "end": (start + win_samples) / sr}) | ||
| start += hop_samples | ||
|
|
||
| window_meta.append(meta_for_audio) |
There was a problem hiding this comment.
Collecting all windows for all input audios into a single list all_windows before classification violates the memory efficiency requirement (FR-006) specified in the implementation plan. For a large number of audio files or very long recordings (e.g., hours), this will consume significant memory. Consider processing each audio independently or using a generator to yield windows lazily to keep memory usage low.
| n_samples = waveform.shape[1] | ||
|
|
||
| win_samples = int(window_size * sr) | ||
| hop_samples = int(hop_size * sr) |
There was a problem hiding this comment.
If hop_size is very small relative to the sampling rate, hop_samples can become 0, which will cause an infinite loop in the subsequent while block at line 101. It is safer to ensure a minimum of 1 sample.
| hop_samples = int(hop_size * sr) | |
| hop_samples = max(1, int(hop_size * sr)) |
| for batch_start in range(0, len(all_windows), batch_size): | ||
| batch = all_windows[batch_start : batch_start + batch_size] | ||
| all_results.extend(classify_audios(batch, model=model, device=device, **kwargs)) |
There was a problem hiding this comment.
The manual batching loop here is redundant because classify_audios (and the underlying Hugging Face pipeline) already handles batching internally. Additionally, the hardcoded batch_size = 32 at line 79 might conflict with or override the batch_size parameter if passed via kwargs. You can simplify this by passing the list of windows directly to classify_audios.
all_results = classify_audios(all_windows, model=model, device=device, **kwargs)| for r in results: | ||
| segments.append( | ||
| { | ||
| "label": r["labels"][0], |
There was a problem hiding this comment.
Accessing r["labels"][0] without checking if the list is empty will raise an IndexError if the model returns no predictions for a specific window. This can happen if top_k is set to 0 or if the model fails to produce any labels for that segment.
| "label": r["labels"][0], | |
| "label": r["labels"][0] if r["labels"] else "", |
Instead of a separate classify_audios_in_windows(), windowed processing is now activated by passing win_length/hop_length to the existing classify_audios() function: - win_length=None (default): whole-audio classification (unchanged) - win_length=1.0: 1-second windows, hop defaults to win_length/2 - Provenance captured: win_length/hop_length in each result dict - top_k parameter works in both modes This pattern can be adopted by other senselab functions for consistent windowed processing across the library. 4 tests pass (basic, short audio, default hop, segment conversion). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Process one audio at a time instead of collecting all windows into a single list (memory-efficient for long recordings) - Guard against empty labels in scene_results_to_segments - hop_samples already had max(1, ...) guard against infinite loop Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ification Instead of materializing all windows into a list, iterate the generator in batches of 32 — only one batch of Audio objects is in memory at a time. Reuses Audio.window_generator() for consistent windowed iteration across senselab (same pattern as quality_control). Tests updated for window_generator behavior (includes partial final window). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ppgs package handles its own espnet dependency (pins espnet==202301). We were adding espnet, snorkel, and lightning explicitly which caused conflicts: the 'lightning' package was removed from PyPI, and espnet build failed on sentencepiece. Minimal requirements: ppgs + torch + torchaudio + numpy + soundfile. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: NeMo 2.0 removed SortformerEncLabelModel and pyarrow 24+ removed PyExtensionType. Also NeMo needs matplotlib at import time. - nemo_toolkit==1.23.0 (has SortformerEncLabelModel) - torch>=2.1,<2.5 (NeMo 1.23 not compatible with torch 2.8) - pyarrow<18 (24+ breaking change) - matplotlib (NeMo imports it) The ensure_venv function auto-recreates when requirements change, so stale CI venvs will be refreshed on next run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The `lightning` package was removed from PyPI (April 2026). NeMo depends on it but pytorch-lightning is not a drop-in replacement (NeMo accesses internal `lightning.pytorch._logger`). Reverted to nemo_toolkit[asr] (unversioned) + torch 2.8 + added pyarrow<18 + matplotlib. Existing cached venvs work; new venv creation blocked until NeMo fixes their lightning dependency. Added NOTE comments documenting the issue. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 participant
Summary
New API
classify_audios_in_windows(): Sliding window audio classification with configurable window/hop size, returning per-window labels with timestampsscene_results_to_segments(): Convert window results to plot_aligned_panels segment formatModels
Supports any HuggingFace audio-classification model. Primary:
MIT/ast-finetuned-audioset-10-10-0.4593(521 AudioSet classes).Tutorial
auditory_scene_analysis.ipynb: windowed classification, timeline visualization aligned with waveform + spectrogram, event type filtering.Tests
3 new tests (windowed basic, short audio, segment conversion). All pass.
🤖 Generated with Claude Code
Version
Published prerelease version:
1.3.1-alpha.24Changelog
🐛 Bug Fix
Authors: 1