sensein
diff --git a/‎specs/20260429-201758-auditory-scene-analysis/checklists/requirements.md‎
Lines changed: 36 additions & 0 deletions b/‎specs/20260429-201758-auditory-scene-analysis/checklists/requirements.md‎
Lines changed: 36 additions & 0 deletions
diff --git a/‎specs/20260429-201758-auditory-scene-analysis/plan.md‎
Lines changed: 76 additions & 0 deletions b/‎specs/20260429-201758-auditory-scene-analysis/plan.md‎
Lines changed: 76 additions & 0 deletions
diff --git a/‎specs/20260429-201758-auditory-scene-analysis/quickstart.md‎
Lines changed: 16 additions & 0 deletions b/‎specs/20260429-201758-auditory-scene-analysis/quickstart.md‎
Lines changed: 16 additions & 0 deletions
diff --git a/‎specs/20260429-201758-auditory-scene-analysis/research.md‎
Lines changed: 42 additions & 0 deletions b/‎specs/20260429-201758-auditory-scene-analysis/research.md‎
Lines changed: 42 additions & 0 deletions
diff --git a/‎specs/20260429-201758-auditory-scene-analysis/spec.md‎
Lines changed: 99 additions & 0 deletions b/‎specs/20260429-201758-auditory-scene-analysis/spec.md‎
Lines changed: 99 additions & 0 deletions
@@ -0,0 +1,36 @@
+# Specification Quality Checklist: Auditory Scene Analysis
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-04-29
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Spec is ready for `/speckit.plan`.
+- YAMNet may need subprocess venv if TensorFlow-based; HuggingFace AST models are PyTorch-native alternatives.
+- Windowed iteration pattern can reuse Audio chunking or simple tensor slicing.
@@ -0,0 +1,76 @@
+# Implementation Plan: Auditory Scene Analysis
+
+**Branch**: `20260429-201758-auditory-scene-analysis` | **Date**: 2026-04-29 | **Spec**: [spec.md](spec.md)
+
+## Summary
+
+Add windowed audio scene classification using HuggingFace AST (Audio Spectrogram Transformer) and other audio-classification models. Sliding windows with configurable size/hop produce per-window event labels with timestamps. Integrates with existing classification and visualization infrastructure.
+
+## Technical Context
+
+**Language/Version**: Python 3.11-3.12
+**Primary Dependencies**: transformers (HuggingFace audio-classification pipeline), existing senselab classification module
+**Testing**: pytest + papermill for tutorial
+**Target Platform**: CPU (GPU optional), Google Colab
+**Project Type**: Library feature + tutorial
+**Constraints**: Must work on CPU; window processing must be memory-efficient
+
+## Constitution Check
+
+| Principle | Status | Notes |
+|-----------|--------|-------|
+| I. UV-Managed Python | PASS | All via uv |
+| II. Encapsulated Testing | PASS | pytest in uv venv |
+| IV. CI Must Stay Green | PASS | New tests + existing pass |
+| VII. Simplicity First | PASS | Reuses existing classification pipeline + plot_aligned_panels |
+| VIII. No Hardcoded Parameters | PASS | Window/hop/top_k configurable with defaults |
+
+## Project Structure
+
+```text
+src/senselab/audio/tasks/classification/
+├── api.py                          # UPDATED (add classify_audios_in_windows)
+├── huggingface.py                  # UPDATED (windowed iteration support)
+└── doc.md                          # UPDATED (scene classification docs)
+
+tutorials/audio/
+└── auditory_scene_analysis.ipynb   # NEW
+
+src/tests/audio/tasks/
+└── classification_test.py          # UPDATED (windowed classification tests)
+```
+
+## Implementation Phases
+
+### Phase 1: Windowed Classification API (P1)
+
+1. Add `classify_audios_in_windows()` to `classification/api.py`:
+   - Accepts: audios, model, window_size, hop_size, top_k, device
+   - Slices each audio into windows (tensor slicing, not creating new files)
+   - Creates lightweight Audio objects per window
+   - Runs existing `classify_audios_with_transformers` in batches
+   - Returns: List[List[Dict]] — per-audio, per-window results with start/end/labels/scores
+2. Test with AST model on sample audio
+3. Test with different window/hop configurations
+
+### Phase 2: Multiple Models + Visualization (P2)
+
+1. Test with at least 2 models (AST + another HuggingFace audio-classification model)
+2. Create visualization helper or show how to use `plot_aligned_panels` with scene results
+3. Update doc.md with supported models
+
+### Phase 3: Tutorial (P2)
+
+1. Create `auditory_scene_analysis.ipynb`:
+   - Load audio (recording widget + sample)
+   - Run windowed classification with AST
+   - Visualize timeline of detected events
+   - Compare with different models
+   - Show integration with filtering/segmentation
+2. Add to manifest.json
+3. Test via papermill
+
+### Phase 4: Polish
+
+1. Pre-commit, tests, CI
+2. Push, check comments, merge
@@ -0,0 +1,16 @@
+# Quickstart: Auditory Scene Analysis
+
+## Run windowed classification
+```bash
+uv run python -c "
+from senselab.audio.data_structures import Audio
+from senselab.audio.tasks.classification.api import classify_audios_in_windows
+from senselab.utils.data_structures import HFModel
+
+audio = Audio(filepath='tutorial_audio_files/audio_48khz_mono_16bits.wav')
+model = HFModel(path_or_uri='MIT/ast-finetuned-audioset-10-10-0.4593')
+results = classify_audios_in_windows([audio], model=model, window_size=1.0, hop_size=0.5)
+for r in results[0][:5]:
+    print(f'{r[\"start\"]:.1f}-{r[\"end\"]:.1f}s: {r[\"labels\"][0]} ({r[\"scores\"][0]:.3f})')
+"
+```
@@ -0,0 +1,42 @@
+# Research: Auditory Scene Analysis
+
+**Date**: 2026-04-29
+
+## R1: Primary Model Choice
+
+**Decision**: Use HuggingFace Audio Spectrogram Transformer (AST) as the primary model instead of Google's TensorFlow-based YAMNet. Also support YAMNet via TFLite subprocess venv as secondary option.
+
+**Rationale**: AST is natively supported in HuggingFace Transformers (PyTorch), achieves state-of-the-art 0.485 mAP on AudioSet (521 classes), and works with senselab's existing `audio-classification` pipeline. No TensorFlow dependency needed. YAMNet requires TF/TFLite which would conflict with the main PyTorch environment.
+
+**Models to support**:
+1. `MIT/ast-finetuned-audioset-10-10-0.4593` — AST fine-tuned on AudioSet (PyTorch, HuggingFace pipeline)
+2. `google/yamnet` or `STMicroelectronics/yamnet` — YAMNet via HuggingFace if available, or TFLite subprocess venv
+3. Any HuggingFace `audio-classification` model — generic support via existing pipeline
+
+**Alternatives considered**:
+- PANNs: Not natively on HuggingFace Transformers; would need custom loading
+- BEATs: Available on HuggingFace but fewer pre-trained checkpoints for AudioSet
+- CLAP: Good for zero-shot but different use case (text-audio retrieval)
+
+## R2: Windowed Iteration Approach
+
+**Decision**: Implement windowed classification by slicing the audio waveform tensor into overlapping windows, batching them, and running the classifier on each batch. Return results as a list of per-window classification outputs with timestamps.
+
+**Rationale**: Simple tensor slicing is efficient and doesn't require creating separate Audio objects per window. The existing `classify_audios_with_transformers` can process a list of Audio objects, so we create lightweight Audio objects from window slices.
+
+**Parameters**:
+- `window_size`: float (seconds), default 1.0
+- `hop_size`: float (seconds), default 0.5
+- `top_k`: int (number of top predictions per window), default 5
+
+## R3: Output Format
+
+**Decision**: Return `List[Dict]` where each dict has `start`, `end`, `labels`, `scores`. This is compatible with the `segments` panel type in `plot_aligned_panels`.
+
+**Rationale**: The dict format is simple, JSON-serializable, and compatible with existing visualization patterns. It can be easily converted to DataFrames or segment annotations.
+
+## R4: Visualization
+
+**Decision**: Reuse `plot_aligned_panels` with a new panel approach — the "segments" panel type already supports colored bars with labels. For scene classification, we show the top-1 predicted class per window as a colored bar.
+
+**Rationale**: No new plotting code needed — existing infrastructure handles this.
@@ -0,0 +1,99 @@
+# Feature Specification: Auditory Scene Analysis with YAMNet and Windowed Classification
+
+**Feature Branch**: `20260429-201758-auditory-scene-analysis`
+**Created**: 2026-04-29
+**Status**: Draft
+**Input**: Add YAMNet and other related models for auditory scene analysis in iterated form over windows.
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Classify Audio Scenes Over Time Windows (Priority: P1)
+
+A researcher has a long audio recording (e.g., 30 minutes of environmental audio, a clinical interview, or a classroom recording) and wants to understand what sound events occur and when. They run an auditory scene classifier (YAMNet or similar) over sliding windows, producing a timeline of detected sound events — speech, music, laughter, traffic, silence, dog bark, etc. — with timestamps showing when each event occurs and how confident the model is.
+
+**Why this priority**: Windowed audio classification is a fundamental building block for audio understanding. Current senselab classification is whole-file only — no temporal resolution. Windowed analysis is needed for long recordings where the acoustic environment changes over time.
+
+**Independent Test**: Run YAMNet on a 30-second audio sample with 1-second windows, verify output contains per-window classifications with timestamps and confidence scores.
+
+**Acceptance Scenarios**:
+
+1. **Given** a long audio recording, **When** the user runs windowed scene classification, **Then** they receive a list of per-window results, each containing the top predicted sound event(s) with confidence scores and the time interval (start/end in seconds).
+2. **Given** a window size and hop size, **When** the user specifies them, **Then** windows are generated accordingly (e.g., 1s windows with 0.5s hop for overlapping analysis).
+3. **Given** default parameters, **When** the user runs without specifying window/hop, **Then** reasonable defaults are used (e.g., 1s window, 0.5s hop).
+4. **Given** the windowed results, **When** the user visualizes them, **Then** they see a timeline plot showing sound events over time aligned with the audio waveform.
+
+---
+
+### User Story 2 - Use Multiple Audio Scene Models (Priority: P2)
+
+A user wants to compare different audio scene classifiers on the same recording. They can choose from multiple models (YAMNet, AudioSet-based models, PANNs, BEATs, etc.) through the same API, each providing different label sets and granularities. The API abstracts the model-specific details.
+
+**Why this priority**: Different models have different strengths — YAMNet has 521 AudioSet classes, PANNs have fine-grained event detection, BEATs uses self-supervised pre-training. Users need to compare and choose the right model for their domain.
+
+**Independent Test**: Run the same audio through 2 different scene classifiers and verify both return per-window classifications with their respective label sets.
+
+**Acceptance Scenarios**:
+
+1. **Given** an audio file, **When** the user runs the scene classifier with a YAMNet-family model, **Then** they receive predictions from the 521-class AudioSet ontology.
+2. **Given** the same audio file, **When** the user runs with a different model (e.g., a PANN or BEATs variant), **Then** they receive predictions from that model's label set.
+3. **Given** results from multiple models, **When** displayed together, **Then** the user can compare which events each model detects.
+
+---
+
+### User Story 3 - Integrate Scene Analysis into Quality Control and Pipelines (Priority: P3)
+
+A researcher uses windowed scene classification as part of a larger pipeline — for example, detecting non-speech segments (music, noise) for quality control, or identifying when a speaker transitions from quiet speech to laughter. The per-window results can be used to filter, segment, or annotate audio programmatically.
+
+**Why this priority**: Scene classification becomes most valuable when integrated into workflows — filtering out non-speech regions before ASR, detecting environmental noise for quality assessment, or segmenting recordings by acoustic context.
+
+**Independent Test**: Use windowed classification to identify and extract only speech segments from a mixed audio recording.
+
+**Acceptance Scenarios**:
+
+1. **Given** windowed classification results, **When** the user filters for "Speech" labels above a confidence threshold, **Then** they get time segments containing speech.
+2. **Given** windowed results, **When** used as input to another senselab task (e.g., extract only speech segments for ASR), **Then** the integration works seamlessly with senselab's Audio chunking utilities.
+
+---
+
+### Edge Cases
+
+- What happens when the audio is shorter than the window size? (Use the full audio as a single window.)
+- What happens when the model returns many low-confidence predictions? (Return top-k results per window, configurable.)
+- What happens with very long audio (hours)? (Process in batches; memory-efficient iteration over windows.)
+- What happens with multi-channel audio? (Require mono; raise error with guidance to downmix first.)
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: The system MUST support windowed audio classification where the user specifies window size (in seconds) and hop size (in seconds), with reasonable defaults (1s window, 0.5s hop).
+- **FR-002**: The system MUST support YAMNet (521-class AudioSet ontology) as the primary auditory scene classifier, accessible through senselab's existing classification API or a new dedicated function.
+- **FR-003**: The system MUST support at least one additional audio scene model beyond YAMNet (e.g., a PANN, BEATs, or CLAP variant from HuggingFace) to demonstrate model flexibility.
+- **FR-004**: Each window's result MUST include: start time (seconds), end time (seconds), top-k predicted labels, and confidence scores.
+- **FR-005**: The system MUST provide a visualization function that plots detected sound events over time, aligned with the audio waveform (similar to plot_aligned_panels with a "segments" panel).
+- **FR-006**: The windowed classification MUST handle audio of any length efficiently, processing windows in batches rather than loading all windows into memory at once.
+- **FR-007**: The system MUST follow established senselab tutorial conventions with a tutorial demonstrating windowed scene analysis on sample audio.
+
+### Key Entities
+
+- **AudioWindow**: A segment of audio with defined start and end time, extracted from a longer recording.
+- **SceneClassificationResult**: Per-window classification output containing time interval, predicted labels, and confidence scores.
+- **AudioSet Ontology**: The 521-class label hierarchy used by YAMNet and many audio scene classifiers.
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: Windowed classification on a 30-second sample produces at least 30 per-window results (with 1s window, 0.5s hop = ~59 windows).
+- **SC-002**: At least 2 different audio scene models are supported and produce results through the same API.
+- **SC-003**: The visualization function produces a timeline plot showing detected events aligned with the waveform.
+- **SC-004**: Processing a 5-minute audio file completes within 60 seconds on CPU.
+- **SC-005**: A tutorial notebook demonstrates windowed scene analysis and passes CI via papermill.
+
+## Assumptions
+
+- YAMNet is available as a TensorFlow/TFLite model or via a HuggingFace-compatible wrapper. If TensorFlow-only, it may need a subprocess venv to avoid TF/PyTorch conflicts.
+- HuggingFace has audio-classification models trained on AudioSet that can serve as alternatives to YAMNet (e.g., MIT/ast-finetuned-audioset-10-10-0.4593).
+- The windowed iteration uses senselab's existing Audio chunking utilities (chunk_audios) or a simple sliding window over the waveform tensor.
+- The output format is compatible with senselab's existing classification API (AudioClassificationResult) extended with time intervals.
+- Window size and hop size are specified in seconds and converted to samples internally based on sampling rate.