|
| 1 | +# Implementation Plan: Pedagogical Audio Tutorials and PPG Phoneme Durations |
| 2 | + |
| 3 | +**Branch**: `20260423-213942-pedagogical-tutorials` | **Date**: 2026-04-23 | **Spec**: [spec.md](spec.md) |
| 4 | +**Input**: Feature specification from `/specs/20260423-213942-pedagogical-tutorials/spec.md` |
| 5 | + |
| 6 | +## Summary |
| 7 | + |
| 8 | +Create two new pedagogical audio tutorials for senselab, update two existing course notebooks to use senselab APIs, and implement fresh PPG phoneme duration analysis (superseding PR #431). Tutorial 1 covers audio recording/loading, waveform/spectrogram visualization, pitch extraction, formant extraction, emotion detection, and speaker matching. Tutorial 2 covers ASR, forced alignment, and PPG-based phoneme timeline analysis. |
| 9 | + |
| 10 | +## Technical Context |
| 11 | + |
| 12 | +**Language/Version**: Python 3.11-3.12 (Colab uses 3.12) |
| 13 | +**Primary Dependencies**: senselab (the library being tutorialized), papermill (CI execution), ipywebrtc or JS widgets (recording) |
| 14 | +**Storage**: N/A (notebooks are files in the repo) |
| 15 | +**Testing**: papermill for notebook execution in CI, pytest for PR #431 unit tests |
| 16 | +**Target Platform**: Google Colab (primary), local Jupyter (secondary) |
| 17 | +**Project Type**: Library tutorials (Jupyter notebooks) |
| 18 | +**Performance Goals**: Each tutorial completes in <30 minutes for students; CI execution within timeout limits |
| 19 | +**Constraints**: Must work on CPU (GPU optional); recording cells must be skippable for CI |
| 20 | +**Scale/Scope**: 4 notebooks (2 new, 2 updated), 1 PR merge, manifest updates |
| 21 | + |
| 22 | +## Constitution Check |
| 23 | + |
| 24 | +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* |
| 25 | + |
| 26 | +| Principle | Status | Notes | |
| 27 | +|-----------|--------|-------| |
| 28 | +| I. UV-Managed Python | PASS | Tutorials use `!pip install uv` + `!uv pip install` in Colab; CI uses `uv run papermill` | |
| 29 | +| II. Encapsulated Testing | PASS | CI runs notebooks via papermill in uv-managed venv | |
| 30 | +| III. Commit Early and Often | PASS | Plan calls for incremental commits per task | |
| 31 | +| IV. CI Must Stay Green | PASS | All notebooks tested in CI via `test-tutorials` label | |
| 32 | +| V. Memory-Driven Anti-Pattern Avoidance | PASS | Checked memory for relevant warnings | |
| 33 | +| VI. No Unnecessary API Calls | PASS | Tutorials use local sample files; no redundant model downloads | |
| 34 | +| VII. Simplicity First | PASS | Tutorials use existing senselab APIs, no new abstractions | |
| 35 | +| VIII. No Hardcoded Parameters | PASS | Device auto-detected; sample files downloaded from URLs | |
| 36 | + |
| 37 | +## Project Structure |
| 38 | + |
| 39 | +### Documentation (this feature) |
| 40 | + |
| 41 | +```text |
| 42 | +specs/20260423-213942-pedagogical-tutorials/ |
| 43 | +├── plan.md # This file |
| 44 | +├── research.md # Phase 0 output |
| 45 | +├── data-model.md # Phase 1 output |
| 46 | +├── quickstart.md # Phase 1 output |
| 47 | +└── tasks.md # Phase 2 output (created by /speckit.tasks) |
| 48 | +``` |
| 49 | + |
| 50 | +### Source Code (repository root) |
| 51 | + |
| 52 | +```text |
| 53 | +tutorials/ |
| 54 | +├── audio/ |
| 55 | +│ ├── audio_recording_and_acoustic_analysis.ipynb # NEW Tutorial 1 |
| 56 | +│ ├── transcription_and_phonemic_analysis.ipynb # NEW Tutorial 2 |
| 57 | +│ ├── shbt205_lab.ipynb # NEW (adapted from course) |
| 58 | +│ ├── 00_getting_started.ipynb # UPDATED |
| 59 | +│ └── ... (existing tutorials unchanged) |
| 60 | +├── manifest.json # UPDATED (add 3 entries) |
| 61 | +
|
| 62 | +src/senselab/audio/tasks/features_extraction/ |
| 63 | +├── ppg.py # UPDATED (fresh phoneme duration implementation) |
| 64 | +└── __init__.py # UPDATED (phoneme duration exports) |
| 65 | +
|
| 66 | +src/tests/audio/tasks/ |
| 67 | +└── features_extraction_test.py # UPDATED (PR #431 tests) |
| 68 | +``` |
| 69 | + |
| 70 | +**Structure Decision**: No new modules or directories. Tutorials go in existing `tutorials/audio/`. Phoneme duration functions added to existing ppg.py. |
| 71 | + |
| 72 | +## Implementation Phases |
| 73 | + |
| 74 | +### Phase 1: Implement PPG phoneme duration analysis (fresh) |
| 75 | + |
| 76 | +**Goal**: Implement phoneme duration extraction and timeline plotting fresh on the current codebase, reviewing PR #431 for ideas but not merging it. |
| 77 | + |
| 78 | +1. Read PR #431 code thoroughly (`extract_mean_phoneme_durations`, `plot_ppg_phoneme_timeline`) |
| 79 | +2. Implement equivalent functionality cleanly against current senselab APIs in `ppg.py` |
| 80 | +3. Add exports to `__init__.py` |
| 81 | +4. Write tests in `features_extraction_test.py` |
| 82 | +5. Run tests locally: `uv run pytest src/tests/audio/tasks/features_extraction_test.py -v -k "ppg or phoneme"` |
| 83 | +6. Run pre-commit: `uv run pre-commit run --all-files` |
| 84 | +7. Push, verify CI passes, merge to alpha |
| 85 | +8. Close PR #431 as superseded |
| 86 | + |
| 87 | +### Phase 2: Create Tutorial 1 — Audio Recording & Acoustic Analysis |
| 88 | + |
| 89 | +**Goal**: New notebook `tutorials/audio/audio_recording_and_acoustic_analysis.ipynb` |
| 90 | + |
| 91 | +**Notebook structure** (sections with markdown + code cells): |
| 92 | + |
| 93 | +1. **Title & Overview** (markdown): What students will learn — recording/loading audio, visualization, pitch, formants, emotion, speaker matching |
| 94 | +2. **Install senselab** (code): `!pip install -q uv` + `!uv pip install --pre --system "senselab"` |
| 95 | +3. **Restart runtime admonition** (markdown) |
| 96 | +4. **Imports & device setup** (code): senselab imports, auto-detect device |
| 97 | +5. **Section: Record or Load Audio** (markdown + code): |
| 98 | + - Markdown: Explain what audio is (waveform, sampling rate) |
| 99 | + - Code: JS-based recording widget (conditional on Colab) OR download sample audio |
| 100 | + - Code: Load as `Audio` object, display properties |
| 101 | +6. **Section: Waveform Visualization** (markdown + code): |
| 102 | + - Markdown: What a waveform represents (amplitude over time) |
| 103 | + - Code: `plot_waveform(audio)`, `play_audio(audio)` |
| 104 | +7. **Section: Spectrogram** (markdown + code): |
| 105 | + - Markdown: What a spectrogram shows (frequency content over time), mel scale |
| 106 | + - Code: `plot_specgram(audio)`, `plot_specgram(audio, mel_scale=True)` |
| 107 | +8. **Section: Pitch Extraction** (markdown + code): |
| 108 | + - Markdown: What pitch (F0) is, how it relates to vocal fold vibration |
| 109 | + - Code: `extract_pitch_from_audios([audio])`, plot pitch contour with matplotlib |
| 110 | +9. **Section: Formant Extraction** (markdown + code): |
| 111 | + - Markdown: What formants are (F1, F2), relationship to vowel identity |
| 112 | + - Code: `extract_praat_parselmouth_features_from_audios([audio])`, display F1/F2 values |
| 113 | +10. **Section: Speech Emotion Recognition** (markdown + code): |
| 114 | + - Markdown: How machines detect emotion from acoustic cues |
| 115 | + - Code: `classify_emotions_from_speech([audio], model)`, display labels + scores |
| 116 | +11. **Section: Speaker Matching** (markdown + code): |
| 117 | + - Markdown: How speaker verification works (embeddings + cosine similarity) |
| 118 | + - Code: Load/record second audio, `verify_speaker([(audio1, audio2)])`, interpret score |
| 119 | + |
| 120 | +### Phase 3: Create Tutorial 2 — Transcription & Phonemic Analysis |
| 121 | + |
| 122 | +**Goal**: New notebook `tutorials/audio/transcription_and_phonemic_analysis.ipynb` |
| 123 | + |
| 124 | +**Notebook structure**: |
| 125 | + |
| 126 | +1. **Title & Overview** (markdown): ASR, forced alignment, PPG phoneme timeline |
| 127 | +2. **Install senselab** (code): `!pip install -q uv` + `!uv pip install --pre --system "senselab[nlp]"` |
| 128 | +3. **Restart runtime admonition** (markdown) |
| 129 | +4. **Imports & device setup** (code) |
| 130 | +5. **Section: Load Audio** (code): Download sample audio or use recording from Tutorial 1 |
| 131 | +6. **Section: Automatic Speech Recognition** (markdown + code): |
| 132 | + - Markdown: What ASR is, how Whisper works at a high level |
| 133 | + - Code: `transcribe_audios([audio], model=HFModel("openai/whisper-small"))`, display transcription |
| 134 | +7. **Section: Forced Alignment** (markdown + code): |
| 135 | + - Markdown: What forced alignment does (maps words/phones to time) |
| 136 | + - Code: `align_transcriptions([(audio, script, Language.EN)])`, display word and phone boundaries |
| 137 | + - Code: Visualize alignment on waveform (matplotlib overlay) |
| 138 | +8. **Section: Phonetic Posteriorgrams (PPGs)** (markdown + code): |
| 139 | + - Markdown: What PPGs are (probability distributions over phonemes per frame) |
| 140 | + - Code: `extract_ppgs_from_audios([audio])`, inspect shape and contents |
| 141 | +9. **Section: Phoneme Duration Analysis** (markdown + code): |
| 142 | + - Markdown: How to derive phoneme timing from PPGs |
| 143 | + - Code: `extract_mean_phoneme_durations(audio, ppg)`, display duration table |
| 144 | + - Code: `plot_ppg_phoneme_timeline(audio, ppg)`, interpret the timeline |
| 145 | + |
| 146 | +### Phase 4: Update Existing Course Notebooks |
| 147 | + |
| 148 | +**Goal**: Adapt `00_getting_started.ipynb` and `SHBT205-Lab.ipynb` for current senselab. |
| 149 | + |
| 150 | +**4a: Update 00_getting_started.ipynb** (already in `tutorials/`): |
| 151 | +- Verify install cell uses current convention (already done in prior PR) |
| 152 | +- Verify all API calls use current senselab functions |
| 153 | +- Minor: This notebook is already up to date from the prior tutorial fix PR |
| 154 | + |
| 155 | +**4b: Create shbt205_lab.ipynb** (new file adapted from course materials): |
| 156 | +- Start from `~/Downloads/drive-download-20260424T013242Z-3-001/SHBT205-Lab.ipynb` |
| 157 | +- Replace raw Whisper ASR → `transcribe_audios()` |
| 158 | +- Replace raw promonet PPG → `extract_ppgs_from_audios()` + PR #431 functions |
| 159 | +- Replace raw SPARC calls → `SparcFeatureExtractor.extract_sparc_features()` |
| 160 | +- Update install cell, add restart admonition, auto-detect device |
| 161 | +- Keep recording widget (conditional on Colab) |
| 162 | +- Add fallback sample audio download for CI |
| 163 | +- Clear all outputs |
| 164 | + |
| 165 | +### Phase 5: Manifest & CI Integration |
| 166 | + |
| 167 | +1. Add 3 new entries to `tutorials/manifest.json`: |
| 168 | + - `audio_recording_and_acoustic_analysis.ipynb` (timeout_cpu: 600, timeout_gpu: 300) |
| 169 | + - `transcription_and_phonemic_analysis.ipynb` (timeout_cpu: 1800, timeout_gpu: 600) |
| 170 | + - `shbt205_lab.ipynb` (timeout_cpu: 1800, timeout_gpu: 600) |
| 171 | +2. Test all notebooks locally via papermill |
| 172 | +3. Run pre-commit |
| 173 | +4. Push and verify CI with `test-tutorials` label |
| 174 | + |
| 175 | +## Risk Mitigation |
| 176 | + |
| 177 | +| Risk | Mitigation | |
| 178 | +|------|-----------| |
| 179 | +| PR #431 code outdated | Fresh implementation (Phase 1) avoids merge conflicts entirely | |
| 180 | +| Recording widget doesn't work in CI | All tutorials have downloaded sample audio fallback path | |
| 181 | +| PPG/SPARC subprocess venvs slow on CPU CI | Generous timeouts (1800s CPU) | |
| 182 | +| SHBT205-Lab SPARC voice conversion has no senselab wrapper | Document as known limitation; use SPARC feature extraction (which IS wrapped) and note voice conversion needs future API | |
| 183 | +| Colab JS recording approach varies across browsers | Provide clear instructions + sample audio fallback | |
| 184 | + |
| 185 | +## Complexity Tracking |
| 186 | + |
| 187 | +No constitution violations. All work uses existing senselab APIs and established tutorial patterns. |
0 commit comments