Skip to content

Commit 2588352

Browse files
satraclaude
satra
and
claude
authored
Rebuild SER tutorial: better models + text sentiment (#468)
* Rebuild SER tutorial: better models, text sentiment, interpretation Completely rewritten speech_emotion_recognition.ipynb: - 3 SER models compared: IEMOCAP-trained (superb, best for natural speech), RAVDESS-trained (ehcalabres), continuous (audeering) - Side-by-side bar chart comparison of all models - Interpretation guidance: why near-uniform scores occur, acted vs natural speech, relative ordering vs absolute values - Acted speech comparison (RAVDESS sample shows clear discrimination) - Text sentiment from transcription (whisper-turbo + cardiffnlp) - Comparison of acoustic emotion vs text sentiment - Practical guidance for applying to own data - Recording widget with sample audio fallback Research: surveyed INTERSPEECH/ICASSP 2024-2025, Papers with Code IEMOCAP leaderboard, HuggingFace model hub. emotion2vec is best overall but incompatible with HF pipeline; superb model is best compatible option. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add SpeechBrain SER backend + 4-model tutorial comparison New senselab capability: - classify_emotions_from_speech now accepts SpeechBrainModel - Auto-discovers module structure from model hyperparams.yaml - Creates dynamic Pretrained subclass with correct MODULES_NEEDED - Caches loaded models for reuse - Works with speechbrain/emotion-recognition-wav2vec2-IEMOCAP (4-class: neu/ang/hap/sad, highly confident scores) SER tutorial now shows 4 models: 1. superb/wav2vec2-base-superb-er (HF, IEMOCAP) 2. speechbrain/emotion-recognition-wav2vec2-IEMOCAP (SpeechBrain) 3. ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition (RAVDESS) 4. audeering continuous (valence/arousal/dominance) Added practical "process your audio through multiple models" example showing how to loop over models and display results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3764a5f commit 2588352

9 files changed

Lines changed: 935 additions & 5833 deletions

File tree

‎specs/20260424-152323-improve-ser-tutorial/checklists/requirements.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Specification Quality Checklist: Improve SER Tutorial
2+
3+
**Purpose**: Validate specification completeness and quality before proceeding to planning
4+
**Created**: 2026-04-24
5+
**Feature**: [spec.md](../spec.md)
6+
7+
## Content Quality
8+
9+
- [x] No implementation details (languages, frameworks, APIs)
10+
- [x] Focused on user value and business needs
11+
- [x] Written for non-technical stakeholders
12+
- [x] All mandatory sections completed
13+
14+
## Requirement Completeness
15+
16+
- [x] No [NEEDS CLARIFICATION] markers remain
17+
- [x] Requirements are testable and unambiguous
18+
- [x] Success criteria are measurable
19+
- [x] Success criteria are technology-agnostic (no implementation details)
20+
- [x] All acceptance scenarios are defined
21+
- [x] Edge cases are identified
22+
- [x] Scope is clearly bounded
23+
- [x] Dependencies and assumptions identified
24+
25+
## Feature Readiness
26+
27+
- [x] All functional requirements have clear acceptance criteria
28+
- [x] User scenarios cover primary flows
29+
- [x] Feature meets measurable outcomes defined in Success Criteria
30+
- [x] No implementation details leak into specification
31+
32+
## Notes
33+
34+
- All items pass validation. Spec is ready for `/speckit.plan`.
35+
- The core problem is model selection — the current model produces near-uniform scores on non-acted speech.
36+
- Text sentiment adds a complementary linguistic dimension to the acoustic emotion analysis.

‎specs/20260424-152323-improve-ser-tutorial/data-model.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Data Model: Improve SER Tutorial
2+
3+
**Date**: 2026-04-24
4+
5+
No new data models needed. The tutorial uses existing senselab entities.
6+
7+
## Existing Entities Used
8+
9+
### AudioClassificationResult
10+
- **labels**: List[str] — emotion class names, sorted by score descending
11+
- **scores**: List[float] — confidence scores, sorted descending
12+
- **top_label()**: Returns highest-confidence label
13+
- **top_score()**: Returns highest confidence score
14+
15+
### Audio
16+
- Standard senselab Audio object loaded from recording or file
17+
18+
### HFModel
19+
- Model specification for HuggingFace models
20+
21+
## Models Used in Tutorial
22+
23+
| Model | Type | Training Data | Best For |
24+
|-------|------|--------------|----------|
25+
| superb/wav2vec2-base-superb-er | Discrete | IEMOCAP | Conversational speech |
26+
| ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition | Discrete | RAVDESS | Acted speech |
27+
| audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim | Continuous | MSP-Podcast | Dimensional emotion |
28+
| cardiffnlp/twitter-roberta-base-sentiment-latest | Text | Twitter | Text sentiment |

‎specs/20260424-152323-improve-ser-tutorial/plan.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Implementation Plan: Improve SER Tutorial
2+
3+
**Branch**: `20260424-152323-improve-ser-tutorial` | **Date**: 2026-04-24 | **Spec**: [spec.md](spec.md)
4+
**Input**: Feature specification from `/specs/20260424-152323-improve-ser-tutorial/spec.md`
5+
6+
## Summary
7+
8+
Update the speech emotion recognition tutorial to use better-performing SER models that produce meaningful (not near-uniform) emotion scores, compare multiple models side-by-side, and add text-based sentiment analysis from transcription as a complementary signal.
9+
10+
## Technical Context
11+
12+
**Language/Version**: Python 3.11-3.12 (Colab uses 3.12)
13+
**Primary Dependencies**: senselab, transformers (for text sentiment pipeline)
14+
**Storage**: N/A
15+
**Testing**: papermill for notebook execution
16+
**Target Platform**: Google Colab
17+
**Project Type**: Tutorial notebook update
18+
**Performance Goals**: Tutorial completes in <10 minutes on CPU
19+
**Constraints**: Must work on CPU; models must be freely available on HuggingFace
20+
**Scale/Scope**: 1 notebook update
21+
22+
## Constitution Check
23+
24+
| Principle | Status | Notes |
25+
|-----------|--------|-------|
26+
| I. UV-Managed Python | PASS | Tutorial uses uv install in Colab |
27+
| II. Encapsulated Testing | PASS | CI via papermill |
28+
| III. Commit Early and Often | PASS | Single notebook change |
29+
| IV. CI Must Stay Green | PASS | Tested via papermill locally + CI |
30+
| V. Memory-Driven Anti-Pattern Avoidance | PASS | Addresses user feedback directly |
31+
| VI. No Unnecessary API Calls | PASS | Models cached after first download |
32+
| VII. Simplicity First | PASS | Uses existing senselab API + direct transformers for text sentiment |
33+
| VIII. No Hardcoded Parameters | PASS | Device auto-detected |
34+
35+
## Project Structure
36+
37+
### Source Code
38+
39+
```text
40+
tutorials/
41+
└── audio/
42+
└── speech_emotion_recognition.ipynb # UPDATED
43+
```
44+
45+
**Structure Decision**: Single notebook update. No new senselab modules — text sentiment uses transformers pipeline directly in the notebook since adding a full senselab text classification API is unnecessary for one tutorial section.
46+
47+
## Implementation Phases
48+
49+
### Phase 1: Rebuild SER Tutorial
50+
51+
**Notebook structure**:
52+
53+
1. **Install + restart** (standard pattern)
54+
2. **Imports + device**
55+
3. **Record or load audio** (recording widget + sample fallback)
56+
4. **Section: Speech Emotion Recognition with Multiple Models**
57+
- Load 3 models (IEMOCAP-trained, RAVDESS-trained, continuous)
58+
- Run all three on the same audio
59+
- Side-by-side comparison table + bar chart visualization
60+
- Explain which model suits which use case
61+
5. **Section: Understanding Emotion Scores**
62+
- Explain near-uniform distributions
63+
- Show acted vs natural speech examples (RAVDESS sample vs user audio)
64+
- Guidance on interpreting scores (relative ordering > absolute values)
65+
6. **Section: Text Sentiment from Transcription**
66+
- Transcribe audio with whisper-large-v3-turbo
67+
- Run text sentiment (cardiffnlp/twitter-roberta-base-sentiment-latest)
68+
- Compare acoustic emotion vs text sentiment
69+
- Explain when they agree and disagree
70+
7. **Section: Apply to Your Own Data**
71+
- How to batch-process multiple files
72+
- How to choose the right model
73+
8. **Summary**
74+
75+
### Phase 2: Test and Push
76+
77+
1. Test with papermill locally
78+
2. Run pre-commit
79+
3. Push to branch, create PR with `test-tutorials` label
80+
4. Verify CI passes
81+
5. Merge to alpha then main
82+
83+
## Risk Mitigation
84+
85+
| Risk | Mitigation |
86+
|------|-----------|
87+
| superb/wav2vec2-base-superb-er may not work with senselab's API | Test locally first; the model uses standard HF pipeline format |
88+
| Text sentiment model is large | Use a small RoBERTa-base model (~500MB), not a large model |
89+
| CI timeout with multiple model downloads | Generous timeout (1200s CPU) |

‎specs/20260424-152323-improve-ser-tutorial/quickstart.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Quickstart: Improve SER Tutorial
2+
3+
## Test the updated tutorial
4+
5+
```bash
6+
export HF_TOKEN=<your-token>
7+
uv run papermill tutorials/audio/speech_emotion_recognition.ipynb /dev/null --cwd . -k python3 --execution-timeout 1200
8+
```
9+
10+
## Key files
11+
12+
| File | Purpose |
13+
|------|---------|
14+
| `tutorials/audio/speech_emotion_recognition.ipynb` | Updated SER tutorial |

‎specs/20260424-152323-improve-ser-tutorial/research.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Research: Improve SER Tutorial
2+
3+
**Date**: 2026-04-24
4+
5+
## R1: Best SER Models for Non-Acted Speech
6+
7+
**Decision**: Use multiple models to show the range of behavior:
8+
1. `superb/wav2vec2-base-superb-er` — trained on IEMOCAP (semi-natural conversational speech), better for real-world audio
9+
2. `ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition` — trained on RAVDESS (acted speech), good for dramatic emotions
10+
3. `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim` — continuous SER (valence/arousal/dominance), dimensional approach
11+
12+
**Rationale**: The user's near-uniform scores (~0.12 per class) indicate the current model doesn't discriminate well on natural speech. Models trained on conversational data (IEMOCAP) generalize better. Showing multiple models teaches users to match the model to their data type.
13+
14+
**Verified locally**:
15+
- `superb/wav2vec2-base-superb-er`: WORKS — produces clear discrimination (0.67 neutral vs 0.32 happy on test audio)
16+
- `ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition`: WORKS — confirms near-uniform (~0.127) on same audio
17+
- `audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim`: WORKS — runs in subprocess venv
18+
- `speechbrain/emotion-recognition-wav2vec2-IEMOCAP`: WORKS with custom Pretrained subclass (model defines modules `wav2vec2, avg_pool, output_mlp` but EncoderClassifier expects `compute_features, mean_var_norm, embedding_model, classifier`). Fix: create a custom `EmotionRecognizer(Pretrained)` with correct MODULES_NEEDED. 4-class: neu/ang/hap/sad. Produces very confident scores (1.0 neutral on casual speech)
19+
- `emotion2vec/emotion2vec_plus_large`: FAILS — no model_type in config.json, needs custom API (not HF pipeline)
20+
21+
**Literature survey** (INTERSPEECH/ICASSP/ACL 2024-2025):
22+
- emotion2vec (ACL 2024): Best overall SER model, but uses custom API — not compatible with senselab's HF pipeline
23+
- EmoBox (2024): Multilingual SER toolkit, benchmarks multiple models
24+
- Papers with Code IEMOCAP leaderboard: Various approaches, 73-82% UAR typical for audio-only models
25+
- Key insight: Self-supervised models (wav2vec2, HuBERT, WavLM) dominate leaderboards
26+
27+
**Alternatives considered**:
28+
- Single "best" model: Doesn't teach model selection
29+
- Fine-tuning on user data: Out of scope for a tutorial
30+
- emotion2vec: Best SER model but requires custom API, not compatible with senselab's classify_emotions_from_speech
31+
- SpeechBrain IEMOCAP model: Incompatible with current version
32+
33+
## R2: Text Sentiment Analysis
34+
35+
**Decision**: Use HuggingFace `text-classification` pipeline directly in the tutorial (no new senselab API needed). Model: `cardiffnlp/twitter-roberta-base-sentiment-latest` (3-class: positive/negative/neutral).
36+
37+
**Rationale**: Text sentiment is a simple pipeline call. Adding a full senselab API for text classification is overkill for one tutorial section — it can be done directly with the transformers library. The tutorial already installs transformers via senselab dependencies.
38+
39+
**Alternatives considered**:
40+
- Add senselab text classification API: More engineering than needed for the tutorial
41+
- Use sentence-transformers for sentiment: Not designed for classification
42+
- Use a large LLM for sentiment: Overkill, slow, requires API keys
43+
44+
## R3: Tutorial Structure
45+
46+
**Decision**: Restructure the tutorial around practical use:
47+
1. Load your own audio (recording or upload)
48+
2. Run multiple SER models → comparison table
49+
3. Transcribe the audio (ASR)
50+
4. Run text sentiment on transcription
51+
5. Compare acoustic emotion vs text sentiment
52+
6. Guidance section on model selection and interpretation
53+
54+
**Rationale**: The current tutorial focuses on benchmarking accuracy on RAVDESS. While educational, users need to see how to apply SER to their own data and interpret results. The benchmark can be a secondary section.
55+
56+
## R4: Why Near-Uniform Scores Occur
57+
58+
**Decision**: Add an explicit section explaining this phenomenon:
59+
- Models trained on acted speech (RAVDESS) expect exaggerated emotional expression
60+
- Natural speech has subtler cues, so softmax distributes probability more evenly
61+
- The absolute scores matter less than the relative ordering and margin
62+
- Some models are simply not good at certain speech types
63+
64+
**Rationale**: This directly addresses the user feedback. Without this explanation, users lose trust in the tool.

‎specs/20260424-152323-improve-ser-tutorial/spec.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Feature Specification: Improve SER Tutorial with Better Models and Text Sentiment
2+
3+
**Feature Branch**: `20260424-152323-improve-ser-tutorial`
4+
**Created**: 2026-04-24
5+
**Status**: Draft
6+
**Input**: User feedback: SER tutorial produces near-uniform probability distributions (~0.12 each class) that don't meaningfully discriminate emotions. The current model gives poor signal on real-world (non-acted) speech. Tutorial needs better models and should also add text-based sentiment detection from transcription.
7+
8+
## User Scenarios & Testing *(mandatory)*
9+
10+
### User Story 1 - Researcher Gets Meaningful Emotion Scores from Own Audio (Priority: P1)
11+
12+
A researcher loads their own voice recording into the SER tutorial and runs emotion classification. They receive emotion probability scores where the dominant emotion is clearly distinguishable (e.g., highest score >0.3 vs others <0.15), rather than near-uniform distributions (~0.12 across all classes). The tutorial explains how to interpret the scores and which models perform best on different types of speech (acted vs. natural).
13+
14+
**Why this priority**: This is the core problem — users are getting uninformative results. Without meaningful emotion discrimination, the tutorial fails its educational purpose and provides no practical value for researchers analyzing voice recordings.
15+
16+
**Independent Test**: Run the tutorial with a clearly emotional audio sample (e.g., from RAVDESS acted speech dataset). The top emotion score should be at least 2x higher than the average of other scores.
17+
18+
**Acceptance Scenarios**:
19+
20+
1. **Given** an audio file with clearly expressed emotion, **When** the user runs the best-performing SER model, **Then** the predicted emotion matches the expressed emotion with a confidence score at least 2x higher than the mean of other classes.
21+
2. **Given** a natural (non-acted) speech recording, **When** the user runs the SER model, **Then** the tutorial explains that natural speech produces less dramatic score differences than acted speech, and provides guidance on interpretation.
22+
3. **Given** the tutorial runs on RAVDESS test data, **When** using the recommended model, **Then** accuracy exceeds 70% (compared to ground truth labels).
23+
24+
---
25+
26+
### User Story 2 - User Runs Multiple SER Models for Comparison (Priority: P2)
27+
28+
A user runs the same audio through multiple SER models to compare their strengths. The tutorial shows at least 2-3 models with different characteristics (e.g., one trained on acted speech, one on conversational speech) and presents results side-by-side so the user can choose the most appropriate model for their use case.
29+
30+
**Why this priority**: Different models perform differently on different types of speech. Showing multiple models helps users understand the landscape and make informed choices for their own data.
31+
32+
**Independent Test**: Run the tutorial with sample audio through all listed models and display a comparison table of scores.
33+
34+
**Acceptance Scenarios**:
35+
36+
1. **Given** an audio file, **When** the user runs all recommended models, **Then** they see a side-by-side comparison table of emotion scores from each model.
37+
2. **Given** the comparison table, **When** the user examines the results, **Then** the tutorial explains why different models may disagree and which is most appropriate for their data type.
38+
39+
---
40+
41+
### User Story 3 - User Gets Sentiment from Transcribed Text (Priority: P2)
42+
43+
A user transcribes their audio using ASR, then runs text-based sentiment analysis on the transcription. This complements the acoustic-based SER by providing linguistic sentiment (positive/negative/neutral) from the words spoken, not just how they were spoken.
44+
45+
**Why this priority**: Emotion from voice (paralinguistic) and sentiment from text (linguistic) are complementary signals. A user saying "I'm fine" in a sad voice shows neutral text sentiment but sad vocal emotion. Both perspectives together give a richer picture.
46+
47+
**Independent Test**: Run the tutorial with sample audio, get transcription, and see sentiment scores that reflect the textual content.
48+
49+
**Acceptance Scenarios**:
50+
51+
1. **Given** an audio file, **When** the user runs ASR followed by text sentiment analysis, **Then** they see sentiment labels (positive/negative/neutral) with confidence scores.
52+
2. **Given** both acoustic emotion and text sentiment results, **When** displayed together, **Then** the tutorial explains how they can agree or diverge and what that means.
53+
54+
---
55+
56+
### Edge Cases
57+
58+
- What happens when the audio is very short (<1 second)? (Some models may not produce reliable results; the tutorial should warn about minimum duration.)
59+
- What happens when the audio contains multiple emotions within one clip? (The model averages over the whole clip; the tutorial should mention segmentation for longer recordings.)
60+
- What happens when the transcription is empty or the model fails to transcribe? (Text sentiment falls back gracefully with a message.)
61+
- What happens with non-English audio? (The tutorial should note which models support which languages.)
62+
63+
## Requirements *(mandatory)*
64+
65+
### Functional Requirements
66+
67+
- **FR-001**: The tutorial MUST use at least one SER model that produces clearly differentiated emotion scores (not near-uniform) on both acted and natural speech.
68+
- **FR-002**: The tutorial MUST compare at least 2-3 SER models and present results in a side-by-side table, explaining the strengths and weaknesses of each.
69+
- **FR-003**: The tutorial MUST include a section on text-based sentiment analysis from transcribed speech, showing how linguistic sentiment complements acoustic emotion.
70+
- **FR-004**: The tutorial MUST include guidance on interpreting emotion scores: what "good" discrimination looks like, why near-uniform scores indicate poor model fit, and how acted vs. natural speech affects results.
71+
- **FR-005**: The tutorial MUST allow users to load their own audio file (recording or upload) and run all analyses on it.
72+
- **FR-006**: The tutorial MUST show emotion scores as a clear visualization (bar chart with labeled emotions and scores), not just raw numbers.
73+
- **FR-007**: The tutorial MUST follow established senselab tutorial conventions (install cell, restart admonition, device auto-detect, recording widget with sample fallback).
74+
75+
### Key Entities
76+
77+
- **Audio**: The speech recording being analyzed.
78+
- **Emotion Score**: A probability distribution over emotion classes (e.g., happy, sad, angry, neutral, fearful, surprised, calm, disgust) from acoustic analysis.
79+
- **Sentiment Score**: A classification (positive/negative/neutral) with confidence from text-based analysis of the transcription.
80+
- **SER Model**: A pre-trained model that maps audio features to emotion probabilities.
81+
- **Transcription**: The text output from automatic speech recognition.
82+
83+
## Success Criteria *(mandatory)*
84+
85+
### Measurable Outcomes
86+
87+
- **SC-001**: On RAVDESS acted speech samples, the best recommended SER model achieves >70% accuracy and the top emotion score is at least 2x the mean of other classes.
88+
- **SC-002**: The tutorial presents at least 3 SER models with a comparison table showing scores on the same audio.
89+
- **SC-003**: Text sentiment analysis produces a clear positive/negative/neutral classification with confidence >0.5 on sample text.
90+
- **SC-004**: Users can load their own audio and see both acoustic emotion and text sentiment results within 5 minutes of the analysis cells.
91+
- **SC-005**: The tutorial includes at least one paragraph explaining why near-uniform scores occur and how to choose the right model for different speech types.
92+
93+
## Assumptions
94+
95+
- Users run the tutorial in Google Colab with access to a microphone or their own audio files.
96+
- SER models are available via HuggingFace and can run on CPU (GPU optional but faster).
97+
- Text sentiment analysis uses a pre-trained text classification model (no fine-tuning needed).
98+
- The tutorial updates the existing `speech_emotion_recognition.ipynb` rather than creating a new notebook.
99+
- The existing senselab `classify_emotions_from_speech()` API is used; model selection is the key improvement.
100+
- For text sentiment, the tutorial uses senselab's ASR for transcription and a standard text sentiment model for classification.

0 commit comments

Comments
 (0)