Conversation
* Rebuild SER tutorial: better models, text sentiment, interpretation Completely rewritten speech_emotion_recognition.ipynb: - 3 SER models compared: IEMOCAP-trained (superb, best for natural speech), RAVDESS-trained (ehcalabres), continuous (audeering) - Side-by-side bar chart comparison of all models - Interpretation guidance: why near-uniform scores occur, acted vs natural speech, relative ordering vs absolute values - Acted speech comparison (RAVDESS sample shows clear discrimination) - Text sentiment from transcription (whisper-turbo + cardiffnlp) - Comparison of acoustic emotion vs text sentiment - Practical guidance for applying to own data - Recording widget with sample audio fallback Research: surveyed INTERSPEECH/ICASSP 2024-2025, Papers with Code IEMOCAP leaderboard, HuggingFace model hub. emotion2vec is best overall but incompatible with HF pipeline; superb model is best compatible option. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add SpeechBrain SER backend + 4-model tutorial comparison New senselab capability: - classify_emotions_from_speech now accepts SpeechBrainModel - Auto-discovers module structure from model hyperparams.yaml - Creates dynamic Pretrained subclass with correct MODULES_NEEDED - Caches loaded models for reuse - Works with speechbrain/emotion-recognition-wav2vec2-IEMOCAP (4-class: neu/ang/hap/sad, highly confident scores) SER tutorial now shows 4 models: 1. superb/wav2vec2-base-superb-er (HF, IEMOCAP) 2. speechbrain/emotion-recognition-wav2vec2-IEMOCAP (SpeechBrain) 3. ehcalabres/wav2vec2-lg-xlsr-en-speech-emotion-recognition (RAVDESS) 4. audeering continuous (valence/arousal/dominance) Added practical "process your audio through multiple models" example showing how to loop over models and display results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the Speech Emotion Recognition (SER) tutorial with better models and text sentiment analysis, supported by a new backend for SpeechBrain models in the core API. The review feedback highlights several opportunities to improve the robustness of the SpeechBrain implementation, specifically by including model revisions in cache keys and downloads to avoid stale data, using standard YAML parsing for configuration files, and documenting or generalizing the architectural assumptions made during model inference.
| user_preference=device, compatible_devices=[DeviceType.CUDA, DeviceType.CPU] | ||
| ) | ||
|
|
||
| key = f"{model.path_or_uri}-{device_type.value}" |
There was a problem hiding this comment.
The cache key for _speechbrain_ser_models should include the model revision to ensure that different versions of the same model are cached separately. Otherwise, switching revisions for the same model path might return a stale cached model from the dictionary.
| key = f"{model.path_or_uri}-{device_type.value}" | |
| key = f"{model.path_or_uri}-{model.revision}-{device_type.value}" |
| from speechbrain.inference import Pretrained | ||
|
|
||
| # Download hyperparams to discover MODULES_NEEDED | ||
| hp_path = hf_hub_download(str(model.path_or_uri), "hyperparams.yaml") |
There was a problem hiding this comment.
The hf_hub_download call should specify the revision to ensure the correct version of the hyperparams.yaml file is retrieved, matching the model's specified revision.
| hp_path = hf_hub_download(str(model.path_or_uri), "hyperparams.yaml") | |
| hp_path = hf_hub_download(str(model.path_or_uri), "hyperparams.yaml", revision=model.revision) |
| with open(hp_path) as f: | ||
| content = f.read() | ||
|
|
||
| # Extract MODULES_NEEDED from the yaml | ||
| modules_needed = None | ||
| for line in content.split("\n"): | ||
| if "MODULES_NEEDED" in line: | ||
| # Parse: MODULES_NEEDED: ["wav2vec2", "avg_pool", "output_mlp"] | ||
| import ast | ||
|
|
||
| _, _, value = line.partition(":") | ||
| modules_needed = ast.literal_eval(value.strip()) | ||
| break |
There was a problem hiding this comment.
Parsing hyperparams.yaml with manual string splitting and ast.literal_eval is fragile and assumes the MODULES_NEEDED list is defined on a single line without comments. Since speechbrain is a dependency (which depends on PyYAML), it is safer and more robust to use yaml.safe_load to parse the configuration file.
| with open(hp_path) as f: | |
| content = f.read() | |
| # Extract MODULES_NEEDED from the yaml | |
| modules_needed = None | |
| for line in content.split("\n"): | |
| if "MODULES_NEEDED" in line: | |
| # Parse: MODULES_NEEDED: ["wav2vec2", "avg_pool", "output_mlp"] | |
| import ast | |
| _, _, value = line.partition(":") | |
| modules_needed = ast.literal_eval(value.strip()) | |
| break | |
| import yaml | |
| with open(hp_path) as f: | |
| hp_data = yaml.safe_load(f) or {} | |
| modules_needed = hp_data.get("MODULES_NEEDED") |
| for mod_name in recognizer.MODULES_NEEDED: | ||
| mod = getattr(recognizer.mods, mod_name) | ||
| if mod_name == recognizer.MODULES_NEEDED[0]: | ||
| # First module (feature extractor) gets wavs + lens | ||
| out = mod(wavs, wav_lens) | ||
| elif "pool" in mod_name: | ||
| out = mod(out, wav_lens) | ||
| out = out.view(out.shape[0], -1) | ||
| else: | ||
| out = mod(out) |
There was a problem hiding this comment.
The module iteration logic is highly specific to certain SpeechBrain model architectures (e.g., assuming the first module is the feature extractor and that 'pool' modules require wav_lens). This may not be compatible with all SpeechBrain SER models. Consider documenting these architectural assumptions or implementing a more flexible way to handle different model structures to improve the robustness of the SpeechBrain backend.
1 participant
SpeechBrain SER backend, 4-model comparison tutorial, text sentiment, interpretation guidance.