Skip to content

Commit 237f221

Browse files
satraclaude
satra
and
claude
authored
Tutorial 1: recording, pitch/formant overlay, speech enhancement (#462)
* Add recording widget, pitch/formant overlay, speech enhancement Tutorial 1 updates: - Recording widget (Colab JS) with sample audio fallback - Pitch and formant tracks overlaid on spectrogram - Speech enhancement before/after comparison (spectrogram + pitch) - Speaker verification: same-speaker (split recording) + different-speaker - Additional play_audio cells for loaded/processed audio - Ruff per-file-ignores expanded for notebook patterns Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Merge SHBT205 into Tutorial 2, recording widget, aligned PPG viz - Tutorial 1: recording widget, pitch/formant overlay on spectrogram, speech enhancement before/after, same/different speaker demo, play_audio cells for loaded/processed audio, fix unicode escapes - Tutorial 2: rebuilt with SPARC articulatory coding (merged from shbt205_lab), time-aligned PPG visualization (waveform + spectrogram + phoneme segments on shared time axis) - Remove shbt205_lab.ipynb (content merged into Tutorial 2) - Ruff per-file-ignores expanded for notebook patterns - Both tutorials pass papermill locally Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Lazy-load SQUIM models to avoid 388MB download on import torchaudio_squim.py was calling SQUIM_OBJECTIVE.get_model() and SQUIM_SUBJECTIVE.get_model() at module scope, triggering 28MB + 360MB downloads every time any features_extraction submodule was imported. Now uses lazy initialization — models are loaded only when the quality extraction functions are actually called. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add speech representations lab + SPARC decode/convert New tutorial: - speech_representations_lab.ipynb: full course lab comparing acoustic (PPG, pitch, loudness) vs articulatory (SPARC EMA) representations of speech, with resynthesis and voice conversion New senselab APIs: - SparcFeatureExtractor.decode_sparc_features(): resynthesize audio from articulatory features (subprocess venv) - SparcFeatureExtractor.convert_voice(): voice conversion between speakers (subprocess venv) Lab covers: - Part 1: Recording/loading two sentences - Part 2: Articulatory analysis (SPARC encode, EMA visualization, pitch/loudness, resynthesis, voice conversion) - Part 3: Acoustic analysis (pitch contour, PPG, phoneme durations, time-aligned visualization) - Part 4: Side-by-side comparison with discussion prompts All 38 cells pass via papermill locally. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix formatting in speech representations lab notebook Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Lab: replace observations/takeaways with student questions Key Observations → Questions for Analysis (6 guided questions asking students to uncover differences in pitch, interpretability, independence, content modification, resynthesis, and neural control). Summary: removed pre-formed conclusions, kept API reference table, added prompt for students to formulate their own conclusions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent d8d3b51 commit 237f221

9 files changed

Lines changed: 1505 additions & 721 deletions

File tree

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,7 @@ convention = "google"
177177
[tool.ruff.lint.per-file-ignores]
178178
"src/tests/**/*.py" = []
179179
# Notebooks: cells share state but ruff lints per-cell, causing false positives
180-
"tutorials/**/*.ipynb" = ["F821", "E402", "I001"]
180+
"tutorials/**/*.ipynb" = ["F821", "E402", "I001", "F811", "ANN001", "ANN201", "D103"]
181181

182182
[tool.ruff.format]
183183
quote-style = "double"

‎src/senselab/audio/tasks/features_extraction/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22

33
from .api import extract_features_from_audios # noqa: F401
44
from .ppg import extract_mean_phoneme_durations, plot_ppg_phoneme_timeline # noqa: F401
5+
from .sparc import SparcFeatureExtractor # noqa: F401

‎src/senselab/audio/tasks/features_extraction/sparc.py‎

Lines changed: 247 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
from senselab.audio.data_structures import Audio
1818
from senselab.audio.tasks.preprocessing import resample_audios
1919
from senselab.utils.data_structures import DeviceType, Language, _select_device_and_dtype, logger
20-
from senselab.utils.subprocess_venv import ensure_venv, parse_subprocess_result
20+
from senselab.utils.subprocess_venv import _clean_subprocess_env, ensure_venv, parse_subprocess_result
2121

2222
# SPARC venv specification
2323
_SPARC_VENV = "sparc"
@@ -89,6 +89,90 @@
8989
"""
9090

9191

92+
_DECODE_WORKER_SCRIPT = r"""
93+
import json
94+
import sys
95+
import traceback
96+
from pathlib import Path
97+
98+
import numpy as np
99+
import soundfile as sf
100+
import torch
101+
from sparc import load_model
102+
103+
args = json.loads(sys.stdin.read())
104+
language = args["language"]
105+
device = args["device"]
106+
output_dir = args["output_dir"]
107+
feature_dir = args["feature_dir"]
108+
109+
coder = load_model(language, device=device)
110+
111+
try:
112+
# Load features from numpy files
113+
ema = np.load(str(Path(feature_dir) / "ema.npy"))
114+
pitch = np.load(str(Path(feature_dir) / "pitch.npy"))
115+
loudness = np.load(str(Path(feature_dir) / "loudness.npy"))
116+
spk_emb = np.load(str(Path(feature_dir) / "spk_emb.npy"))
117+
118+
waveform = coder.decode(ema, pitch, loudness, spk_emb)
119+
120+
# Save output as FLAC
121+
out_path = str(Path(output_dir) / "decoded.flac")
122+
if isinstance(waveform, torch.Tensor):
123+
wav_np = waveform.detach().cpu().numpy().squeeze()
124+
else:
125+
wav_np = np.asarray(waveform).squeeze()
126+
sf.write(out_path, wav_np, coder.output_sr)
127+
128+
print(json.dumps({"output_path": out_path, "sample_rate": coder.output_sr}))
129+
except Exception as e:
130+
print(f"Error decoding SPARC features: {e}", file=sys.stderr)
131+
print(traceback.format_exc(), file=sys.stderr)
132+
print(json.dumps({"error": {"type": type(e).__name__, "message": str(e)}}))
133+
sys.exit(1)
134+
"""
135+
136+
_CONVERT_WORKER_SCRIPT = r"""
137+
import json
138+
import sys
139+
import traceback
140+
from pathlib import Path
141+
142+
import numpy as np
143+
import soundfile as sf
144+
import torch
145+
from sparc import load_model
146+
147+
args = json.loads(sys.stdin.read())
148+
language = args["language"]
149+
device = args["device"]
150+
output_dir = args["output_dir"]
151+
source_path = args["source_path"]
152+
target_path = args["target_path"]
153+
154+
coder = load_model(language, device=device)
155+
156+
try:
157+
waveform = coder.convert(src_wav=source_path, trg_wav=target_path)
158+
159+
# Save output as FLAC
160+
out_path = str(Path(output_dir) / "converted.flac")
161+
if isinstance(waveform, torch.Tensor):
162+
wav_np = waveform.detach().cpu().numpy().squeeze()
163+
else:
164+
wav_np = np.asarray(waveform).squeeze()
165+
sf.write(out_path, wav_np, coder.output_sr)
166+
167+
print(json.dumps({"output_path": out_path, "sample_rate": coder.output_sr}))
168+
except Exception as e:
169+
print(f"Error in SPARC voice conversion: {e}", file=sys.stderr)
170+
print(traceback.format_exc(), file=sys.stderr)
171+
print(json.dumps({"error": {"type": type(e).__name__, "message": str(e)}}))
172+
sys.exit(1)
173+
"""
174+
175+
92176
class SparcFeatureExtractor:
93177
"""A factory for managing feature extraction pipelines using SPARC."""
94178

@@ -206,3 +290,165 @@ def extract_sparc_features(
206290
features_list.append(features)
207291

208292
return features_list
293+
294+
@classmethod
295+
def decode_sparc_features(
296+
cls,
297+
features: Dict[str, torch.Tensor],
298+
lang: Optional[Language] = None,
299+
device: Optional[DeviceType] = None,
300+
) -> Audio:
301+
"""Resynthesize audio from SPARC articulatory features.
302+
303+
Takes the feature dict returned by ``extract_sparc_features`` and
304+
runs ``coder.decode(ema, pitch, loudness, spk_emb)`` in the isolated
305+
subprocess venv to produce a waveform.
306+
307+
Args:
308+
features: Feature dict with keys ``ema``, ``pitch``, ``loudness``,
309+
and ``spk_emb`` (torch.Tensor values, as returned by
310+
``extract_sparc_features``).
311+
lang: Language for the SPARC model. None means multi-language.
312+
device: Device to use (CUDA or CPU).
313+
314+
Returns:
315+
Resynthesized Audio object.
316+
"""
317+
device, _ = _select_device_and_dtype(
318+
user_preference=device, compatible_devices=[DeviceType.CUDA, DeviceType.CPU]
319+
)
320+
321+
if lang is None:
322+
used_language = "multi"
323+
elif lang.name == "english":
324+
used_language = "en+"
325+
else:
326+
raise ValueError(f"Language {lang.name} not supported. Supported: english or None (multi-language).")
327+
328+
required_keys = {"ema", "pitch", "loudness", "spk_emb"}
329+
missing = required_keys - set(features.keys())
330+
if missing:
331+
raise ValueError(f"Missing required feature keys: {missing}")
332+
333+
venv_dir = ensure_venv(_SPARC_VENV, _SPARC_REQUIREMENTS, python_version=_SPARC_PYTHON)
334+
python = str(venv_dir / "bin" / "python")
335+
336+
with tempfile.TemporaryDirectory(prefix="senselab-sparc-decode-") as tmpdir:
337+
tmp = Path(tmpdir)
338+
339+
# Serialize feature tensors as numpy files
340+
feature_dir = tmp / "features"
341+
feature_dir.mkdir()
342+
for key in required_keys:
343+
tensor = features[key]
344+
if isinstance(tensor, torch.Tensor):
345+
np.save(str(feature_dir / f"{key}.npy"), tensor.cpu().numpy())
346+
else:
347+
np.save(str(feature_dir / f"{key}.npy"), np.asarray(tensor))
348+
349+
input_json = json.dumps(
350+
{
351+
"language": used_language,
352+
"device": device.value,
353+
"output_dir": str(tmp),
354+
"feature_dir": str(feature_dir),
355+
}
356+
)
357+
358+
result = subprocess.run(
359+
[python, "-c", _DECODE_WORKER_SCRIPT],
360+
input=input_json,
361+
capture_output=True,
362+
text=True,
363+
timeout=600,
364+
env=_clean_subprocess_env(),
365+
)
366+
367+
output = parse_subprocess_result(result, "SPARC decode")
368+
369+
audio_path = output["output_path"]
370+
sample_rate = output["sample_rate"]
371+
logger.info("SPARC decode produced audio at %d Hz: %s", sample_rate, audio_path)
372+
373+
audio = Audio(filepath=audio_path)
374+
# Force lazy-load while the temp file still exists
375+
_ = audio.waveform
376+
return audio
377+
378+
@classmethod
379+
def convert_voice(
380+
cls,
381+
source_audio: Audio,
382+
target_audio: Audio,
383+
lang: Optional[Language] = None,
384+
device: Optional[DeviceType] = None,
385+
) -> Audio:
386+
"""Convert the voice of source audio to match the target speaker.
387+
388+
Uses ``coder.convert(src_wav, trg_wav)`` in the isolated subprocess
389+
venv. Both audios are saved as WAV files and passed by path.
390+
391+
Args:
392+
source_audio: Audio whose content to preserve.
393+
target_audio: Audio whose speaker identity to adopt.
394+
lang: Language for the SPARC model. None means multi-language.
395+
device: Device to use (CUDA or CPU).
396+
397+
Returns:
398+
Voice-converted Audio object.
399+
"""
400+
device, _ = _select_device_and_dtype(
401+
user_preference=device, compatible_devices=[DeviceType.CUDA, DeviceType.CPU]
402+
)
403+
404+
if lang is None:
405+
used_language = "multi"
406+
elif lang.name == "english":
407+
used_language = "en+"
408+
else:
409+
raise ValueError(f"Language {lang.name} not supported. Supported: english or None (multi-language).")
410+
411+
for label, audio in [("source", source_audio), ("target", target_audio)]:
412+
if audio.waveform.squeeze().dim() != 1:
413+
raise ValueError(f"Only mono audio is supported ({label}).")
414+
415+
venv_dir = ensure_venv(_SPARC_VENV, _SPARC_REQUIREMENTS, python_version=_SPARC_PYTHON)
416+
python = str(venv_dir / "bin" / "python")
417+
418+
with tempfile.TemporaryDirectory(prefix="senselab-sparc-convert-") as tmpdir:
419+
tmp = Path(tmpdir)
420+
421+
src_path = str(tmp / "source.wav")
422+
trg_path = str(tmp / "target.wav")
423+
source_audio.save_to_file(src_path, format="wav")
424+
target_audio.save_to_file(trg_path, format="wav")
425+
426+
input_json = json.dumps(
427+
{
428+
"language": used_language,
429+
"device": device.value,
430+
"output_dir": str(tmp),
431+
"source_path": src_path,
432+
"target_path": trg_path,
433+
}
434+
)
435+
436+
result = subprocess.run(
437+
[python, "-c", _CONVERT_WORKER_SCRIPT],
438+
input=input_json,
439+
capture_output=True,
440+
text=True,
441+
timeout=600,
442+
env=_clean_subprocess_env(),
443+
)
444+
445+
output = parse_subprocess_result(result, "SPARC convert")
446+
447+
audio_path = output["output_path"]
448+
sample_rate = output["sample_rate"]
449+
logger.info("SPARC convert produced audio at %d Hz: %s", sample_rate, audio_path)
450+
451+
audio = Audio(filepath=audio_path)
452+
# Force lazy-load while the temp file still exists
453+
_ = audio.waveform
454+
return audio

‎src/senselab/audio/tasks/features_extraction/torchaudio_squim.py‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
from typing import Any, Dict, List, Optional
55

66
import numpy as np
7+
import torch.nn
78

89
from senselab.audio.data_structures import Audio
910
from senselab.utils.data_structures import DeviceType, _select_device_and_dtype, logger
@@ -13,8 +14,24 @@
1314
if TORCHAUDIO_AVAILABLE:
1415
from torchaudio.pipelines import SQUIM_OBJECTIVE, SQUIM_SUBJECTIVE
1516

16-
objective_model = SQUIM_OBJECTIVE.get_model()
17-
subjective_model = SQUIM_SUBJECTIVE.get_model()
17+
_objective_model = None
18+
_subjective_model = None
19+
20+
21+
def _get_objective_model() -> torch.nn.Module:
22+
"""Lazily load the SQUIM objective quality model."""
23+
global _objective_model # noqa: PLW0603
24+
if _objective_model is None:
25+
_objective_model = SQUIM_OBJECTIVE.get_model()
26+
return _objective_model
27+
28+
29+
def _get_subjective_model() -> torch.nn.Module:
30+
"""Lazily load the SQUIM subjective quality model."""
31+
global _subjective_model # noqa: PLW0603
32+
if _subjective_model is None:
33+
_subjective_model = SQUIM_SUBJECTIVE.get_model()
34+
return _subjective_model
1835

1936

2037
def extract_objective_quality_features_from_audios(
@@ -53,7 +70,7 @@ def extract_objective_quality_features_from_audios(
5370
for audio in audios:
5471
audio_features = {}
5572
try:
56-
stoi, pesq, si_sdr = objective_model.to(device.value)(audio.waveform.to(device.value))
73+
stoi, pesq, si_sdr = _get_objective_model().to(device.value)(audio.waveform.to(device.value))
5774
audio_features["stoi"] = stoi.cpu().item()
5875
audio_features["pesq"] = pesq.cpu().item()
5976
audio_features["si_sdr"] = si_sdr.cpu().item()
@@ -104,7 +121,7 @@ def extract_subjective_quality_features_from_audios(
104121
for i, audio in enumerate(audios):
105122
audio_features = {}
106123
try:
107-
mos = subjective_model(audio.waveform, non_matching_references[i].waveform)
124+
mos = _get_subjective_model()(audio.waveform, non_matching_references[i].waveform)
108125
audio_features["mos"] = mos.item()
109126
except RuntimeError as e:
110127
audio_features["mos"] = np.nan

0 commit comments

Comments
 (0)