WhisperX Audio Pipeline

End-to-end audio processing pipeline: speech recognition and emotion analysis for Russian-language recordings. (Speaker diarization is wired in via Pyannote, but speaker-to-segment attribution is still WIP — see the Output Format note.)

Pipeline

Audio File (mp3/wav/m4a/...)
    │
    ▼
┌─────────────────────────┐
│  1. Transcription        │  WhisperX (large-v3)
│     Speech → Text        │  Character-level alignment
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  2. Diarization (WIP)    │  Pyannote 3.1 runs,
│     labels not attached  │  segment attribution: TODO
└──────────┬──────────────┘
           │
           ▼
┌─────────────────────────┐
│  3. Emotion Analysis     │  DUSHA HuBERT (5 emotions)
│     How they said it     │  Aniemore HuBERT (7 emotions)
└──────────┬──────────────┘
           │
           ▼
  Output: text / HTML / aligned

Output Format

[00:00:05.123 - 00:00:10.456][Позитивная][Счастье] -> Привет, как дела?
[00:00:10.789 - 00:00:15.321][Нейтральная][Нейтральная] -> Нормально, давай обсудим проект.

Each segment includes timestamps and emotion predictions from both models (DUSHA + Aniemore, Russian labels), and the transcribed text. Speaker labels are not emitted yet — diarization runs but speaker-to-segment attribution is WIP. HTML output adds color-coded emotions with an interactive legend.

Emotion Models

Model	Emotions	Source
DUSHA HuBERT	neutral, angry, positive, sad, other	xbgoose/hubert-speech-emotion-recognition-russian-dusha-finetuned
Aniemore HuBERT	anger, disgust, enthusiasm, fear, happiness, neutral, sadness	Aniemore/hubert-emotion-russian-resd

Components

File	Description
audio_pipeline.py	Main pipeline — transcription, diarization, emotion analysis
audio_converter.py	Audio format converter (any format → WAV 16kHz mono via ffmpeg)
emotion_xbgoose_05_prod.py	Standalone emotion analyzer for pre-existing transcriptions
check.py	Environment verification (Python, CUDA, ffmpeg, libraries)

Usage

# Process all audio files in a directory
python audio_pipeline.py --input-dir ./recordings --output-dir ./results

# Process a single file
python audio_pipeline.py --single-file meeting.mp3

# Transcription only (no emotion analysis)
python audio_pipeline.py --single-file call.wav --no-emotions

# Convert audio to WAV
python audio_converter.py recording.m4a -o ./wav_output

Requirements

• Python 3.10+
• CUDA-capable GPU (recommended, falls back to CPU)
• ffmpeg and ffprobe in PATH
• Hugging Face token for Pyannote diarization

Key Dependencies

whisperx
pyannote.audio
torch + torchaudio
transformers
soundfile
rich (optional, for terminal preview)

Tech Stack

Python WhisperX Pyannote PyTorch HuBERT DUSHA speaker diarization emotion analysis ffmpeg

License

MIT