Qwen3-TTS Podcast Generator

Generate NotebookLM-style podcasts with voice cloning using Qwen3-TTS.

Features

• Voice cloning from short audio samples (~2-5 seconds)
• Natural prosody with tuned generation parameters
• Multi-speaker podcast generation
• Automatic audio preprocessing

Requirements

• Python 3.10+
• ffmpeg (for audio preprocessing)
• CUDA GPU recommended (also works on Apple Silicon MPS or CPU)

Setup

python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Usage

1. Prepare voice samples

Place your voice samples in the project root:

• nikhil_voice_sample.m4a (or any format ffmpeg supports)
• anup_voice_sample.opus

Each sample should be a short recording (~2-5 seconds) of the speaker saying:

"The quick brown fox jumped over the lazy dog."

Preprocess them:

python preprocess_audio.py

This creates normalized 24kHz mono WAV files for optimal voice cloning.

2. Edit the script

Modify podcast_script.py with your dialogue:

PODCAST_SCRIPT = [
    {"speaker": "nikhil", "text": "Welcome to the show!"},
    {"speaker": "anup", "text": "Great to be here."},
    # ...
]

REFERENCE_TEXT = "The quick brown fox jumped over the lazy dog."

3. Generate

python generate_podcast.py

Output goes to output/podcast_slm_2026.wav.

How It Works

1. Voice Clone Prompts: Creates speaker embeddings using ICL (in-context learning) mode, which captures voice characteristics from reference audio + transcript.

2. Generation: Uses the Qwen3-TTS Base model with tuned parameters:

   • temperature=1.0 for the main model
   • subtalker_temperature=1.1 for more natural prosody variation

3. Stitching: Concatenates segments with 0.4s pauses between speakers.

Model Architecture

Qwen3-TTS is a transformer-based TTS system built on Qwen2.5 LLM with:

• X-Vector: Speaker embedding extracted from reference audio
• Talker: Generates semantic/phonetic speech tokens
• Subtalker: Controls prosody, rhythm, and intonation
• Vocoder: Converts discrete tokens to audio waveforms

Gotchas & Lessons Learned

Voice Cloning vs Style Instructions: Pick One

Qwen3-TTS has three models:

• Base: Voice cloning via generate_voice_clone() - no style control
• CustomVoice: Style instructions via generate_custom_voice() - preset speakers only (Ryan, Aiden, etc.)
• VoiceDesign: Create voices from text descriptions

You cannot have both voice cloning AND style instructions. The Base model ignores the instruct parameter. The CustomVoice model doesn't support custom voice samples.

Voice Drift at Segment Start

The first few seconds of a segment may use a different voice than intended. Fix with a warmup technique:

# Prepend text from another segment, then trim the audio
warmup = "Some text the speaker says elsewhere. "
full_text = warmup + original_text
wavs, sr = tts.generate_voice_clone(text=full_text, ...)

# Trim the warmup portion
warmup_ratio = len(warmup.split()) / len(full_text.split())
trim_samples = int(len(audio) * warmup_ratio)
audio = audio[trim_samples:]

Complex Style Instructions Cause Gibberish

When using CustomVoice, keep instructions simple:

• Works: "Friendly and warm"
• Gibberish: "Enthusiastic with emphasis on 'absolutely wild', pause between points, build excitement"

The model struggles with multi-part instructions or specific word emphasis.

Generation Parameters Matter

We tested various settings. What worked best:

Parameter	Default	Our Setting	Effect
temperature	0.9	1.0	Slightly more variation
subtalker_temperature	0.9	1.1	Less robotic prosody
subtalker_top_k	50	80	More diverse rhythm patterns

Lower subtalker_temperature (0.7-0.8) sounds more robotic. Higher values (1.1-1.2) sound more natural but less consistent.

Audio Preprocessing

The model handles varying audio levels, but preprocessing helps:

• Trim silence from start/end
• Normalize loudness to -16 LUFS
• Convert to 24kHz mono WAV

The preprocess_audio.py script handles this via ffmpeg.

Speed Adjustment

Generated audio tends to be slower than natural speech. Use ffmpeg to speed up:

# 1.25x speed (recommended)
ffmpeg -i podcast.wav -filter:a 'atempo=1.25' podcast_fast.wav

# 1.5x speed
ffmpeg -i podcast.wav -filter:a 'atempo=1.5' podcast_faster.wav

ICL vs X-Vector Only Mode

Two voice cloning modes:

• x_vector_only_mode=False (ICL): Richer voice capture, but can drift
• x_vector_only_mode=True: More consistent, but sounds flatter

ICL mode uses both the audio embedding AND the reference text/audio tokens. X-vector mode only uses the speaker embedding.

Tips

• Longer reference audio (up to 10s) can improve voice capture
• Use x_vector_only_mode=True for more consistent but less expressive output
• Increase subtalker_temperature for more animated delivery
• Save individual segments to debug voice issues before concatenating

License

MIT