Looking for guidance on speaker stability / steerability in cloned TTS

[EgilSandfeld]

First off, thank you for open-sourcing TADA.
We’ve been evaluating TADA for a race-engineer-style TTS use case where the main requirement is one stable cloned voice for short operational lines like:

• Listening.
• Good afternoon, Egil.
• Yellow ahead.
• Yeah, no... that was actually quite good.

Main issue

We were able to improve quality a lot, but we still see intermittent failures like:

• wrong-voice drift (different male voice, occasionally high-pitched/feminine lead-in)
• whispery / too-low-volume lead-ins
• occasional truncation of the last word(s)
• occasional background/static-like artifacts
  The problem is no longer “always bad” — it is now more like mostly good, but not deterministic enough.

What helped most

The biggest improvement was better prompt audio, more than most inference knobs.
We tested several neutral prompt samples:

• a more constrained, technical neutral prompt worked best
• more conversational prompt samples sometimes sounded nicer on individual lines, but reduced speaker identity stability overall
  So for us, the best prompt was the one that constrained the model most, not the one that sounded most conversational in isolation.

Best setup we found so far

On HumeAI/tada-1b, our best result was approximately:

python
InferenceOptions(
    noise_temperature=0.75,
    acoustic_cfg_scale=2.2,
    duration_cfg_scale=0.8,
    num_flow_matching_steps=8,
    num_acoustic_candidates=2,
    scorer="likelihood",
)

We also:

• pre-encoded prompts at startup
• unloaded the encoder afterward
• used torch.bfloat16

This improved both latency and output quality, but did not fully eliminate speaker drift.

What did not clearly solve it

• increasing model size to tada-3b-ml
• larger candidate counts
• more expressive / more conversational prompt samples
• higher flow matching steps alone

The 3B model was heavier/slower and still showed the same class of failures.

What we are asking

Do maintainers/community have recommendations for improving speaker stability / steerability specifically?

In particular:

1. Are there best practices for prompt clips that maximize identity consistency?
2. Should we be leaning harder on spkr_verification, and if so what spkr_verification_weight range is worth trying?
3. Are whispery / under-energized lead-ins on conversational phrases a known issue?
4. Is there any known way to reduce run-to-run voice drift further?
5. Is there anything about the prompt encoding/alignment step that is especially important beyond clean audio + exact transcript?

Short summary

TADA can sound excellent, and prompt quality matters a lot, but the remaining blocker for us is consistent voice identity across runs, not just raw fidelity.

Any guidance would be appreciated.

[trungd]

​Thank you for your interest in testing the model. While the model design should effectively mitigate content type of hallucinations, speaker switching remains a challenge, as the model has not yet undergone post-training on well diarized data. ​

To mitigate this, our technical report proposes using streaming rejection sampling. This currently serves as the most reliable method for steering the model, provided a robust scorer is used. We encourage you to experiment with this approach and share your findings. ​The provided speaker scorer head evaluates two acoustic features and generates a score indicating the likelihood that they originate from the same speaker. Because each feature corresponds to a full text token, the model typically has sufficient context to produce a reliable score. For improved performance, you may consider:

• Selection strategy: Choose the one with the highest score or choose randomly among those whose score passes a certain threshold.
• ​Training another standalone speaker scorer on a single-speaker dataset for higher reliability than what we provided.
• ​Training a verification scorer optimized for a specific voice.

​A simple MLP operating on a single vector should be sufficient for this network. This method should also mitigate speaker drifting which may be caused by the model not being able to recover from one bad sampling step. We can increase the number of samples in the flow matching head without much higher inference cost due to batching efficiency. The 3b model should be more stable and capable of generating samples >1m without noticeable drifting. For the 1b model we recommend short samples.

For Q5: ​The prompt should work provided the alignment is accurate and the input is free of strange words or symbols. But some prompts may work better than others so we recommend you try multiple ones.

[EgilSandfeld]

Thanks — we tested the speaker-verification / rejection-sampling direction you suggested and wanted to report back.

What we tried

We kept our best prompt/settings baseline and switched to a speaker-consistency selection path:

• prompt: our best neutral prompt (a constrained technical neutral sample)
• noise_temperature = 0.75
• num_flow_matching_steps = 8
• acoustic_cfg_scale = 2.2
• duration_cfg_scale = 0.8

Then we tested:

• scorer = "spkr_verification"
• num_acoustic_candidates = 6
• external rejection sampling:
  • accept only if max speaker score >= 0.45
  • retry up to 4 rounds per utterance

We tested repeated short/fragile lines like:

• Listening.
• Good afternoon, Egil.
• Yeah, no... that was actually quite good.
• I think that sounds a lot better now, um, especially in the middle.

What improved

This did help with the specific problem of obvious wrong-speaker drift.

Compared to our earlier runs, we saw much better speaker identity consistency overall. So the speaker-verification path was definitely useful.

What did NOT improve enough

The remaining failures were less about identity and more about delivery quality:

• low-volume / under-energized lead-ins
• occasional whispery / far-from-mic feeling on colloquial starts like Yeah, no...
• small end kinks or slight phasing/flutter on some runs
• some runs still felt off in delivery even when the speaker identity seemed correct

So our current interpretation is:

• spkr_verification + rejection sampling helps speaker identity
• but it does not reliably solve:
  • energy consistency
  • lead-in confidence
  • minor acoustic artifacts / phasing
  • prosody stability on hesitant conversational phrases

Interesting correlation

High speaker scores did not always mean the output was fully good.

We had cases where:

• score looked strong
• same speaker identity was preserved
• but the line still sounded low-energy / whispery / slightly phasy

So speaker verification seems directionally right, but not sufficient as the only quality gate.

Practical takeaway from our side

This was the most promising TADA mitigation so far for identity stability, but it still does not get us to fully deterministic production behavior for our use case.

At this point our best understanding is:

1. prompt quality matters a lot
2. speaker-verification-based selection helps identity drift
3. remaining issues are now more about prosody / energy / acoustic quality than gross speaker switching

Question

Does that align with your expectations for the current scorer head?

And if so, would your next recommendation be:

1. push harder on spkr_verification_weight
2. use a better custom speaker scorer
3. train a scorer for the target voice
4. something else for handling the low-energy / whispery lead-in behavior

Thanks again — your earlier guidance was definitely useful.

[thiswillbeyourgithub]

Related to #15