XTTS v2 - please help out a noob

[andykaufseo]

I have a long list of questions, recently starting using coqui (xtts v2 cloned voice), i have some experience with LLMs and more experience Stable Diffusion, but i'm just getting started with TTS (and STT). Here are my questions:

1. i tested xtts v2 with a cloned voice, it sounds great - a bit slow tho but no biggie - question is, if this sounds so good, why is there a fine tuning setup for this? i mean, do i fine tune the base with the cloned voice - to sound even more like the cloned voice? if that's the case, if i clone another voice (while using the fine tuned model with the first cloned voice, what will happen)

2. as of today, which is the best model for cloning voices? xtts2 sounds great, i remember trying bark a few months ago when it got out and it was a mess, so was vits (code was messed up, couldn't get it to work and the demo was bad).

3. i see that with cloned voices, xtts2 is a bit slow (not a pain in the ass, but could be faster). i see some things that claim to do almost real time voice gen (https://github.com/KoljaB/RealtimeTTS) - is this possible with xtts2?

4. i remember Bark could insert things like laughter, music, etc - could i use that in combination with xtts2

[m000lie]

the finetuning setup is meant for you to further refined the base model when cloning your target voice. there are some instances where the base model doesn't perform to it's best on zero shot inference (unseen data).

as such, fine tuning is still necessary for a subset of use-cases.

[guptaprakhariitr]

Hi @andykaufseo, If you've found answers to these, please update them here. Would like to hear from your experience.

[MethanJess]

@andykaufseo

1. Fine-tuning a model can get your voice to sound more like the target voice, the 0-shot cloning is not perfect, I have noticed a massive difference between a finetuned model and a 0-shot one.
2. The best model i would definitely say is the one used on myvoice.speechfiy, it can clone even the most unique voices, and gives out very realistic voices that almost perfectly match the original audio, and can do very well even on a 1 minute sample, sadly it's locked behind a very hefty subscription, a free one is play.ht, but it is not as good, for open source ones, Metavoice and VoiceCraft are very good.
3. Yeah, that repository seems to be legit from just looking at it...
4. I could be wrong, but I believe that if you fine tune your own model and add laughter sounds in the sample then put in the transcript something like "[laughter]" then I think the model will be output a laughter sound each time you input "[laughter]", but not sure if there's a way currently to combine bark and coqui together...

[jash2129]

Does it have Telugu(Indian language) library.

[MethanJess]

@jash2129 not sure.... coqui supports 16 langues, but i doubt that Telugu is one of them...
You can try other open source projects like this one: https://github.com/JarodMica/ai-voice-cloning
I heard it can train on any language or something like that

[AjibolaPy]

I tried xtts base model with my own voice it got it quite close but the accent was very different. Will fine-tuning get my voice and accent correct, or close atleast?, And in finetuning do I need sample of only my voice or different voice with same accent

[Deathgod999]

Hello everyone anyone pls give me s code how to use xtts v2 model 😁

[Deathgod999]

I always face some error, sometimes about the lang and sometimes about something else. Please can I get the update code for 2025 now?

[ClaudeRobbinCR]

Hi! Does coqui have a future?
I am looking for a reliable web ui!

[eginhard]

We'll continue maintaining the fork, yes, and that webui is already using it.

[ClaudeRobbinCR]

Yes. But will there be major releases?

Like v3 for example.

I want to helo but I'm only a student and my knowledge isn't deep enough.

If we were to work on v3 how would it be done?

1. We need to solve the acronym issue.

2. We need the a single language targeted model that NEVER stumbles in any pronunciation.

Can I do this as a solo dev?

[eginhard]

We're mainly maintaining the code, but sometimes new models are added, see the release notes for more details: https://github.com/idiap/coqui-ai-TTS/releases

If by v3 you mean a new version of the XTTS model, then no. We don't have any affiliation with the original Coqui company, so don't have access to the full model weights and code needed for training.

1. We need to solve the acronym issue.

This is independent of the TTS model. You could use external libraries to convert acronyms/numbers to text first.

2. We need the a single language targeted model that NEVER stumbles in any pronunciation.

Coqui includes many single-language models (e.g. Vits) that don't make pronunciation errors like XTTS, although the output speech quality won't be as good.

[ClaudeRobbinCR]

Oh that's really sad.
The community was onto something big here. XTTS sometimes hallucinates and sounds like a human person it's scary.

How much funding would you need to continue this work/obtain the weights you would need to build a new version of XTTS?

Can we build a community that can come together and pick up where the devs left?

Also I noticed that you volunteered to maintain this repo. Thank you so much for doing this!

XTTS helps me earn in my side gigs as a student. Thank you so much. 🙏