XTTS_v2 model: How to reduce end-of-sentence hallucinations? How to achieve consistent results?

[Hexatona]

Hey there. I've written up a nice program that turns properly tagged text documents into multi-chaptered audiobook files, and while I've gotten used to the phenomenon, I feel like the random end of sentence hallucinations that pop up from time to time are the last big hurdle before it's a real success.

I have found a few bugs in how text is parsed and spoken, and found workarounds - but the random hallucinations I can't seem to find a pattern for.

For example, if you send in a text like this: " , and why not, I should know, shouldn't I?" when generation starts with a comma, and there's another comma, the text between the 2nd and 3rd commas is completely ignored.

I'm sure I'm not the only one who has noticed the phenomena of the end of sentence hallucinations though - does anyone have any tips to get more consistent output?

And while we'r eon the subject of consistent output, every once in a while you'll find a word the model pronounces differently on random occasions. I've found one of the worst offenders would be "Cave" sometimes its sounds perfect, other times it sounds like "Cavey" or "Cavave". Is there any way to kind of affix the model to, say, speak the same line in exactly the same way everytime?

[Hexatona]

Minor breakthrough! You can experiment with this too.

Repeatedly trying this sentence:

"Let me up"

Will result in a lot of hallucinations

Sentences ending in an alpha character, or alpha and spaces tend to mess up the worst.

"Let me up." or "Let me up "

However, if you end the sentence with an underscore, it seems to smarten up.

"Let me up_"

I tried the above experiment with all other kinds of punctiation like ! or ? or !?, and the hallucinations don't seem to happen then.

[Wazaki-Ou]

Allow me to join in this discussion please. While I have no issue in English, when I use Korean, the ending sometimes include some weird breathing sound and some "aah" "yee" or some gibberish that almost sounds like Chinese. Has anyone please figured out a way to prevent that? I also tried to save the chunks of audio (since I'm using it as a stream on multiple phrases/sentences, and the chunk with the weird sound is always saved separately at the end when it happens.

[Rimini312]

Same thing happens to me with the Spanish model — XTTS v2 adds "uh ohs" and words that don't exist in Spanish or any language, and sometimes the voice even mutates completely.

[Hexatona]

You know, I wonder if I should go back to the original XTTS model and see if it has those issues...

[saurabhthesuperhero]

@Rimini312 @Wazaki-Ou i exprienced same with hindi at start of sentence anything suggestiions ?

[venki-lfc]

I am getting similar gibberish sounds at the end of some sentences when I use German.

[Sambsop]

I post a similar question about that recently (with other query)
For your question:
Is there any way to kind of affix the model to, say, speak the same line in exactly the same way everytime?
Yes as far as my tests went (not expert here) You can fix the seed used for the prosody randomness in one simple line:
set_seed(your_reference_seed)
I quote:"It's crucial to call set_seed() at the very beginning of your script, even before you initialize the TTS object. This ensures that all subsequent operations that involve randomness are seeded consistently."
So same seed, same render audio. (again in my tests: language is cloned french so ...)
I've been doing tests with random seeds and the halucinations sometimes disappear but there are other glitches (fast noise peaks). so finally i get better samples but not completly cleaned. Also sometimes the best seed comes with a bizarre prosody so more testing.

[Rimini312]

Rn in using kokoro, have less options of voices but never fails, I'm not sure if supports French. U should check

[Sambsop]

Yes i use kokoro but only for english voices, the only french voice availabele: ff_siwiss is not great. Coqui cloned voices are really first class, thats why i'm looking to 'clean' the coqui's audios.
Thanks indeed, Rimini312

[Rimini312]

Yes, coqui, it’s really good. I like how it works. And it’s true that sometimes it has no flaws. But… the truth is that in my case it doesn’t work. Or, better said, it’s not worth it for me. If out of every 10 videos I use coqui in, I have to delete four because of weird noises or it getting confused, I’d rather use a voice that sounds a bit more robotic, but at least I know it won’t fail. If you manage to fix it, please let me know. Best regards.

[Hexatona]

Aaah I see. I shall have to try that out! That will probably be hugely helpful, thank you.

[KaushikNoukha]

Hi there everyone ive trained my xtts v2 finetuned model also tested it in the colab environment now ive hosted it in Huggingaface and tried ti use it in my local application i get errors like model warm up failed but the model is loading ,and failing due to warm up error C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\TTS\TTS\TTS
Successfully imported TTS from local repository (pip fallback disabled)
=== GET_TTS_SERVICE CALLED ===
=== CREATING NEW TTS SERVICE ===
=== TTS SERVICE INIT START ===
Initializing ULTRA-FAST Coqui TTS service...
Using device: cpu
[DIAG] TTS_LANGUAGE=en
[DIAG] TTS_SPEAKER_WAV resolved: C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\stt_server\Voice_samples_Gus_000_norm_fixed.wav
[TTS] Auto-downloading from Hugging Face: repo_id=kaushikaiml/Gus-finegrained rev=main files=(model.pth, config.json)
[TTS] Using HF-cached files:
model: C:\hf_cache\hub\models--kaushikaiml--Gus-finegrained\snapshots\a2045b4c938865cbdf46886cb9341658599906da\model.pth
config: C:\hf_cache\hub\models--kaushikaiml--Gus-finegrained\snapshots\a2045b4c938865cbdf46886cb9341658599906da\config.json
[TTS] Loading LOCAL checkpoint from environment paths...
[TTS] Effective model path: C:\hf_cache\hub\models--kaushikaiml--Gus-finegrained\snapshots\a2045b4c938865cbdf46886cb9341658599906da\model.pth
[TTS] Effective config path: C:\hf_cache\hub\models--kaushikaiml--Gus-finegrained\snapshots\a2045b4c938865cbdf46886cb9341658599906da\config.json
[TTS] Adjusted model path to directory for XTTS: C:\hf_cache\hub\models--kaushikaiml--Gus-finegrained\snapshots\a2045b4c938865cbdf46886cb9341658599906da

Using model: xtts
GPT2InferenceModel has generative capabilities, as prepare_inputs_for_generation is explicitly defined. However, it doesn't directly inherit from GenerationMixin. From 👉v4.50👈 onwards, PreTrainedModel will NOT inherit from GenerationMixin, and this model will lose the ability to call generate and other related functions.

If you're using trust_remote_code=True, you can get rid of this warning by loading the model with an auto class. See https://huggingface.co/docs/transformers/en/model_doc/auto#auto-classes
If you are the owner of the model architecture code, please modify your model class such that it inherits from GenerationMixin (after PreTrainedModel, otherwise you'll get an exception).
If you are not the owner of the model architecture class, please contact the model code owner to update it.
[DIAG] TTS python module path: C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\venv311\Lib\site-packages\TTS_init_.py
[DIAG] Active TTS source: LOCAL checkpoint
[DIAG] self.tts.model_name:
Primary TTS model loaded: xtts_local
Extensive model warm-up for millisecond response...
Text splitted to sentences.
['Hello']
C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\venv311\Lib\site-packages\torchaudio_backend\utils.py:213: UserWarning: In 2.9, this function's implementation will be changed to use torchaudio.load_with_torchcodecunder the hood. Some parameters like normalize, format, buffer_size, and backend will be ignored. We recommend that you port your code to rely directly on TorchCodec's decoder instead: https://docs.pytorch.org/torchcodec/stable/generated/torchcodec.decoders.AudioDecoder.html#torchcodec.decoders.AudioDecoder. warnings.warn( Text splitted to sentences. ['Hello'] Text splitted to sentences. ['Hello'] Warm-up failed for 'Hello': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Hi there'] Text splitted to sentences. ['Hi there'] Text splitted to sentences. ['Hi there'] Warm-up failed for 'Hi there': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Welcome'] Text splitted to sentences. ['Welcome'] Text splitted to sentences. ['Welcome'] Warm-up failed for 'Welcome': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Thank you'] Text splitted to sentences. ['Thank you'] Text splitted to sentences. ['Thank you'] Warm-up failed for 'Thank you': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Yes'] Text splitted to sentences. ['Yes'] Text splitted to sentences. ['Yes'] Warm-up failed for 'Yes': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['No'] Text splitted to sentences. ['No'] Text splitted to sentences. ['No'] Warm-up failed for 'No': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Okay'] Text splitted to sentences. ['Okay'] Text splitted to sentences. ['Okay'] Warm-up failed for 'Okay': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Sure'] Text splitted to sentences. ['Sure'] Text splitted to sentences. ['Sure'] Warm-up failed for 'Sure': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['How are you?'] Text splitted to sentences. ['How are you?'] Text splitted to sentences. ['How are you?'] Warm-up failed for 'How are you?': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ["What's up?"] Text splitted to sentences. ["What's up?"] Text splitted to sentences. ["What's up?"] Warm-up failed for 'What's up?': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Good morning'] Text splitted to sentences. ['Good morning'] Text splitted to sentences. ['Good morning'] Warm-up failed for 'Good morning': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Good evening'] Text splitted to sentences. ['Good evening'] Text splitted to sentences. ['Good evening'] Warm-up failed for 'Good evening': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['I understand'] Text splitted to sentences. ['I understand'] Text splitted to sentences. ['I understand'] Warm-up failed for 'I understand': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Please wait'] Text splitted to sentences. ['Please wait'] Text splitted to sentences. ['Please wait'] Warm-up failed for 'Please wait': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Processing'] Text splitted to sentences. ['Processing'] Text splitted to sentences. ['Processing'] Warm-up failed for 'Processing': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Complete'] Text splitted to sentences. ['Complete'] Text splitted to sentences. ['Complete'] Warm-up failed for 'Complete': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Error'] Text splitted to sentences. ['Error'] Text splitted to sentences. ['Error'] Warm-up failed for 'Error': Model is multi-lingual but nolanguageis provided. Text splitted to sentences. ['Success'] Text splitted to sentences. ['Success'] Text splitted to sentences. ['Success'] Warm-up failed for 'Success': Model is multi-lingual but nolanguageis provided. Model warm-up completed! 0 phrases cached. === TTS SERVICE CREATED, speaker_wav=C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\stt_server\Voice_samples_Gus_000_norm_fixed.wav === Generating speech for text: 'What about 800 culture? Tell me more.' Using model: xtts_local [METRIC] Model source: LOCAL; Speaker: None Text splitted to sentences. ['What about 800 culture?', 'Tell me more.'] [WARN] Direct tts_to_file failed: 'NoneType' object has no attribute 'encode'. Falling back to in-memory synthesis. Text splitted to sentences. ['What about 800 culture?', 'Tell me more.'] Text splitted to sentences. ['What about 800 culture?', 'Tell me more.'] Text splitted to sentences. ['What about 800 culture?', 'Tell me more.'] [ERROR] In-memory synth fallback failed: Model is multi-lingual but nolanguage` is provided.
[METRIC] TTS synthesis time: 3185.0 ms
Error generating TTS: TTS engine failed to create output file: C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\audio_output\tts_output_20250915_134428.wav
[CHAT] TTS failed: Failed to generate TTS: TTS engine failed to create output file: C:\Users\admin\OneDrive\Desktop\Voice-model\intuitionX-voice-assistant\audio_output\tts_output_20250915_134428.wav
INFO: 127.0.0.1:56975 - "POST /chat HTTP/1.1" 200 OK

Make sure to help me through this and this is my hf hosted model https://huggingface.co/kaushikaiml/Gus-finegrained

[dinhphuong98]

Using this script to cut hallucination at the end of sentence.

Example for Vietnamese, try it on your target language.
`from TTS.tts.layers.xtts.tokenizer import VoiceBpeTokenizer
import librosa
import soundfile as sf

if name == "main":

tokenizer = VoiceBpeTokenizer(vocab_file="checkpoints/fine-tuning/vocab.json")


input_text = "xe có sạc nhanh không?"

ref_text = "Chào buổi sáng hôm nay bạn khỏe không tôi đang có rất nhiều điều muốn được chia sẻ cùng bạn."

ref_text = ref_text.strip().lower()
input_text = input_text.strip().lower()

ref_tokens = tokenizer.encode(ref_text, lang="vi")
input_tokens = tokenizer.encode(input_text, lang="vi")
ref_audio_len = librosa.get_duration(path="./assets/zipvoice_output.wav")

print(ref_tokens)
print(input_tokens)

print()
print(len(ref_tokens))
print(len(input_tokens))

print()

input_audio_len = ref_audio_len*len(input_tokens)/len(ref_tokens)

print(ref_audio_len)
print(input_audio_len*1.25)

audio_file = "output.wav"
y, sr = librosa.load(audio_file, sr=None) 
# chunk duration 2 seconds
chunk_duration = input_audio_len*1.25

chunk_samples = int(chunk_duration * sr)

chunks = [y[i:i + chunk_samples] for i in range(0, len(y), chunk_samples)]

for i, chunk in enumerate(chunks):
    output_file = f"chunk_{i}.wav"
    sf.write(output_file, chunk, sr)`

[saurabhthesuperhero]

how is it working for you