Generating text in Spanish gives truncated or mangled output.

[markd89]

1. Short sentence issue:

pocket-tts generate --language=spanish --text "Alguna sugerencia."
pocket-tts generate --language=spanish --text "Alguna sugerencia." --voice jean
pocket-tts generate --language=spanish --text "Alguna sugerencia." --voice embeddings/mexican1.safetensors

Result: sounds like "Alguna su"

1b. I thought of adding a word at the end so: "Alguna sugerencia. hola"
Result: sounds like "auna sugerencia. hola". Sugerencia is now handled but Alguna is mangled.

2. Long sentence is mangled in different ways depending on voice.
   pocket-tts generate --language=spanish [--voice jean | --voice embeddings/mexican1.safetensors] \
"Sé que no es perfecta, pero espero que sea aceptable. Esta es la cubierta del próximo libro que voy a publicar. Ya está casi terminado."

a. Default (alba). Result:
"[mangled] + aceptable. Esta es la cubierta del próximo libro que voy a publicar. Ya está casi terminado."
Issue: [first part of sentence is mangled]

b. jean. Result:
"é que no es perfecta, pero que sea aceptable. Esta es la cubierta del próximo libro que voy a publicar. Ya está casi terminao."
Issue: [ 1. se not spoken, 2. espero missing, 3. (d) in terminado is not spoken ]

c. mexican1.safetensors. Result:
"Sé que no es perfecta, pero espero que sea aceptable. Esta es la cubierta del próximo libro que voy a publicar. Ya está casi terminao."
Issue: [ (d) in terminado is not spoken ]

Note: I can provide the output WAV files if that would be helpful. I do not have distribution rights to the source for mexican1.safetensors but can provide it to you privately if needed for testing purposes only.

[gabrieldemarmiesse]

I think you can tune the frames_after_eos argument to ask the model to go on despite the end of sequence token being issued. Each frame is 80ms. Let us know if that fixes your problem. Otherwise we might also want to play with the threashold.

[markd89]

1. DEFAULT_FRAMES_AFTER_EOS = 14
   pocket-tts generate --language=spanish --text "alguna sugerencia." --voice embeddings/mexican1.safetensors;aplay tts_output.wav

Result: Improved. Sugerencia is no longer truncated, but there is a subtle issue. We are pronouncing "a una sugerencia".
Note: Using --voice jean does the same thing

2. DEFAULT_FRAMES_AFTER_EOS = 2, same test as (1) above. Result: Text truncated/mangled. So 2 is not enough for the short text but 14 is.

3. DEFAULT_FRAMES_AFTER_EOS = 14
   pocket-tts generate --language=spanish --text "Sé que no es perfecta, pero espero que sea aceptable. Esta es la cubierta del próximo libro que voy a publicar. Ya está casi terminado." --voice embeddings/mexican1.safetensors;aplay tts_output.wav

This is also improved. It's not mangled, but two issues.

a. There is an audio artifact after the end of the text.
b. Subtle issue.. "sea aceptable" is spoken like "se aceptable". "sea" sounds like "Say-Ah". The Ah is soft but should be a little more than what we're speaking. I am not a native Spainsh speaker but believe this to be correct.

Note: Using --voice jean does not have #a but does have #b.

4.DEFAULT_FRAMES_AFTER_EOS = 2, same test as (3) above. #a not present, #b present.

So a bit of a catch-22, where = 14 is needed for short text but that causes an artifact on longer text.

5. Finally, an English issue.

pocket-tts generate --text "XFRA appears to be a convenient sidestep of local opposition by tech bros, transforming homes into miniature data center nodes for $12 billion."; aplay tts_output.wav

a. We speak 2 billion not 12 billion
b. Ideally, the model should see text "$12 billion" and return audio speaking "12 billion dollars"

[gabrieldemarmiesse]

For the 4 first points, it's expected that depending on the model and the voices, people will have to adjust the default_frames_after_eos. Some other languages models might not have a quality as good as the english models, but we also provide the 24 layers variants for those, and the 24 layers variants do less mistakes. If the 24 layers are still real-time on your cpu, it might be worth it to explore this option. You can also use the quantized version of the 24 layers variants to improve the inference speed.

For 5) we expect users to do text pre-processing and transform numbers/dates/etc into something easily readable by a TTS. Typically transforming the text "$12 billion" into "12 billion dollars"

[shadowofsoul]

Hello to @markd89 and @gabrieldemarmiesse on this subject, i actually trained a LoRa in spanish for the pocketts version of before the new languages was launched and i think i had a better results in inference. i'm not a 100% sure the problem or advantage is in the layers, but in the LSD steps, new versions of pocketts are fixed at only 1 LSD step when originally you can do up to 10.

if you wanna give it a try, the ONNX flavor and safetensors are at: https://huggingface.co/ipsilondev/pocket-tts-spanish-streaming_v2/tree/main

[markd89]

1. Spanish. I played a little with spanish_24l.

DEFAULT_FRAMES_AFTER_EOS = 14
pocket-tts generate --language=spanish_24l --text "alguna sugerencia." --voice embeddings/mexican1.safetensors;aplay tts_output.wav

Still speaks "a una sugerencia". Any other suggestions?

2. I implemented pre-processing in FrontPocket to handle the "$12 billion dollars" scenario as you suggested and some other similar cases.

3. Back to English.. I sometimes get mispronounced words.

a. deny

pocket-tts generate --text "deny it.";aplay tts_output.wav
The deny is mispronounced is de-nee

When it is not the first word, it is pronounced properly, i.e.
pocket-tts generate --text "don't deny it.";aplay tts_output.wav

b. won't

won't is mispronounced as won
pocket-tts generate --text "won’t bother going through all the reasons";aplay tts_output.wav

won't is correctly pronounced here
pocket-tts generate --text "won't";aplay tts_output.wav

Is there another paramater I should adjust for that one?
Should I continue to report little issues like this?

4. @shadowofsoul My (limited) understanding is that more LSD steps takes more cycles to produce the output. The spanish_24l is slower than real-time on my CPU. Is your model able to perform TTS at real-time speeds on CPU? Regardless, thanks very much for creating your variation. It makes perfect sense to have a hispanohablante build the Spanish TTS.