Fine tune anime-voice TTS for local inference

[supreme-gg-gg]

We're looking for alternatives to replace all API calls to ElevenLabs and external voice services with local inference models (if possible). There seem to be quite a few option existing for regular TTS model, but anime voice specific ones are still difficult to find as of the time of first writing.

If you know any pls suggest we're happy to make the entire extension run locally!

*I know that Gemma3n and other multimodal local LLM can already process audio + text to text, and there are many speech to text options, but a reliable TTS locally is far more important

[UPDATE] Unsloth supports fine tuning TTS models which might be really helpful: https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning We need some brave contributors to try this out, find a dataset, fine tune, and try running it locally to replace ElevenLabs!!

[supreme-gg-gg]

The result for local inferencing was not promising unfortunately. We still need to rely on 11labs for now. I tried Piper TTS which is a community one and quite lightweight, but there is no existing anime voice available. https://huggingface.co/rhasspy/piper-voices/tree/main/en/en_US

Then there is the VITS pretrained on anime voice you can just inference locally BUT it's a python app and it's pretty big and it has a ton of dependencies, it is also in Python which makes integration with VSCode difficult... https://huggingface.co/spaces/Plachta/VITS-Umamusume-voice-synthesizer/tree/main (I tried cloning and running their repo locally it seems to be ok but somehow their Gradio web UI is broken? But I think we can still use the ONNX runtime, BUT it is very difficult to natively and efficiently make work in Typescript it is also very slow compared to Elevenlabs)

Notably the same author for the VITS anime voice also worth this guide for fine tuning VITS: https://github.com/Plachtaa/VITS-fast-fine-tuning This would be useful if we ever want to do that, just gather some anime voice we can specialise to English I believe.

[supreme-gg-gg]

https://huggingface.co/spaces/hexgrad/Kokoro-TTS