NVIDIA Canary & Parakeet Models

[logxdx]

Nvidia Parakeet and Canary models perform way better than OpenAI Whisper, would love to see them integrated into this already well built system.
Parakeet only supports ASR while Canary is a multitask model.

I can make edits and create a PR if you agree to this 😊

[KoljaB]

Yes pls that would be awesome. I was hesitating so far isolating all transcription related parts. Not sure how to solve the configuration overhead elegantly. With having in mind that I want something similar to how RealtimeTTS handles it's different TTS engines.

[logxdx]

Both of these models use the NVIDIA Nemo framework.
What I thought of is creating a configuration class for transcription model selection along with their parameters (if any), and pass it during initialization, allowing the user to select faster-whisper or nemo backends and their corresponding models. Please let me know if you have any other idea or perspective in mind.
This will take time so let's do this properly 🙌

[KoljaB]

Sounds good.
Setup is another thing. In RealtimeTTS I made several different installations possible (pip install RealtimeTTS [coqui] or pip install RealtimeTTS [elevenlabs]).
We will need something similar here. The Nvidia stuff uses Triton which does not work under Windows. We'd break installation for Windows if we don't split it up.
Guess I can take care of that if the code is working.

[logxdx]

Just saw that NVIDIA nemo currently does not have windows support :(
This will to wait for now i guess.