Arabic support

[cod3r0k]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The paper

Question

Do you have Arabic support?

[LaurentMazare]

The aim of this repo is to be language agnostic and to make it easy to finetune a moshi like conversational agent on some specific datasets. So if you have a dataset of arabic speech, you can give a try at finetuning the model on it and hopefully it would start working on arabic.
And certainly do come back if you try this, we would be pretty curious to know how it goes.

[cod3r0k]

"Great, @LaurentMazare. You mean that I just need to create the dataset in the format specified by the repo (stereo) and then fine-tune it, and it's language-agnostic? Do you have an estimate of the minimum number of hours required for the dataset?

[LaurentMazare]

It's hard to know how many hours will be required for this, for finetuning on English our main example only uses 20h of data and that's enough but because of the language change you might need more.
You might want to have a look at j-moshi as they finetuned moshi on japanese but I don't think they mention the number of hours used.

[ashutoshsaboo]

@LaurentMazare is it possible to fine-tune moshi/either just fine-tune sub components of moshi (either or, whichever is possible for this usecase), for speech tasks such as say improving ASR and the likes, rather than for just end to end conversational response modelling? The reason is that some of our domain is very diverse accents, where out of the box ASRs don't work and typically a fine-tune is a must.

[LaurentMazare]

Moshi is a single model that handles speech to speech so there are no separate sub-components. That said you can certainly try to adapt the loss so that it only considers the text stream. It's also possible to specialize the model to perform ASR and TTS, see the "Deriving streaming ASR and TTS" section, page 16 of our arxiv paper. The current code would have to be adapted to introduce the delay mentioned in this section though that shouldn't be very difficult.