Question about creating train set for lora-finetuing Moshi

[ranck626]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

Other / All

Question

Hello, thank you for open-sourcing the Moshi fine-tuning code. I've been listening to several audio clips from the DailyTalkContijuous dataset. I noticed that they all seem to be multi-turn conversations from two characters.
I'm wondering:

1. Is this build necessary? Can each audio clip contain only a single turn of dialogue, such as User: XXXX; Moshi: XXXX?
2. 2. Can Moshi learn multiple voices? I've seen improvements in previous cases, with Moshi able to simulate pirates and other characters.

[ranck626]

@LaurentMazare Looking forward to your reply.

[LaurentMazare]

Hello,

1. You can fine-tune on single turn only, but then you should probably expect your model to only work in this fashion so at inference time you will only be able to do a single turn and the model won't learn that it can be interrupted by the user and that it should stop speaking.
2. Yes learning multiple voices is doable. If you fine-tune the model on sequence such as " Speak like a pirate. ..." the user will be able to control the voice by asking the model to do so.