How to design training data

[pabloriera]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

Paper / Data design

Question

Hi, I have been experimenting finetuning moshi with different data and have some questions.

I am using the same training config provided but using custom data.

I am having two problems:

1. Whenever I finetune the model (including the daily-talk data example) the time-to-respond of the system degrades. This means, it takes so much time to respond comparing to the behavior observed in default moshi. I have tried biasing the pad_mul and after that the time-to-respond improves, but the quality of the generation is a little degraded, and I think it will be much better to avoid touching that parameter. I tried to include overlaps between speech turns, it seem to improve a little, but it didn't fix it.
   Do you have any tips on how to design the data to fine tune and prevent that behavior?

2. The second problem is with the begging of the conversations. After fine tuning, I loss the Moshi's "Hello, whats goin on?" or other hellos. This happens even my data has a similar hello message at the beggining for every data example. What I observe is the systems start speaking from anyplace in a conversation similir to the topics on the data.
   Is there any other thing to try besides training with dialogues that alwas start with the same "hello" message? My first thoughrs were training with random pieces of conversation first, and the use the final batches to put a lot of the welcome part.

Thanks, awesome project.

[manukyutai]

Hello! Thanks for using the codebase!

For 1., pad_mul should be the simplest boost to preventing silence. Otherwise make sure the data you made has minimal silence between turns. There not much more to it to my knowledge.

For 2., it could be the the duration_sec you set is smaller than the size of your data, and as a result audios are chunked and you train on audio which don't start at the beginning of the conversation.