Finetuning for language vs finetuning for voice vs finetuning for industry

[0xrushi]

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The paper

Question

Hi, could someone explain which components of a model typically need to be fine-tuned when adapting it to different scenarios?
Like how would fine tuning differ between Arabic, or English with a different voice/accent, or English tailored for a specific domain like finance ? (this one i think only the helium model needs to be finetuned)

[manukyutai]

Hi!

I would assume that if you want to change the language, ideally you want to retrain from scratch, but I haven't tried such a scenario with fine tuning.

To change the voice or tailor it to a specific domain, lora fine tuning should be more than enough.

[0xrushi]

hi @manukyutai
Quick question - why does the default Moshi voice sound different from the voices in the kyutai/DailyTalkContiguous dataset?
I lora finetuned on the same dataset (about ~1000 files, all converted to a single voice style), but the output at inference still has a different style.
Do I just need to train for more epochs?

data:
  train_data: '/home/user/DailyTalkContiguous/dailytalk.jsonl'
  eval_data: '' # Optional - you can leave this empty or use the same file
  shuffle: true

moshi_paths:
  hf_repo_id: "kyutai/moshiko-pytorch-bf16"

full_finetuning: false
lora:
  enable: true
  rank: 128
  scaling: 2.
  ft_embed: false

first_codebook_weight_multiplier: 100.
text_padding_weight: .5

# training hyperparameters
duration_sec: 100
batch_size: 16
max_steps: 400
gradient_checkpointing: true
num_microbatches: 8

# memory optimization
param_dtype: "bfloat16"  # Ensure bfloat16 precision

optim:
  lr: 1e-6
  weight_decay: 0.1
  pct_start: 0.05

seed: 42
log_freq: 10
eval_freq: 50
do_eval: false
do_ckpt: true
ckpt_freq: 100
save_adapters: true

run_dir: "/home/user/DailyTalkContiguous/moshi_7B" 
overwrite_run_dir: false

[manukyutai]

We proposed fine tuning on DailyTalk as a proof that you can change the voice of Moshi using finetuning. So before finetuning it has Moshi voice and after it should have DailyTalk voice - if not, you're probably not training for long enough.