Whether the latest F5-TTS V1 supports Cantonese training

[liutong0127]

@SWivid Hello,I trained one night with the Cantonese dataset on a rented RTX3090 with update=32736

The reference audio is also in Cantonese, but the inferred audio still speaks Mandarin, without a Cantonese trend.
tmpzbe_yjqr.zip

[SWivid]

How is effective batchsize, use ema or not

[liutong0127]

非常感谢你的回复。下面是我执行的训练命令，以及nvidia的使用

加速启动 --mixed_precision=fp16 /data/project/F5-TTS/src/f5_tts/train/finetune_cli.py --exp_name F5TTS_v1_Base --learning_rate 1e-05 --batch_size_per_gpu 4996 --batch_size_type frame --max_samples 64 --grad_accumulation_steps 1 --max_grad_norm 1 --epochs 10480 --num_warmup_updates 6505 --save_per_updates 500 --keep_last_n_checkpoints -1 --last_per_updates 200 --dataset_name cantonese --finetune --tokenizer pinyin --log_samples

我认为我没有启用 ema。大多数进展都是使用 Gradio ui 默认值完成的

| 0 NVIDIA GeForce RTX 3090 On | 00000000:00:03.0 Off | N/A |
| 30% 58C P2 206W / 350W | 17350MiB / 24576MiB | 94% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:00:04.0 Off | N/A |
| 31% 59C P2 218W / 350W | 12979MiB / 24576MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GeForce RTX 3090 On | 00000000:00:05.0 Off | N/A |
| 30% 55C P2 211W / 350W | 13149MiB / 24576MiB | 100% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GeForce RTX 3090 On | 00000000:00:06.0 Off | N/A |
| 58% 62C P2 225W / 350W | 14395MiB / 24576MiB | 98% Default |
| | | N/A |

[SWivid]

Got it, I would suggest train longer.

But take efficiency into consideration, could try modify text tokenizer (write a new Cantonese version to replace

F5-TTS/src/f5_tts/model/utils.py

Line 137 in f968e13

def convert_char_to_pinyin(text_list, polyphone=True):

) i.e. to have a specific g2p for Cantonese, might boost (though I don't know if there is good open-source g2p tool for Cantonese)
Cuz currently, traditional words will also be converted to pinyin but we know Cantonese is not pronunced like that.

[liutong0127]

Okay, I will continue to study and try. Thanks again for your reply.

[liutong0127]

明白了，我建议训练更长时间。

但考虑到效率，可以尝试修改文本标记器（编写一个新的粤语版本来替换

F5-TTS/src/f5_tts/model/utils.py

Line 137 in f968e13

def convert_char_to_pinyin(text_list, polyphone=True):

) 即有一个专门针对粤语的 g2p，可能会有所提升（虽然我不知道是否有好的针对粤语的开源 g2p 工具）
因为目前，繁体字也会转换为拼音，但我们知道粤语不是那样发音的。

I added Cantonese pinyin, and the tone of speech was closer to Cantonese, but unfortunately the content deduced was not the reasoning content I entered.
Do you think I need to stop my current training and start again using Cantonese Pinyin?

This is the inference effect of the two methods
audio.zip

[SWivid]

From my personal perspective, using Cantonese Pinyin is better.