How to Transcribe Conversational Speech for F5-TTS: Fillers, Pauses, and Acronyms

[ahm3texe]

Hello everyone, and thank you for creating such a powerful tool.

I am fine-tuning F5-TTS on a dataset of natural, conversational Turkish speech. My goal is to produce a highly realistic and human-like voice. For context, Turkish is a highly phonetic ("what you see is what you get") language, which makes the following transcription choices particularly impactful on the model's performance.

I've encountered several critical decision points and I'm unsure about the best practices, especially for an alignment-free model like F5-TTS. My assumption is that the model will try to map whatever is in the transcript directly to the audio, making these decisions crucial. I'm hoping the community can share their experiences and help create a sort of "cookbook" or guide for these common challenges.

Here are the specific problems I'm facing:

1. Handling Conversational Disfluencies:

Filler Words ('uh', 'um'): What is the most effective approach?

• a) Literal Transcription: Write them as they sound (e.g., "it's uhm very important").
• b) Special Token: Use a single token for all fillers (e.g., "<filler>").
• c) Removal: Clean the transcript entirely for a fluent output.

Non-Speech Sounds (Laughs, Coughs): Should these be transcribed with tokens like [laugh] or ignored completely?

2. Transcribing Standard Written Forms:

Spelled-Out Acronyms: For an acronym that is pronounced letter-by-letter (like "FBI" in English, or "TBMM" in my Turkish data), what's best?

• a) As-is: "TBMM" (Risk: model might try to read it as a word, "tıbımımı").
• b) Spaced Letters: "T B M M" (Seems safer for forcing individual letter pronunciation).
• c) Phonetic Spelling: Write out the names of the letters (e.g., "TE BE ME ME").

Numbers, Dates, and Currencies: What is the most robust method?

• a) Digits & Symbols: Keep them as "5", "1990", "50%", "100₺".
• b) Written Out (Normalized): Convert everything to words (e.g., "five", "nineteen ninety", "fifty percent", "one hundred Turkish lira"). This seems more consistent, but is it the standard practice?

3. The Role of Punctuation:

• Grammar vs. Pauses: Should punctuation strictly follow grammatical rules, or should it reflect the speaker's actual prosody (rhythm and intonation)?
  For example, should I use a comma for every short pause and a period for a longer, sentence-ending pause, even if it's grammatically unconventional? My feeling is that this would better teach the model the rhythm of speech.

I would be incredibly grateful for any insights, rules of thumb, or experiences you could share on these topics. What has worked best for you in achieving a natural-sounding model with F5-TTS?

Thank you for your time and help

[SWivid]

Hi @ahm3texe , the choices of the setup may come from how you consider your inference, two major cases:

• train with diverse data, apply the minimum level of manual design -> many aspects during inference are not under control, but the model may occasionally offer some unexpected and pleasant surprises, such as spontaneous and natural non-verbal cues.
• train with selected data, especially a comprehensive text frontend -> the inference process is highly controllable, synthesize whatever is given (,and no more than given).

In particular, these two directions are not binary. You can adjust the data proportions (uncensored / normalized) to some extent to achieve a certain degree of balance and coordination.

To your three points in detail, we may offer some feedback and insights with v1_base_zh_en model just for reference:

1. Handling Conversational Disfluencies:
   • Emilia transcription is derived from ASR, there might be almost no explicit annotations of filler words. So if you use high-quality non-verbal data for finetuning, you might be able to control the relevant abilities, though somehow effective for limited speakers cuz thus data is not much, not much publicly available.
2. Transcribing Standard Written Forms:
   • If your text frontend is good, perform normalization will be better. Just ensure the the form is consistent, it doesn't matter whether it's "T B M M" or "T.B.M.M.". Same for numbers, and etc.
3. The Role of Punctuation:
   • Personally thought this point could be a bit over-designed. Perhaps retaining diverse texts can make the inference process more robust.
   • It is also worth trying, that is, complete control might turn out unexpectedly to be quite good.

Best wishes.