Hi VoxCPM Team,
I am exploring the fine-tuning code of VoxCPM and noticed the is_prompt parameter in the dataset processing/collation stage packers.py By default, is_prompt is set to False during fine-tuning. However, looking into the mechanics, flipping this parameter to True changes the structural arrangement of the sequence and the behavior of the loss_mask:
- When
is_prompt=True: The sequence layout is restructured. The reference audio/text segment is explicitly separated and treated as a "prefix" or contextual prompt at the beginning of the sequence, paving the way for the subsequent continuation sequence.
- When
is_prompt=True: The loss_mask for the prompt prefix section is forcefully set to 0. The loss computation and gradient backpropagation are tightly constrained only to the continuation part (the text/audio following the prompt prefix).
Questions & Core Concerns
- Is this parameter explicitly designed for Continuous and Ultimate Clone training?
- Fine-tuning mix without is_prompt=True: Will it fail to optimize continuation ability?
Could you confirm if keeping is_prompt=False on data with references will indeed dilute or fail to optimize the targeted continuation/generation capability? What is your recommended data configuration and is_prompt toggling strategy when mixing vanilla TTS data and prefix-conditioned audio data?
Thank you for your time and for open-sourcing this amazing framework! Looking forward to your insights.
Hi VoxCPM Team,
I am exploring the fine-tuning code of VoxCPM and noticed the is_prompt parameter in the dataset processing/collation stage packers.py By default, is_prompt is set to False during fine-tuning. However, looking into the mechanics, flipping this parameter to True changes the structural arrangement of the sequence and the behavior of the loss_mask:
is_prompt=True: The sequence layout is restructured. The reference audio/text segment is explicitly separated and treated as a "prefix" or contextual prompt at the beginning of the sequence, paving the way for the subsequent continuation sequence.is_prompt=True: The loss_mask for the prompt prefix section is forcefully set to 0. The loss computation and gradient backpropagation are tightly constrained only to the continuation part (the text/audio following the prompt prefix).Questions & Core Concerns
Could you confirm if keeping
is_prompt=Falseon data with references will indeed dilute or fail to optimize the targeted continuation/generation capability? What is your recommended data configuration and is_prompt toggling strategy when mixing vanilla TTS data and prefix-conditioned audio data?Thank you for your time and for open-sourcing this amazing framework! Looking forward to your insights.