Question about Flow-matching training from scratch

[CharlesNii]

I’m trying to train CosyVoice2’s Flow-Matching model from scratch using the Dual Codec's code, since I need its streaming inference capability.
⚙️ Training Setup
Hardware: 8 × A800 GPUs
Training time:
1e-5 LR → trained for 3 days, stable but still not well-fitted (poor audio quality, inconsistent timbre).

1e-4 LR → trained for 1 day, but quickly leads to gradient explosion and no convergence.

Optimizer: AdamW
Dataset: internal speech dataset (not using LibriTTS, the scale of dataset is about 2000 hours)
📉 Observations
The 1e-5 model produces overly smooth and unclear results, while the 1e-4 model diverges rapidly.

Below are 1e-5 model's mel-spectrogram comparisons:

predict mel：

gt mel:

❓ Question
Has anyone managed to successfully train the Flow-Matching model from scratch (not fine-tuning pretrained weights)?
Any advice or experience on:

Choosing an appropriate learning rate or LR schedule

Using EMA, gradient clipping, or warmup strategies

Adjusting flow noise schedule or loss balancing to stabilize early training

Any hints would be greatly appreciated!

[ScottishFold007]

I’m trying to train CosyVoice2’s Flow-Matching model from scratch using the Dual Codec's code, since I need its streaming inference capability. ⚙️ Training Setup Hardware: 8 × A800 GPUs Training time: 1e-5 LR → trained for 3 days, stable but still not well-fitted (poor audio quality, inconsistent timbre).

1e-4 LR → trained for 1 day, but quickly leads to gradient explosion and no convergence. Optimizer: AdamW Dataset: internal speech dataset (not using LibriTTS, the scale of dataset is about 2000 hours) 📉 Observations The 1e-5 model produces overly smooth and unclear results, while the 1e-4 model diverges rapidly.

Below are 1e-5 model's mel-spectrogram comparisons:

predict mel：

gt mel: ❓ Question Has anyone managed to successfully train the Flow-Matching model from scratch (not fine-tuning pretrained weights)? Any advice or experience on:

Choosing an appropriate learning rate or LR schedule

Using EMA, gradient clipping, or warmup strategies

Adjusting flow noise schedule or loss balancing to stabilize early training

Any hints would be greatly appreciated!

The CosyVoice project's flow-matching model (part of the broader FunAudioLLM/CosyVoice repo on GitHub) is indeed trained from scratch in the original implementations described in the associated papers, using large-scale datasets like 130k hours of Chinese speech and 30k hours of English data for CosyVoice-2. While the papers don't detail community experiences, a related GitHub issue (#308) reports one user successfully stabilizing training from scratch by freezing non-decoder components (e.g., encoder, length regulator, embeddings) and only optimizing the decoder—this avoided loss crashes around 15k steps and improved audio quality compared to training the full model, which led to instability. Your specific issue (#1625) remains open with no comments as of now, so no direct resolutions there yet.

Based on the CosyVoice papers, the original flow-matching paper, and general tutorials/blogs on training flow-matching (FM) models, here's structured advice tailored to your setup (8x A800 GPUs, ~2000-hour internal dataset, AdamW optimizer). FM models like this are simulation-free CNFs (continuous normalizing flows) that regress a velocity field for ODE-based generation, often more stable than diffusion models but still prone to gradient issues or poor fitting if hyperparameters aren't tuned. Your observations—stable but underfitted at 1e-5 LR, exploding at 1e-4—align with common early-stage challenges in FM training from scratch, where low LRs can lead to slow convergence and "smooth" outputs (due to insufficient exploration), while high LRs cause divergence. I'll focus on your key questions, drawing from these sources.

Choosing an Appropriate Learning Rate or LR Schedule

• Base LR Suggestions: Start with 1e-4 as a baseline for "normal" models (as in CosyVoice v1 for multilingual setups) or 5e-4 for smaller experiments (from the original FM paper on image datasets like CIFAR-10). Your 1e-4 attempt exploding suggests scaling it down initially to 5e-5 or 2e-5, especially since your dataset (~2000 hours) is smaller than the papers' 160k+ hours—larger datasets often tolerate higher LRs due to better gradient estimates. For fine-tuning-like stages (e.g., multi-speaker SFT in CosyVoice-2), they drop to 1e-5, which matches your stable but underfitted run.
• Schedule: Use a polynomial decay with warmup (linear ramp-up from near-zero, e.g., 1e-8, to peak LR over 10k-20k steps, then decay to 1e-8 by end). This is common in FM to stabilize early training and prevent explosions. Avoid constant LRs, as they can lead to plateaus or divergence in long runs. In the original FM paper, warmup is 20k-45k steps depending on dataset size. For your 3-day run at 1e-5, extending with decay might help escape the "overly smooth" regime.
• Batch Size Impact: Aim for effective batch sizes of 256-2048 (scale with your 8 GPUs via gradient accumulation if needed). Larger batches stabilize gradients at higher LRs.

Using EMA, Gradient Clipping, or Warmup Strategies

• EMA (Exponential Moving Average): Apply EMA to model weights (decay ~0.999) during training—it smooths updates and improves generalization, especially for audio where timbre consistency is key. In CosyVoice v1, EMA is used in the speech tokenizer (decay α for codebook updates), but extending it to the full FM model can help with your inconsistent timbre issues. Tutorials recommend EMA for stability in FM, as it reduces noise in velocity field predictions. Update EMA every step or few steps, and use the EMA model for evaluation/inference.
• Gradient Clipping: Absolutely use this to prevent explosions—clip global norm to 1.0 or 0.5. It's not explicitly in the CosyVoice papers but is standard for FM stability, especially with Adam/AdamW, to handle outliers in velocity regression. In your 1e-4 run, this could cap gradients before they blow up.
• Warmup: As noted, 10k-45k steps linear warmup is crucial for convergence from scratch. It allows the model to learn coarse velocity fields before fine details, reducing early instability. Combine with your AdamW (β1=0.9, β2=0.999, ε=1e-8, no weight decay initially).

Adjusting Flow Noise Schedule or Loss Balancing to Stabilize Early Training

• Flow Noise Schedule: Use a cosine scheduler for timesteps t: t := 1 - cos( (1/2) t π ), as in both CosyVoice versions—this concentrates sampling in curved trajectory regions, improving efficiency and stability over uniform U[0,1]. For early training, bias toward lower t (e.g., time warping with s=0.5) to focus on high-curvature areas where explosions often start. In CosyVoice-2's chunk-aware setup for streaming, mix masks (non-causal, full-causal, chunk-M) uniformly in batches to balance latency and quality.
• Loss Balancing: The core loss is L1 (CosyVoice-2) or L2 (original FM/CosyVoice v1) on velocity fields—stick to L1 for robustness to outliers in mel-spectrograms. Add classifier-free guidance (CFG) dropout (0.2 prob) during training, with β=0.7 at inference for better sample quality. For stability, optionally drop weighting terms (as in some FM variants) or use Optimal Transport (OT) paths over diffusion paths—these are linear and converge faster with fewer steps (e.g., 157k-500k total iterations in FM paper). If divergence persists, try path straightening/rectification: Train on linear forward processes (α(t)=1-t, σ(t)=t) to reduce trajectory entanglement.

Additional Tips for Your Setup

• Freezing Strategy: From About the training of the flow model #308, freeze everything except the decoder initially—train for ~15k steps, then gradually unfreeze. This stabilizes from-scratch runs on speech data.
• Monitoring: Track velocity field smoothness (e.g., via mel-spec visualizations like yours) and use higher-order ODE solvers (e.g., RK4 over Euler) for inference to avoid numerical instability in streaming.
• Dataset Scaling: Your 2000 hours is solid but smaller than papers—augment with noise/mixing if needed, and ensure multilingual balance if applicable.
• Expected Training Time: Aim for 100-1000 epochs or 200k-800k steps total, scaling with your hardware (should be feasible in days-weeks).

If this doesn't resolve it, consider posting more logs (e.g., grad norms) to the GitHub issue for community input, or experiment with the original FM codebases for baselines. Let me know if you share code snippets for further debugging!