Causal CD leads to static and blurry results

[WJohnnyW]

Hi,

Thank you for the great work!

I am training on my own AV-joint model based on Ovi. After the S1 AR training stage, the AR generation quality is slightly worse than the base model, but still acceptable.

However, after the S2 Causal CD training stage, I found that the inference results are problematic: the generated video is almost static, and sometimes becomes very blurry. When the person is talking, there is corresponding audio, but the volume is very low.

So far, I have trained for roughly 2700 steps × 14 samples. Do you think this is simply because the training is still insufficient, or could there be something wrong with my training setup?

[zhuhz22]

1. Could you share a video for me to see? Without actually watching it, it's hard for me to give a very specific interpretation.
2. Generally speaking, I have the following points for discussion:
3. What do you mean by "I have trained for roughly 2700 steps × 14 samples"? Or more specifically, what does "14 samples" mean?
4. When you do sampling, are you using the CM solver instead of the Euler solver? 4 steps?
5. What is the CFG of the teacher model for CD?
6. If the problem is hard to solve, how about simply using some ODE data (say, 500 to 1000 samples) to try causal ODE and see how it works? If it performs significantly better than CD, then there really is an issue with the CD implementation.

I am willing to continue discussing this with you.

[WJohnnyW]

sample_gen_audio_000007_V00_S0426_I00000372_P0564.mp4

sample_gen_audio_000010_V00_S0675_I00000379_P0831.mp4

(1) 14 sample mean total batchsize for each step
(2) CM solver, 4 steps
(3) cfg 2.75 for video, 4 for audio

[zhuhz22]

I think this is pretty good; it meets the requirements of this stage. You can continue training for a few more steps, then proceed to Stage 3.

[WJohnnyW]

I think this is pretty good; it meets the requirements of this stage. You can continue training for a few more steps, then proceed to Stage 3.

Thanks for the quick reply！ I will continue the training first.
Also I use the 2700 ckpt for Stage 3 (1600 step * 7 sample/batch). Getting results like below which is blurry first then flicking an finaly total arfifacts, is it correct?

sample_gen_audio_000001_V01_S0223_I00000307_P1506.mp4

[zhuhz22]

Obviously, this is not correct—the results are way too bad! I think this is no longer an issue of training effectiveness; there must be a bug in the code. I suggest you let an AI agent help you debug. For example, is the teacher-critic bidirectional? In DMD, the teacher is frozen, while the critic is trained. I think the problem already lies in this kind of macro-level setup.

[WJohnnyW]

OK, I will debug first