Questions regarding DIffusion with CFG

[ssvision]

@eje24 Great Course and I want to thank you so much for this content. I have a few queries and some clarifications that I am seeking.

I am following the solutions as per Exercise 3 to train a MNIST model for DIffusion and Flow Matching.

1. Regarding diffusion if I train the score network, the training is highly unstable. The losses don't seem to converge. What's the workaround for this? $$L(\theta) = | \epsilon_{\theta_t}(x_t) - \epsilon |^2$$ is the alternative for this I guess. Can this trained network be sufficient enough to act as surrogate for score network?
2. If I train the vector field(The Probability Field ODE) using the formula below (Eq 54 in Notes), then what should I use for Sampling ? Euler Simulator or Euler-Maryama Simulator?

[eje24]

Hi @ssvision , that's right, and I'm sorry to hear your training has been unstable. In practice, regressing against epsilon itself is more stable, provided that you weight appropriately to avoid a blowup in the variance of the objective near zero (Var[epsilon|x_t] grows like O(1 / beta_t) close to zero...). In the first iteration of the course, and for pedagogical reasons, we opted for the standard velocity regression, but this could change if it's really causing issues. Re integration/simulation, please see my email response. Take care!