About adding noise to anchor trajectories

[GeJintian]

Hi, thank you so much for your great work. I have a question about the noise injection during training.

I noticed that you modified the step function in DDIMScheduler_with_logprob to incorporate the multiplicative noise described in the paper, and this is used during the diffusion sampling process. However, when adding noise to the anchor trajectories, the code still calls the original add_noise implementation in both training and test. As far as I understand, add_noise applies additive noise, which seems inconsistent with the paper’s discussion and might lead to the issues it mentions.

Could you clarify whether this is an intentional design choice (e.g., applying multiplicative noise only during the denoising steps), or if it might be an oversight?

[Doctor-James]

While noise injection is utilized in both instances, the specific implications of the noise differ significantly between the two contexts.In the training of DiffusionDrive v1, noise is introduced to corrupt the trajectory (consistent with the forward process in standard diffusion models). Therefore, standard additive noise is utilized to gradually degrade the signal from $x_0$ to $x_t$:

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$$

In contrast, DiffusionDrive v2 is based on DPPO and primarily operates during the denoising process. In this context, noise is injected to facilitate trajectory exploration, necessitating the use of multiplicative noise.

$$x_{t-1} = \underbrace{\mu_\theta(x_t, t)}_{\text{Deterministic Mean}} + \underbrace{\sigma_t z}_{\text{Exploration Noise}}$$

[GeJintian]

Thank you so much for your reply! I have another question regarding the computation of IL loss. I found that L1 supervision is applied to all explorations of all groups (intentions?). Although you add weights according to their advantages, would it still cause mode collapse?

[Doctor-James]

I have not conducted extensive ablation studies on this specific component. The primary purpose of the IL loss is to stabilize training. Under our current strategy, certain scenarios yield zero reward (for example, when all generated trajectories are collision-free but none outperform the ground truth). In such instances, no gradients are backpropagated for the sample, which can compromise training stability when processed within a batch. The IL loss serves to provide a consistent gradient signal; without it, training becomes unstable, though the specific weight assigned to this loss is not critically sensitive.