Huge gradients when training

[lyndonlauder]

Hello @crlandsc @iver56

I have tried out log-wsme as the loss function for training audio separation model but I noticed the gradients are really big. My setting was adamw with 0.0001 learning rate.

Do you have any advice for parameter settings such as learning rate?

Thanks

[crlandsc]

Hi @lyndonlauder - What type of data are you training with? And does your model train well with other losses? torch's basic L1 on the waveform is a good place to start. Your learning rate is likely fine, but that can depend on the model dynamics.

I also recommend gradient clipping and using a warmup scheduler can help, but if the model is relatively stable these shouldn't be absolutely necessary.

I had run into issues a while back with some instability using this, but also had some success. But I haven't tested it in some time (had to shift focus to other work), so I'm not sure if it was an issue inherent to the loss function code or if it was something I was doing with my training code.

Can you perhaps give some more context about what you are running into and compare it to the behavior with a simple L1 loss? For example, what model are you using, what dataset, what do the loss curves look like, etc. If this is an issue inherent to the code, it would be good to root it out for other users. Thanks for flagging this!