GPU usage during training

[segalinc]

Trying to replicate SAM training from scratch.
I notice that during training the GPU usage goes up and down 0-100 even when not logging info. I am not computing embedding on the fly as I am loading the already precomputed.
I wonder if it's related to the grad accumulation step or something else?
I also see training is slower than sd2.1 in comparison even if I can use a bigger batch
Any insights?

[lawrence-cj]

If you want faster training speed, u can try to set the grad_checkpointing = False and fp32_attention = False in your config file. However, it's possible that the former one will cause higher GPU memory, while the later one may cause NaN loss. The low usage of your GPU may be caused by the large batchsize which maybe higher than the bandwidth of the machine.