About GPUs and Training time.

[Yanping9]

Thank you for your wonderful work！May I have a question? What Gpus do you use in the training process and how much time do you spend? It's important for me to reproduce.

[hxqstree]

Hi，I also want to know the training time, which is also very important to me

[Artanic30]

Hi, thanks for interested in our work. For fully supervised setting, the training takes about 2 days on 4 2080Ti GPUs.

[kkayla99]

Hi, thanks for interested in our work. For fully supervised setting, the training takes about 2 days on 4 2080Ti GPUs.

Hello, I keep encountering a CPU OOM issue during the second epoch while training on 4 NVIDIA 3090 GPUs, which causes the code to stop automatically. Could you please provide some advice on why this might be happening and how to resolve it?
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 1 (pid: 453294) of binary