CUDA Memory

[superxi]

Hi, I tried run this work on V100-32G, but it was CUDA out of memory. I am wondering how much cuda memory need to infer this repo?

[andyjiang1116]

I also got OOM on 2 X A100-40G

[Anshull-py]

I'm able to run the inference on 48GB single gpu setup (CUDA 12.6)

[555sad]

I used an RTX 5090 for inference, and the VRAM usage during inference was less than 18GB.
Therefore, I believe that a single RTX 4090 should be sufficient to complete the inference task.

[Anshull-py]

@555sad Are you planning to perform fine-tuning?