GPU memory during inference

[jrdimale]

Thanks for your work!

I would like to inquire about the possibility of optimising the inference speed and GPU memory usage of the model.

I saw in previous issues that you are considering it, but have seen no updates.

Is it possible? What should be the way to proceed to reduce it (a lot)?

cheers

[xuelunshen]

hello, @jrdimale , This matter has not yet made progress. However, I can give you a tip to save GPU memory, such as reducing the size of the input image can greatly reduce the consumption of GPU memory.

[jrdimale]

Hi! Indeed I tried reducing the input size of the image, but the GPU consumed during inference is almost the same.
I tried with 480x640, as well as 672x896.
Im using GIMdkm, as its the one that provided the best results in my dataset. Is there another approach to leverage to reduce as much as possible the memory used?

[xuelunshen]

Could it be this piece of code causing the issue? demo.py changes the size of the input image. You can comment out this section.

gim/demo.py

Lines 425 to 428 in f09105a

	orig_width0, orig_height0, pad_left0, pad_right0, pad_top0, pad_bottom0 = get_padding_size(image0, width, height)
	orig_width1, orig_height1, pad_left1, pad_right1, pad_top1, pad_bottom1 = get_padding_size(image1, width, height)
	image0_ = torch.nn.functional.pad(image0, (pad_left0, pad_right0, pad_top0, pad_bottom0))
	image1_ = torch.nn.functional.pad(image1, (pad_left1, pad_right1, pad_top1, pad_bottom1))

Other methods you could try include:

1. Save the result of model.match to a file, then clear the GPU memory before proceeding with model.sample.
2. Reduce the input parameter 5000 for model.sample.

gim/demo.py

Lines 432 to 433 in f09105a

	dense_matches, dense_certainty = model.match(image0_, image1_)
	sparse_matches, mconf = model.sample(dense_matches, dense_certainty, 5000)

[jrdimale]

Hi!

I tried multiple configurations and memory is still very high, above 12GB despite making the images smaller and changing this padding.

Furthermore, I'm not planning to use the code in demo.py, actually I use the model in another script by just loading the model with the weights from gim_dkm_100h.ckpt, and performing inference.

I instantiate a DKMv3 object with the desired h and w, then load the weights and run inference. Is there any change I can do to lower the model gpu usage drastically?

Thanks for your time

[xuelunshen]

I see. I think there isn't a very convenient method to quickly achieve your needs right now. The more troublesome approach I can think of is to quantize the model or convert it to TensorRT.

[jrdimale]

Have you tried quantization in any of the trained models? Or conversion to TensorRT/ONNXRT?

If not, do you have any suggestions on how to proceed?

[xuelunshen]

Sorry, I'm not familiar with this and haven't tried it before. You need to search Internet to learn the relevant information.