Matrix inverse alternative

[lnexenl]

I have done some work before to quantize RoMa and deploy it on some mobile platform like NVIDIA Jetson, however, I found out that the matrix inverse is the bottleneck. To keep the precision, the matrix inversion should be calculated in high precision (fp16 even fp32), which is extremely slow on a mobile platform.

Have you ever tried some other methods like direct regression to get gp_posterior?

[Parskatt]

Is this also true for the new version? I.e. cholesky decomp?

[lnexenl]

I have tested cholesky on server, it's about 4x faster in fp32 precision. But on mobile platform, the int8 data needs to be converted to fp16 and then do linalg operations and then convert to int8 again, the two conversions are time costing, and the floating-point units on NVIDIA Jetson are much less than that on server GPU, this will also slow down the computation.

But this is just my guess, the real performance of cholesky on mobile GPU still needs to be checked. The torch.linalg.inv implementation can only run about 1~2fps on NVIDIA Thor-U

I am just wondering that whether we can substitute linalg operators with other operators to enable real-time processing (e.g. 10Hz) of RoMa on embedded devices.

[hxsaaaa]

I have done some work before to quantize RoMa and deploy it on some mobile platform like NVIDIA Jetson, however, I found out that the matrix inverse is the bottleneck. To keep the precision, the matrix inversion should be calculated in high precision (fp16 even fp32), which is extremely slow on a mobile platform.

Have you ever tried some other methods like direct regression to get gp_posterior?

Hello, I've also been working on quantifying RoMa and deploying it on the Jetson platform recently. May I ask if you exported the PTH model as an ONNX/TRT model for deployment? If so, could you share your approach for exporting the ONNX model? I simplified computations by setting upsample=False and manually implementing upscaling, then replacing incompatible operators like linalg.inv and linalg.solve with handwritten calculations. While this successfully exported an ONNX model, inference performance is poor—all matched point pairs are incorrect.

[lnexenl]

I have done some work before to quantize RoMa and deploy it on some mobile platform like NVIDIA Jetson, however, I found out that the matrix inverse is the bottleneck. To keep the precision, the matrix inversion should be calculated in high precision (fp16 even fp32), which is extremely slow on a mobile platform.

Have you ever tried some other methods like direct regression to get gp_posterior?

Hello, I've also been working on quantifying RoMa and deploying it on the Jetson platform recently. May I ask if you exported the PTH model as an ONNX/TRT model for deployment? If so, could you share your approach for exporting the ONNX model? I simplified computations by setting upsample=False and manually implementing upscaling, then replacing incompatible operators like linalg.inv and linalg.solve with handwritten calculations. While this successfully exported an ONNX model, inference performance is poor—all matched point pairs are incorrect.

I cannot help too much on this. I met similar problem, and find out that error is from wrong matrix inv operator. For onnx matrix inv, you can refer yo microsoft's implementation. For trt matrix inv, you can implement it by writing a trt plugin with cuda linalg libraries @hxsaaaa