GPU acceleration

[monreal93]

Hi,

I am working with MRIReco to reconstruct some high-resolution fMRI 3D Stack of Spirals data. So far I can reconstruct the images in the CPU, but it is slow. I am also looking to perform off-resonance correction for these reconstructions, I expect with this the computation times and memory usage will go even higher.

I believe the NFFT.jl package includes CuNFFT.jl, which should allow for GPU acceleration. I was wondering if there are any plans on fully implementing GPU acceleration in MRIReco?

Thank you,

[tknopp]

Do you have some comparative reconstruction (i.e. BART, Gadgetron, ...) which is faster? I just want to get a feeling what but it is slow means. Also: is it just gridding or iterative reconstruction?

NFFT.jl is now one of the fastest NFFT library out there. Before you test CuNFFT.jl I would encourage you to try multi-threading. With 8-16 threads, the CPU implementation is currently likely to outperform CuNFFT.jl.

Having said that, you will likely need to tweak the NFFT setting a little bit. For instance you can lower the accuracy.

[monreal93]

Dear Tobias,

Thanks for your answer, you gave me the answer to speed up the reconstruction (using multi-threading) !

I am doing iterative reconstruction, using the "cgnr" solver. The dataset I am reconstructing has a trajectory with dimensions 3x491520 and k-data 491520x32. The only other approach I have used to reconstruct the data (without off-resonance correction) is using gpuNUFFT and a matlab cg SENSE function (3 min) vs MRIReco (7 min). After reading your answer, I tried multi-threading with 16 threads, time went down to (1.8min), so faster than gpuNUFFT!

I have to mention that the gpuNUFFT reconstructions are being performed in my personal laptop with 16GB RAM, 4 dual-cores, and NVIDIA GeForce 930MX GPU (which is where I have it installed). The MRIReco ones, in a powerful server we have in our lab with 500GB RAM, 18 dual-cores, and 4 GPUs. So quite a powerful one, I am looking to get the best out of it for faster reconstructions.

The other concern I have is memory allocation. Using the @time macro from julia and reconstructing without B0 correction, I get a memory allocation of 120GiB and 570GiB for the B0 corrected one. So the server is able to run this, but takes most/all of the resources.

This being said, do you recommend me giving it a try to CuNFFT.jl ? Not sure how much MRIReco.jl would have to be changed to completely work with it. Also, are there any other tips you could give me to make the best out of our server and speed up the reconstructions?

Thanks again for the help,

[tknopp]

The other concern I have is memory allocation. Using the @time macro from julia and reconstructing without B0 correction, I get a memory allocation of 120GiB and 570GiB for the B0 corrected one. So the server is able to run this, but takes most/all of the resources.

That is hard to judge. @time is not a meaningful tool to track memory allocations since it also reports unrelated memory allocations that are released directly (e.g. during compilation). What is more interesting is to monitor the allocation with top or htop. I actually don't really have a good feeling if we have some larger memory issues.

Before trying out CuNFFT.jl it would be good to first do some profiling and find out, where the time is spend. Profiling is very simple with https://github.com/timholy/ProfileView.jl Its just important to run the profiler twice and look at the second run.

My hope is that 99% of the time is spend within the NFFT. If not we have other opportunities.

If the NFFT is the bottleneck there are different ways forward. Depending on the sizes, the Toeplitz operator can be faster, although our experience in https://arxiv.org/pdf/2101.12624.pdf was that this is not always the case. In case you stick with non-Toplitz the next question is how you chose sigma and m for the NFFT. I think our default parameters are already pretty aggressive (sigma=1.25, m=3) but this should be double checked.

In addition to this analysis I suggest the following:

• Setup a dummy NFFT operator and bechmark NFFT.jl, CuNFFT.jl and gpuNFFT on just that operator. I actually would recommend that you also try out https://github.com/flatironinstitute/cufinufft which claims to be the fastest out there.
  Having some numbers would help us to understand where CuNFFT.jl stands and if it is worth integrating into MRIReco.jl. An alternative is that we wrap cufinufft, which might not be such a big deal.