GPU Memory Issue- How does KomaMRI manage the memory usage?

[timnotavailable]

Hey there! I have a question regarding the KomaMRI bloch simulation GPU memory.

Think about a EPI which has resolution (256,256) , suppose you do not need additional many spins for T2*, so the spin number is 256x256=65536, if you have adc is (256,256), then your final array would be (65536,65536,3) (3 represent for Mx,My,Mz, though KomaMRI uses Cayley-Klein complex parameters to propagate however the Complex float is equivalent to 2 float), so this array is quite big, then how do you manage this GPU memory issue? For every event you dump out the memory or?

[cncastillo]

Hi! Thanks for the question. We use different types of memory, for the Bloch simulation method, most computations are done with GPU register memory, so the only use of GPU global memory is to store the phantom (rho, T1, T2, etc per spin) and the spin state Mag (Mx, My, Mz per spin), check: https://github.com/JuliaHealth/KomaMRI.jl/blob/master/KomaMRICore/src/simulation/SimMethods/Bloch/gpu/ExcitationKernel.jl. Furthermore, to step the simulation in time, we separate into simulation blocks, this is particularly relevant for the precession blocks where we do a warp reduction so we need to allocate Nspins x Ntimes_per_block in local memory, check: https://github.com/JuliaHealth/KomaMRI.jl/blob/master/KomaMRICore/src/simulation/SimMethods/Bloch/gpu/PrecessionKernel.jl. This last part is controlled by sim_params[Nblocks], more blocks mean less time steps per block. In the future, I will probably change the parameter to something like sim_params["max_block_length"] which a little easier to control.

Depending on the problem that you have I can describe what to do, or the technical details further. I would advice you to read the following, as I might have oversimplified the explanation:

• Explanation of GPU code: https://juliahealth.org/KomaMRI.jl/dev/explanation/7-gpu-explanation/
• How to deal with things going out of memory: https://juliahealth.org/KomaMRI.jl/dev/how-to/4-run-distributed-simulations/

If you have memory problems you can always simulate obj[1:N] and obj[N+1:end] and then add the results together. That is what you do for multi-GPU or multi-node systems (parallel), or you can simulate the first part and then the other one (sequential).

[timnotavailable]

Hi , thanks ! Really helpful answer!