A better memory layout for IMEX methods

[MarcoArtiano]

At the moment, we store the variables as follows

du[var, i, j, k, element],

where i, j and k loops over the local node of the given element. This structure makes quite hard to implement IMEX schemes (implicit in the vertical direction), because accessing to the vertical direction components is not straightforward and quite complex.

Some prototype of IMEX schemes in TrixiAtmo (not yet public) at the moment relies on defining a view of the current du structure, through a mapping that transforms the structure as

du_wrap[var, i, element_z, element_horizontal],

where i is the local node in the vertical direction, element_z are the element in the vertical direction, and element_horizontal are the elements in orthogonal direction to the vertical one (that includes $$x$$ and $$y$$ for a 3D example). (Similarly to what CGDycore.jl does)

On the other hand, changing the memory layout of du could reduce the performance of the Trixi.rhs! (I'm not actually sure about that).
Therefore, that may also not be the best option or solution.

[sloede]

There have been several optimizations regarding the memory layout on GPUs. Also, in some cases, we have found that pre-computing primitive variables can speed up computations.

I'd say that using the current layout + views is probably a good strategy for now. However, if you feel that this becomes a bottleneck, you can try to verify this hypothesis by locally modifying the memory layout and then benchmark your kernels to see if it improves performance.

[MarcoArtiano]

There have been several optimizations regarding the memory layout on GPUs. Also, in some cases, we have found that pre-computing primitive variables can speed up computations.

I'd say that using the current layout + views is probably a good strategy for now. However, if you feel that this becomes a bottleneck, you can try to verify this hypothesis by locally modifying the memory layout and then benchmark your kernels to see if it improves performance.

Thanks Michael! I do also think that at the moment the best option is current layout + views. Regarding the pre-computation of primitive variables, do you mean what is currently done in the turbo volume kernel or there's somewhere an implementation that uses also that for the surface fluxes? One may pre-compute the primitive variables once and also the surface fluxes may gain some speed up.