Description
Summary
This is a feature request discussed during the Triqs meeting 2024 in Paris. I would like to have functionality to exploit the total memory distributed over compute nodes in an MPI parallell job to store and work with Triqs Green's functions, e.g.
in combination with an all-to-all transpose operation, this would allow to implement large scale GW like calculations.
Motivation
The lattice calculations in TPRF are today memory limited to the node local memory, even when running calculations on multiple nodes, since there is no direct TRIQS support to distribute Green's function objects in memory over multiple nodes.
On the request of @Wentzell this is an attempt to briefly summarise the idea.
Implementation
The basic containers and operations envisaged to enable this type of Triqs library meshes and Green's functions would be some form of multidimensional tensor with one or more dimension spread across MPI ranks, in combination with a global transpose operation, that re-arranges the tensor, from distributed spread over one dimension to another.
In the spirit of
N = 10**4
A_tt = array((N, N), spread_axes=(0,))
A_tw = fft(A_tt, axis=1)
A_tw = redistribute(A_tw, spread_axes=(1,))
A_ww = fft(A_tw, axis=0)
one known implementation of this idea had been done in the FFTW3 library, to implement 3D fast Fourier transforms, see https://fftw.org/fftw3_doc/FFTW-MPI-Transposes.html
It seems like some of this might have been ironed out by @cnyeh in the AIMBES project.