Use `torch.distributed` as alternative communication backend for Heat

[mrfh92]

Related
As a first step, one might separate the mpi4py-wrappers more clearly from the Heat-communication structures; see, e.g., #1243 and the draft PR #1265

Feature functionality
Currently, communication in Heat is based on MPI via mpi4py; this is very much standard in traditional HPC. In machine learning / deep learning HPC, NCCL/RCCL communication is more standard and seems to be advantageous in particular for GPU-GPU-communication. The most easy way to support this as well and -at the same time- to improve interoperability with PyTorch, would be to allow torch.distributed as alternative backend for communication.

So far, a communicator in Heat is actually an mpi4py-communicator, and communication routines are actually communication routines in mpi4py.
I would like to suggest to keep the API for Heat-communication, but to allow for another backend. Fortunately, the communication in torch.distributed is quite MPI-inspired. The main difference is that no ...v-operations are supported; to deal with them, workaround need to be create.

The overall idea would be that one can run a Heat code script.py both via mpirun -n 4 python script.py or torchrun --nproc-per-node=4 script.py (or similar) and the required backend is chosen automatically.

[mrfh92]

Some additional problems might arise as isend and irecv seem to be the only non-blocking operations.

[github-actions]

Branch features/1772-Use_torch_distributed_as_alternative_communication_backenend_for_Heat created!

[Berkant03]

after discussing this with @Markus-Goetz @mrfh92 @JuanPedroGHM @ClaudiaComito we decided not to accomodate an alternative torch.distributed backend in our communication layer.

using torch.distributed has only upsides in terms of nccl, as it does not need to use the infiband as much as the cuda aware mpi and can give a significant performance uplift. But this comes at the cost of not having any v functions available in torch.distributed, which are needed. The workaround would be to write our own v functions but this would come at a big performance penalty.
Therefore it would be better to integrate Heat in a different way. This being to write a Heat Aware DistributedSampler allowing for non uniformally split arrays to be used for training, without rewriting the complete communication stack.