Tkurth/stats revamp (#31)

* some improvements todata processing scripts, including multi dimensional virtual concatenation, making it easier to extend exiting datasets

* fixing tests

* rewritten stats calculation, now running on GPU and using NCCL for comms. Also, improved comms pattern

* cleanup

Author: azrael417

data_process/concatenate_dataset.py

@@ -40,8 +40,8 @@ def concatenate(input_dirs: List[str], output_file: str, metadata: dict, channel
     Parameters
     ----------
     input_dir: List[str]
-        directories to where the dataset files are located which are to be concatenated. 
-        Files inside that list will concatenated in time and files from different lists will be concatenated in channel dimension.
+        List of directories directories in which the dataset files are located which are to be concatenated. 
+        Files inside a sublist will be concatenated in time and files from different lists will be concatenated in channel dimension.
     output_file: str
         file name of the concatenated dataset, has to include the full path. 
     metadata : dict