Description of feature
Currently some processes, like MERGE_PREDICTIONS use a custom concat_tsv.py script enabling streamed merging of many cvs, as prediction tables tend to get very large and multiple 100ed files are concatenated.
To reduce maintenance load we could implement an nf-core module for:
https://csvkit.readthedocs.io/en/latest/contributing.html#streaming-versus-buffering
Description of feature
Currently some processes, like
MERGE_PREDICTIONSuse a customconcat_tsv.pyscript enabling streamed merging of many cvs, as prediction tables tend to get very large and multiple 100ed files are concatenated.To reduce maintenance load we could implement an nf-core module for:
https://csvkit.readthedocs.io/en/latest/contributing.html#streaming-versus-buffering