Check FASTQ files for readnames not-recommended characters

[mmokrejs]

The page https://ega-archive.org/submission/tools/egacryptor#UsingClient states:

Using the EgaCryptor
Before encrypting your files please make sure that any files that will be uploaded to EGA do not use special characters such as # ? ( ) [ ] / \ = + < > : ; " ' , * ^ | & as the use of these can interfere with the archiving process in addition to causing problems within end users' pipelines. 

I assume it actually speaks of filenames, not of chars inside the files themselves. Probably the check could be done by this alternative tool as well. Umm, well, the java tools evidently does not do this type of filename check either, which is awkward too.

[mmokrejs]

No, the requirement is probably speaking of readnames in FASTQ files and maybe even about the FASTQ file encoding. From https://ega-archive.org/submission/sequence I quote:

The read names must have a suffix identifying the first and second read from the pair, for example '/1' and '/2' (regular expression for the reads "^(.*)([\\.|:|/|_])([12])$").

Quality scores must be in Phred scale. For example, quality scores from early Solexa pipelines must be converted to use this scale. Both ASCII and space delimitered decimal encoding of quality scores are supported. We will automatically detect the Phred quality offset of either 33 or 64.

[vinjana]

Hi @mmokrejs,

the doc quote indeed is ambiguous by itself. Unfortunately, we do not have any experience with any of these chars in filenames, but many of the listed example symbols are special characters in the shell. And provided that in FASTQ format both in the read ID and in the scores / is allowed (and in the Phred even more of the listed forbidden symbols) and that one has no control of the "characters" in compressed or encrypted files, I'm pretty sure that the first quote does refer to filenames only.

The encryptor doc lists only some examples of forbidden chars. A more conservative approach would be to only allow certain characters and then, of course, the question is which characters should be allowed? Only [a-zA-Z0-9.-_]? This could also be made configurable, of course.

I think this needs to be checked for the second column of the map-file.txt that contains the eventual names of the uploaded files and is the argument to 0-generate-links.sh.