Inconsistent mapping to reference results (minimap2)

[fredericgaspar]

Hi!
When I use minimap2 in Geneious (minimap2_macos -x map-ont --frag=yes --secondary=no -a ref.fasta query.fasta -o output.sam) or with the command line (minimap2 -x map-ont --frag=yes --secondary=no -a ref.fasta query.fasta -o output.sam), all my contigs (167) map to my reference genome.
When I use RagTag with minimap2 only 155 map to the reference (ragtag.py scaffold -r -g 1 -f 500 --aligner /home/ubuntu/miniconda3/envs/ragtag/bin/minimap2 --mm2-params '-x map-ont' ref.fasta query.fasta).
Any (obvious) reason for the difference?
Thank you for your help!

[malonge]

Hi there,

It looks like you are using different parameters. You can try the following:

ragtag.py scaffold -r -g 1 -f 500 --aligner /home/ubuntu/miniconda3/envs/ragtag/bin/minimap2 --mm2-params '-x map-ont --frag=yes --secondary=no' ref.fasta query.fasta

[fredericgaspar]

That was fast! 🤩 I tried that option previously ('-x map-ont --frag=yes --secondary=no') but I got an error message (below), hence the shift to '-x map-ont'. Thu Dec 30 00:31:55 2021 --- VERSION: RagTag v2.1.0 Thu Dec 30 00:31:55 2021 --- CMD: ragtag.py scaffold -r -g 1 -f 500 --aligner /home/ubuntu/miniconda3/envs/ragtag/bin/minimap2 --mm2-params -x map-ont --frag=yes --secondary=no ref.fasta query.fasta Thu Dec 30 00:31:55 2021 --- WARNING: Without '-u' invoked, some component/object AGP pairs might share the same ID. Some external programs/databases don't like this. To ensure valid AGP format, use '-u'. Thu Dec 30 00:31:55 2021 --- INFO: Mapping the query genome to the reference genome Traceback (most recent call last): File "/home/ubuntu/miniconda3/envs/ragtag/bin/ragtag_scaffold.py", line 577, in <module> main() File "/home/ubuntu/miniconda3/envs/ragtag/bin/ragtag_scaffold.py", line 420, in main al.run_aligner() File "/home/ubuntu/miniconda3/envs/ragtag/lib/python3.7/site-packages/ragtag_utilities/Aligner.py", line 126, in run_aligner if all([self.params_are_valid(), self.exec_is_valid()]): File "/home/ubuntu/miniconda3/envs/ragtag/lib/python3.7/site-packages/ragtag_utilities/Aligner.py", line 199, in params_are_valid raise ValueError("Alignments must not be in SAM format (-a).") ValueError: Alignments must not be in SAM format (-a). 😓

…

On 30 Dec 2021, at 00:19, Michael Alonge ***@***.***> wrote: Hi there, It looks like you are using different parameters. You can try the following: ragtag.py scaffold -r -g 1 -f 500 --aligner /home/ubuntu/miniconda3/envs/ragtag/bin/minimap2 --mm2-params '-x map-ont --frag=yes --secondary=no' ref.fasta query.fasta — Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARVUVLMEEY5AHVYI4KRP2BTUTOQSDANCNFSM5K6YHJSA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.

[malonge]

Ah ok you have identified a bug in the way RagTag parses the aligner parameters. It currently cannot handle double hyphen command line parameters, such as --frag=yes. I will fix this and included in the next release.

In the meantime, I would just run minimap2 externally and plug it in manually, as described in #86

Sorry about that!

[malonge]

Specifically, I think it would fail using any double hyphen parameter with an "a" or "c" character in it since those trigger a warning.

[fredericgaspar]

Thank you for taking the time to look at this… ☺️ I performed the alignment in minimap2 externally with an output as *.paf minimap2 -x map-ont --frag=yes --secondary=no ref.fasta query.fasta > ragtag.scaffold.asm.paf Created the folder ragtag_output and moved the ragtag.scaffold.asm.paf file inside Performed: ragtag.py scaffold -r -u -g 1 -f 500 ref.fasta query.fasta Everything went fine with no bug/crash. But the problem remains the same as initially… Minimap2 correctly maps 167/167 contigs to the reference, but RagTag only uses 155 of those mapped contigs for the scaffold (not using the res / no contigs were smaller than 500).

…

On 30 Dec 2021, at 01:00, Michael Alonge ***@***.***> wrote: Specifically, I think it would fail using any double hyphen parameter with an "a" or "c" character in it since those trigger a warning. — Reply to this email directly, view it on GitHub <#92 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARVUVLKGJLYA4B7H6GKX4WTUTOVLXANCNFSM5K6YHJSA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.

[malonge]

I suggest running ragtag with --debug, which will output a ragtag.scaffold.debug.merged.paf file. These are the alignments actually used for scaffolding after filtering/merging. Check how many of the contigs have alignments in this file and that should tell us what is going on. I suspect that the alignments for these missing contigs just got filtered out by RagTag.

[fredericgaspar]

I just looked at the alignments and you are correct! The missing contigs (as seen at the end of the ragtag.scaffold.agp output file, sizes between 511 and 1845) were indeed filtered out by ragtag… 😓 They were in the ragtag.scaffold.asm.paf file (in yellow in the pdf in attachment)… Remained in the ragtag.scaffold.debug.unfiltered.paf… And are missing in the filtered and merged files… Also there are 2 contigs with the same name that map to the beginning and end of the chromosome and 1 of them was also filtered out (2 first lines in yellow in the pdf in attachment) (maybe I can’t have 1 sequence broken in 2 sequences with the same name)... Do I have to play with some filtering/scaffolding options to solve the filtering out or maybe in this case RagTag is not the ideal tool? [ragtag_scaffold_debug_compilation.pdf](https://github.com/malonge/RagTag/files/7793287/ragtag_scaffold_debug_compilation.pdf) [ragtag.scaffold.agp.txt](https://github.com/malonge/RagTag/files/7793290/ragtag.scaffold.agp.txt)

[malonge]

1. Yes each sequence should have a unique header in the fasta files.

2. There are some parameters that can be tuned for filtering and merging:

-f
--remove-small
-q
-d
-i
-a
-s

While parameter tuning is sometimes warranted, alignments are generally filtered out for a good reason. If you are very confident that these alignments should remain because they represent unambigous matches to the reference, then you can try to tune parameters or even use a different aligner. But it may also just be the case that the contigs are not long enough to be confidently placed by any homology-based scaffolding software via any whole-genome aligner. Its better to leave a contig unplaced than to inaccurately place it.

[fredericgaspar]

Thank you so much (again) for taking the time to provide all these clarifications... In my specific case I do need to use all contigs for statistics purposes...
However, I do see your point and for sure I will use RagTag in the future! 🤓👍🏼
(Happy New Year!)

[malonge]

Thanks and happy new year! Let's keep this open until I fix this bug.