fast5_out option no longer available

[alanfwilliams]

I was excited to read the WarpSTR paper and apply it to my work, but it seems that just after acceptance, ONT removed the fast5_out option from guppy which is used here. Is there a chance that the application can be modified to run without that option?

[tvinar]

We will look into this, we are aware of the fact that this happened. The guppy annotated fast5 files are needed because they are generally the only way of maintaining some level mapping between the raw signal and the basecall. WarpSTR needs this information to locate the proper section of the raw signal for further analysis. We have some ongoing work internally to re-establish this with newer versions of guppy basecaller, but we don't have anything ready at the moment. As of now, I would recommend using an older version of guppy (there is really not that much of a difference). Are you working on Linux? Best, Tomas -- -------------------------------------------------------------------------- Tomas Vinar, Associate Professor Department of Applied Informatics Faculty of Mathematics, Physics, and Informatics Comenius University, Bratislava E-mail: ***@***.*** Office: M163 Work Phone: +421-2-60295207

…

On Tue, Jul 11, 2023 at 12:28 PM Alan Williams ***@***.***> wrote: I was excited to read the WarpSTR paper and apply it to my work, but it seems that just after acceptance, ONT removed the fast5_out option from guppy which is used here <https://github.com/fmfi-compbio/warpstr/blob/4ea248e307aef7f8cd038b16bb9fe6825ebabc0e/src/guppy_annotate/guppy_annotate.sh#L21>. Is there a chance that the application can be modified to run without that option? — Reply to this email directly, view it on GitHub <#5>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAV5KY6TYNB27FNW3BW3N23XPWSNXANCNFSM6AAAAAA2GOVNZA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[alanfwilliams]

Yes, I'm running this on a Ubuntu machine. I downloaded guppy 6.3.8, and tried running the test set. It seems to complete (everything is in the output folder), but it "freezes" my computer. It seems it is trying to open some window (I assume this is to show the graphical output?, I haven't dug in the code). Given that it worked mostly ok, I tried running it on our own data by replacing everything in the test folder with our own and changing the configuration. However, I got this traceback:

DEBUG_2_EXT: found these bamfiles [('test/test_input/test_run1', 'test/test_input/test_run1/mapping/mapping.bam', 'chr4:3074877-3074933', 'test_run1', 'test_run1', 'test/test_output/HTT/fast5')] in test/test_input
DEBUG_2_EXT: Running with 2
DEBUG_2_EXT: There are 61630 in test/test_input/test_run1
calling script /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/bin/guppy_basecaller --input_path test/test_output/HTT/fast5/test_run1 --save_path test/test_output/HTT/fast5/test_run1/aux --flowcell FLO-MIN106 --kit SQK-LSK109 --fast5_out --cpu_threads_per_caller 2
CRASHPAD MESSAGE: 
ONT Guppy basecalling software version 6.3.8+d9e0f64, minimap2 version 2.22-r1101
config file:        /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg
model file:         /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/data/template_r9.4.1_450bps_hac.jsn
input path:         test/test_output/HTT/fast5/test_run1
save path:          test/test_output/HTT/fast5/test_run1/aux
chunk size:         2000
chunks per runner:  256
minimum qscore:     9
records per file:   4000
num basecallers:    1
cpu mode:           ON
threads per caller: 2

Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA.
The EULA may be found in /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/bin
Warning: fast5_out is deprecated - emitting fast5 files from guppy will be removed in a future version
Found 36634 input read files to process.
Init time: 371 ms

0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************************************
Caller time: 26317773 ms, Samples called: 225596173, samples/s: 8572.01
Finishing up any open output files.
Basecalling completed successfully.
find test/test_output/HTT/fast5/test_run1/aux/workspace -maxdepth 1 -name *.fast5 -print0 | xargs  -0 cp -t test/test_output/HTT/fast5/test_run1/annot;
DEBUG_3_TREX: Extracting TRs - running with 2 threads
Traceback (most recent call last):
  File "/home/alan/warpstr/WarpSTR.py", line 91, in <module>
    run()
  File "/home/alan/warpstr/WarpSTR.py", line 71, in run
    overview_df, collapsed_df = main_wrapper(locus, main_config.threads)
  File "/home/alan/warpstr/src/caller/wrapper.py", line 22, in main_wrapper
    call_wrapper = CallerWrapper(locus, threads)
  File "/home/alan/warpstr/src/caller/wrapper.py", line 66, in __init__
    template_seq, reverse_seq = self.get_seqs(locus.sequence, load_flanks(locus.path))
  File "/home/alan/warpstr/src/caller/wrapper.py", line 75, in get_seqs
    rev = flanks.reverse.left + self.reverse_uniq_sequence(sequence) + flanks.reverse.right
  File "/home/alan/warpstr/src/caller/wrapper.py", line 83, in reverse_uniq_sequence
    rev += tmpl.ENCODING_DICT[i]
KeyError: 'g'

This may be due to this run being on the config dna_r10.4.1_e8.2_400bps_5khz_sup, with SQK-LSK114 and FLO-FLG114 vs. what was supplied to guppy as I wasn't sure how to get the old guppy to recognize the new kits and such.

[tvinar]

Hello Alan, I didn't realize this from you first question, but it seems that you are trying to run warpstr on data produced with V14 chemistry. There are several substantial changes compared to R9.4.1, which was the mainstream for years, and for which warpstr was developed, including: - ONT removing options from guppy basecaller needed for raw signal analysis (this is the first issue you ran into) - substantial increase in size of k-mers that need to be considered in R10 chemistry - ONT not providing up-to-date pore signal models for the new chemistries - unavailability of benchmark data on which these tools could be tested I think, we will eventually adapt our tools to the new V14 chemistry, but I cannot give you any time estimate at the moment. I am Ccing Jozef, who has been the main developer and maintainer for WarpSTR, in case he hase any more positive news to add. Best, Tomas

…

-- -------------------------------------------------------------------------- Tomas Vinar, Associate Professor Department of Applied Informatics Faculty of Mathematics, Physics, and Informatics Comenius University, Bratislava E-mail: ***@***.*** Office: M163 Work Phone: +421-2-60295207

On Fri, Jul 14, 2023 at 6:02 PM Alan Williams ***@***.***> wrote: Yes, I'm running this on a Ubuntu machine. I downloaded guppy 6.3.8, and tried running the test set. It seems to complete (everything is in the output folder), but it "freezes" my computer. It seems it is trying to open some window (I assume this is to show the graphical output?, I haven't dug in the code). Given that it worked mostly ok, I tried running it on our own data by replacing everything in the test folder with our own and changing the configuration. However, I got this traceback: DEBUG_2_EXT: found these bamfiles [('test/test_input/test_run1', 'test/test_input/test_run1/mapping/mapping.bam', 'chr4:3074877-3074933', 'test_run1', 'test_run1', 'test/test_output/HTT/fast5')] in test/test_input DEBUG_2_EXT: Running with 2 DEBUG_2_EXT: There are 61630 in test/test_input/test_run1 calling script /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/bin/guppy_basecaller --input_path test/test_output/HTT/fast5/test_run1 --save_path test/test_output/HTT/fast5/test_run1/aux --flowcell FLO-MIN106 --kit SQK-LSK109 --fast5_out --cpu_threads_per_caller 2 CRASHPAD MESSAGE: ONT Guppy basecalling software version 6.3.8+d9e0f64, minimap2 version 2.22-r1101 config file: /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/data/dna_r9.4.1_450bps_hac.cfg model file: /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/data/template_r9.4.1_450bps_hac.jsn input path: test/test_output/HTT/fast5/test_run1 save path: test/test_output/HTT/fast5/test_run1/aux chunk size: 2000 chunks per runner: 256 minimum qscore: 9 records per file: 4000 num basecallers: 1 cpu mode: ON threads per caller: 2 Use of this software is permitted solely under the terms of the end user license agreement (EULA).By running, copying or accessing this software, you are demonstrating your acceptance of the EULA. The EULA may be found in /home/alan/Downloads/ont-guppy_6.3.8_linux64/ont-guppy/bin Warning: fast5_out is deprecated - emitting fast5 files from guppy will be removed in a future version Found 36634 input read files to process. Init time: 371 ms 0% 10 20 30 40 50 60 70 80 90 100% |----|----|----|----|----|----|----|----|----|----| *************************************************** Caller time: 26317773 ms, Samples called: 225596173, samples/s: 8572.01 Finishing up any open output files. Basecalling completed successfully. find test/test_output/HTT/fast5/test_run1/aux/workspace -maxdepth 1 -name *.fast5 -print0 | xargs -0 cp -t test/test_output/HTT/fast5/test_run1/annot; DEBUG_3_TREX: Extracting TRs - running with 2 threads Traceback (most recent call last): File "/home/alan/warpstr/WarpSTR.py", line 91, in <module> run() File "/home/alan/warpstr/WarpSTR.py", line 71, in run overview_df, collapsed_df = main_wrapper(locus, main_config.threads) File "/home/alan/warpstr/src/caller/wrapper.py", line 22, in main_wrapper call_wrapper = CallerWrapper(locus, threads) File "/home/alan/warpstr/src/caller/wrapper.py", line 66, in __init__ template_seq, reverse_seq = self.get_seqs(locus.sequence, load_flanks(locus.path)) File "/home/alan/warpstr/src/caller/wrapper.py", line 75, in get_seqs rev = flanks.reverse.left + self.reverse_uniq_sequence(sequence) + flanks.reverse.right File "/home/alan/warpstr/src/caller/wrapper.py", line 83, in reverse_uniq_sequence rev += tmpl.ENCODING_DICT[i] KeyError: 'g' This may be due to this run being on the config dna_r10.4.1_e8.2_400bps_5khz_sup, with SQK-LSK114 and FLO-FLG114 vs. what was supplied to guppy as I wasn't sure how to get the old guppy to recognize the new kits and such. — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAV5KY6P46LXVCUW76BAQALXQFUR5ANCNFSM6AAAAAA2GOVNZA> . You are receiving this because you commented.Message ID: ***@***.***>

[frankmartiniv]

Are there any updates to this issue?

[tvinar]

Sorry, no update. As I have written earlier, for now we cannot support V14 chemistry and we do not have ETA on doing so at the moment.