can't open directory: /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin

[Slowlysun]

When I check whether WarpSTR works correctly by running: bash run_test_case.sh , I get this traceback:

ONT Guppy basecalling software version 6.0.6+8a98bbc
config file: /work/software/guppy/ont-guppy-cpu/data/dna_r9.4.1_450bps_hac.cfg
model file: /work/software/guppy/ont-guppy-cpu/data/template_r9.4.1_450bps_hac.jsn
input path: test/test_output/Human_STR_1108232/fast5/test_run1
save path: test/test_output/Human_STR_1108232/fast5/test_run1/aux
chunk size: 2000
chunks per runner: 256
minimum qscore: 9
records per file: 4000
num basecallers: 1
cpu mode: ON
threads per caller: 2
Found 10 fast5 files to process.
Init time: 333 ms

0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|

---

Caller time: 172533 ms, Samples called: 1468186, samples/s: 8509.59
Finishing up any open output files.
Basecalling completed successfully.
find test/test_output/Human_STR_1108232/fast5/test_run1/aux/workspace -maxdepth 1 -name *.fast5 -print0 | xargs -0 cp -t test/test_output/Human_STR_1108232/fast5/test_run1/annot;
qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
DEBUG_3_TREX: Extracting TRs - running with 2 threads
Traceback (most recent call last):
File "WarpSTR.py", line 91, in
run()
File "WarpSTR.py", line 71, in run
overview_df, collapsed_df = main_wrapper(locus, main_config.threads)
File "/work/software/warpstr/src/caller/wrapper.py", line 20, in main_wrapper
workload = get_workload(df_overview, locus.path)
File "/work/software/warpstr/src/caller/wrapper.py", line 52, in get_workload
norm_signal = fast5.get_data_processed((row.l_start_raw, row.r_end_raw))
File "/work/software/warpstr/src/schemas/fast5.py", line 52, in get_data_processed
self.handle['Raw']['Reads'][rname]['Signal'])
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/numpy/core/_asarray.py", line 102, in asarray
return array(a, dtype, copy=False, order=order)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 1005, in array
self.read_direct(arr)
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 966, in read_direct
self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (can't open directory: /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin)

This looks like the software is missing files.

[xsitarcik]

This looks like some OS misconfiguration. It looks for the hdf5 plugin on /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin path but it looks like that it does not exist there. Can you check the path if it really exists? Also, what is your operating system? Have you installed WarpSTR using conda?

[Slowlysun]

I installed WarpSTR using conda.
/work/software/anaconda3/envs/warpstr/lib/hdf5/plugin path is not exist, even I can't find /work/software/anaconda3/envs/warpstr/lib/hdf5 path

This looks like some OS misconfiguration. It looks for the hdf5 plugin on /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin path but it looks like that it does not exist there. Can you check the path if it really exists? Also, what is your operating system? Have you installed WarpSTR using conda?

[xsitarcik]

I would try to activate the installed warpstr conda environment conda activate warpstr and there install hdf5 again conda install hdf5. If conda tells you that the requested packages are already installed that OS probably did some interference. Without knowing how to reproduce your problem it will be hard to help. Can you provide more information? What is your OS? Conda version? etc.

[Slowlysun]

1. My operating system is Ubuntu 18.04.6;
2. Conda version is 23.1.0;
3. I run conda activate warpstr , and then conda install hdf5, but it still get that wrong:

DEBUG_2_EXT: found these bamfiles [('test/test_input/test_run1', 'test/test_input/test_run1/mapping/mapping.bam', 'chr4:183178378-183178421', 'test_run1', 'test_run1', 'test/test_output/Human_STR_1108232/fast5')] in test/test_input
DEBUG_2_EXT: Running with 2
DEBUG_2_EXT: There are 10 in test/test_input/test_run1
DEBUG_2_EXT: All raw fast5 files have been successfully extracted
calling script /work/software/guppy/ont-guppy-cpu/bin/guppy_basecaller --input_path test/test_output/Human_STR_1108232/fast5/test_run1 --save_path test/test_output/Human_STR_1108232/fast5/test_run1/aux --flowcell FLO-MIN106 --kit SQK-LSK109 --fast5_out --cpu_threads_per_caller 2
ONT Guppy basecalling software version 6.0.6+8a98bbc
config file: /work/software/guppy/ont-guppy-cpu/data/dna_r9.4.1_450bps_hac.cfg
model file: /work/software/guppy/ont-guppy-cpu/data/template_r9.4.1_450bps_hac.jsn
input path: test/test_output/Human_STR_1108232/fast5/test_run1
save path: test/test_output/Human_STR_1108232/fast5/test_run1/aux
chunk size: 2000
chunks per runner: 256
minimum qscore: 9
records per file: 4000
num basecallers: 1
cpu mode: ON
threads per caller: 2
Found 10 fast5 files to process.
Init time: 399 ms

0% 10 20 30 40 50 60 70 80 90 100%
|----|----|----|----|----|----|----|----|----|----|

---

Caller time: 137743 ms, Samples called: 1468186, samples/s: 10658.9
Finishing up any open output files.
Basecalling completed successfully.
find test/test_output/Human_STR_1108232/fast5/test_run1/aux/workspace -maxdepth 1 -name *.fast5 -print0 | xargs -0 cp -t test/test_output/Human_STR_1108232/fast5/test_run1/annot;
qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
DEBUG_3_TREX: Extracting TRs - running with 2 threads
Traceback (most recent call last):
File "WarpSTR.py", line 91, in
run()
File "WarpSTR.py", line 71, in run
overview_df, collapsed_df = main_wrapper(locus, main_config.threads)
File "/work/software/warpstr/src/caller/wrapper.py", line 20, in main_wrapper
workload = get_workload(df_overview, locus.path)
File "/work/software/warpstr/src/caller/wrapper.py", line 52, in get_workload
norm_signal = fast5.get_data_processed((row.l_start_raw, row.r_end_raw))
File "/work/software/warpstr/src/schemas/fast5.py", line 52, in get_data_processed
self.handle['Raw']['Reads'][rname]['Signal'])
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/numpy/core/_asarray.py", line 102, in asarray
return array(a, dtype, copy=False, order=order)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 1005, in array
self.read_direct(arr)
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/h5py/_hl/dataset.py", line 966, in read_direct
self.id.read(mspace, fspace, dest, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 192, in h5py.h5d.DatasetID.read
File "h5py/_proxy.pyx", line 112, in h5py._proxy.dset_rw
OSError: Can't read data (can't open directory: /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin)

[xsitarcik]

I just checked and I did not have any problems using mentioned versions. However I used older guppy version, maybe that is the problem. Have you used guppy of that version before? Do you had any problems? I do not have experience with newer guppy versions but maybe the problem is that it produces .fast5 files that are VBZ compressed, and thus requiring decompressing plugin. So I would try to install that plugin.

I see that conda now offers a package for that, so this should be easier option. Just activate the warpstr environment and run: conda install ont_vbz_hdf_plugin -c bioconda, this should install the plugin at /work/software/anaconda3/envs/warpstr/hdf5/lib/plugin/ and also properly set the env variable HDF5_PLUGIN_PATH to that path. Please verify by echo $HDF5_PLUGIN_PATH if it was set correctly, or modify it to match the path where plugin was installed.

The other option is to download the plugin from https://github.com/nanoporetech/vbz_compression/releases, install it somewhere and modify the $HDF5_PLUGIN_PATH to point to the directory where you install it. Or you can just create the missing directory /work/software/anaconda3/envs/warpstr/lib/hdf5/plugin and install the plugin there.

[Slowlysun]

Thank you very much, I can successfully run the scritp after I installed plugin. But when I set ‘motif’ instead of ‘sequence’, there is still an error.

..........................
Finishing up any open output files.
Basecalling completed successfully.
find test/test_output/Human_STR_1108232/fast5/test_run1/aux/workspace -maxdepth 1 -name *.fast5 -print0 | xargs -0 cp -t test/test_output/Human_STR_1108232/fast5/test_run1/annot;
Finished Guppy basecalling for test/test_output/Human_STR_1108232/fast5/test_run1
qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
DEBUG_3_TREX: Extracting TRs - running with 16 threads
Traceback (most recent call last):
File "WarpSTR.py", line 91, in
run()
File "WarpSTR.py", line 71, in run
overview_df, collapsed_df = main_wrapper(locus, main_config.threads)
File "/work/software/warpstr/src/caller/wrapper.py", line 22, in main_wrapper
call_wrapper = CallerWrapper(locus, threads)
File "/work/software/warpstr/src/caller/wrapper.py", line 66, in init
template_seq, reverse_seq = self.get_seqs(locus.sequence, load_flanks(locus.path))
File "/work/software/warpstr/src/caller/wrapper.py", line 75, in get_seqs
rev = flanks.reverse.left + self.reverse_uniq_sequence(sequence) + flanks.reverse.right
File "/work/software/warpstr/src/caller/wrapper.py", line 83, in reverse_uniq_sequence
rev += tmpl.ENCODING_DICT[i]
KeyError: 't'

[xsitarcik]

Looks like that you specified lowercase characters in motif, there must be only uppercase characters (at least in this version)

[Slowlysun]

I used the config file of the test(config_test.yaml), but still reported an error.

loci:

• name: Human_STR_1108232
  coord: chr4:183178378-183178421
  motif: AAAT

[xsitarcik]

When a motif is provided instead of sequence, WarpSTR produces the sequence automatically from the reference file using locus coordinates. In your case reference file contained lowercase characters somewhere in that locus region. We did not test such a case so we missed this. Thank you. We released a bugfix and now it should be working with lowercase letters found in the reference fasta. So please run git pull in the cloned repository and run the test again.

[Slowlysun]

After run git pull, I run python WarpSTR.py test/config_test.yaml, but I get another error:

.................................
Finished Guppy basecalling for test/test_output/Human_STR_1108232/fast5/test_run1
qt.qpa.xcb: X server does not support XInput 2
failed to get the current screen resources
DEBUG_3_TREX: Extracting TRs - running with 16 threads
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/work/software/warpstr/src/caller/wrapper.py", line 289, in warpstr_call_parallel
return warpstr.run(input_data.signal)
File "/work/software/warpstr/src/caller/caller.py", line 133, in run
rescaled_signal2 = rescale_signal(rescaled_signal, resc_alignment)
File "/work/software/warpstr/src/caller/caller.py", line 311, in rescale_signal
spline = interpolate.splrep(filt_val, filt_exp, s=len(filt_val))
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/scipy/interpolate/fitpack.py", line 291, in splrep
res = _impl.splrep(x, y, w, xb, xe, k, task, s, t, full_output, per, quiet)
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/site-packages/scipy/interpolate/_fitpack_impl.py", line 461, in splrep
raise TypeError('m > k must hold')
TypeError: m > k must hold
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "WarpSTR.py", line 93, in
run()
File "WarpSTR.py", line 73, in run
overview_df, collapsed_df = main_wrapper(locus, main_config.threads)
File "/work/software/warpstr/src/caller/wrapper.py", line 23, in main_wrapper
results = call_wrapper.run(workload)
File "/work/software/warpstr/src/caller/wrapper.py", line 109, in run
results = pool.map(warpstr_call_parallel, workload)
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/work/software/anaconda3/envs/warpstr/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
TypeError: m > k must hold

[xsitarcik]

Was this error produced when you set the motif in the config? Further, when you used the sequence in the configuration file have you run into this error?
If you used motif in the config, you can check the automatically produced sequence. There is a sequence.txt file in the test_output directory, the relative path from the cloned repo should be test/test_output/Human_STR_1108232/sequence.txt. Please verify that the automatically produced sequence is correct, it should be (AAAT).

[Slowlysun]

When I used the sequence in the configuration file, it works fine. But when I used the motif, I got that error.
When I used motif in the config, I can find test/test_output/Human_STR_1108232/sequence.txt.

$ cat test/test_output/Human_STR_1108232/sequence.txt
AAATAAATAAATAAATAAATAAATAAATAAATAAATAAATAAAT

[xsitarcik]

Well that is the problem, it should be (AAAT), so it means WarpSTR generated sequence incorrectly from the motif and reference locus. We released another bugfix addressing the issue when the reference fasta file is of a mix of upper and lowercase characters. Please run git pull again and rerun the test.
I would also thank you for your patience and for providing feedback. Really appreciated.

[Slowlysun]

I run git pull and return the test, it works well.
Then I run my own data, there are AAATG and AAATA two motifs, AAATA can get result, and the file of sequence.txt is (AAATA)AAAT, but AAATG still reports the same error, and the file of sequence.txt is AAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAATAAAAT. Can this software distinguish between two motifs that are similar?

[xsitarcik]

These two motifs that you mention are at one locus? or it is two loci, each with its own motif? Please provide the loci configuration so I can verify if it is ok.