Is R version important?

[robertw-hihm]

Hi, I'm using a conda env with python 2.7 and when I enter "R --version" it lists the OS version 4.3. Within the py2_7 conda env, I see that the lowest R pkg available is now 3.3.1 (i.e, via typing "conda search R"). So my question is whether the RefReeDMA script will work with R version 4.3 or even 3.3.1. Thanks.

[jklughammer]

Hi, there is no specific reason we are requiring that R version other than that it is the R version it has been thoroughly tested with.
If you can install and load all the R packages that are listed in the instructions chances are high that it will be fine.

[robertw-hihm]

Hi again, and thanks for your prompt response! I’ve created a conda environment (py2_7) for RefFreeDMA and tried the test run after modifying the …test.cfg file for tools_paths and working_dir, but I received the following error only for the python libraries but not for the required tools Checking for required python libraries... python Bio ... OK python pysam ... FAIL python bitarray ... FAIL python guppy ... FAIL Checking for required tools... samtools ... OK bedtools ... OK trim_galore ... OK cutadapt ... OK bowtie2 ... OK bsmap ... OK samToFastq.jar ... OK I notice that the …test.cfg file doesn’t list a path for the python library directories in /tools/python2.7/ but perhaps this is not needed(?) I assumed the RefFreeDMA.sh script would be using the libraries from folders provided in tools, but perhaps I rather need to install them in the py2_7 env using conda? Please advise. Thanks. Kind regards, Rob Wilson Professor of Molecular Biology Department of Biotechnology INN University Phone: +47 95 04 91 96 E-mail: ***@***.******@***.***> Visiting address: Office 245, Biohus, Holsetgate 22, Hamar, NORWAY Mailing address: PO Box 400, NO-2418 Elverum, NORWAY Website: eng.inn.no<http://eng.inn.no/> (English); www.inn.no<http://www.inn.no/> (norsk) Fra: jklughammer ***@***.***> Dato: mandag, 26. juni 2023 kl. 20:29 Til: jklughammer/RefFreeDMA ***@***.***> Kopi: Rob Wilson ***@***.***>, Author ***@***.***> Emne: Re: [jklughammer/RefFreeDMA] Is R version important? (Issue #4) Hi, there is no specific reason we are requiring that R version other than that it is the R version it has been thoroughly tested with. If you can install and load all the R packages that are listed in the instructions chances are high that it will be fine. — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHPFJPLV7F237UIR3L6LL3XNHIHRANCNFSM6AAAAAAZT7QXNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jklughammer]

Hi, my best guess is that the pre-compiled packages in the tools bundle that don't work (one of them does) are not compatible with the exact python2.7 version you are using. I would recommend trying an older version of python2.7 like python2.7.8. If you find the version that works, please let me know and I will update in the readme.
It is fine that the python subfolder is not specified in the tools path - it is taken care of by RefFreeDMA.

[robertw-hihm]

Dear Professor Klughammer, Thanks for your response. I typed python at the prompt of my py2_7 env and was surprised to see python 3.9 initiated. After quitting, I tried python2.7 and the correct python (2.7) was initiated. I then modified the RefFreeDMA.sh script, changing all instances of python -c to python2.7 -c and now the script runs. However it elicits as new (unexpected and, for me, inexplicable error as I count all 12 samples having been included…):

…

-------Read preparation------- 12 samples selected for reference generation: |Hum_1_G||Hum_1_L||Hum_1_M||Hum_2_G||Hum_2_L||Hum_2_M||Hum_3_G||Hum_3_L||Hum_3_M||Hum_4_G||Hum_4_L||Hum_4_M| Unconverted tag: no unconv_tag BSF_0116_C55NLACXX_3__Hum_2_L submitted BSF_0116_C55NLACXX_3__Hum_2_M submitted BSF_0116_C55NLACXX_3__Hum_3_M submitted BSF_0117_C5578ACXX_5__Hum_1_L submitted BSF_0117_C5578ACXX_5__Hum_1_M submitted BSF_0117_C5578ACXX_5__Hum_4_G submitted BSF_0117_C5578ACXX_5__Hum_4_M submitted BSF_0117_C5578ACXX_6__Hum_2_G submitted BSF_0117_C5578ACXX_6__Hum_4_L submitted BSF_0121_C5CTMACXX_7__Hum_3_G submitted BSF_0121_C5CTMACXX_8__Hum_1_G submitted BSF_0121_C5CTMACXX_8__Hum_3_L submitted Didn't find the expected number of uniq.ref files (12) or trimmed.fq files (12). Exiting! (RefFreeDMA_py2_7) ***@***.*** RefFreeDMA-master]$ Thanks again for your help! Kind regards, Rob Fra: jklughammer ***@***.***> Dato: tirsdag, 27. juni 2023 kl. 11:34 Til: jklughammer/RefFreeDMA ***@***.***> Kopi: Rob Wilson ***@***.***>, Author ***@***.***> Emne: Re: [jklughammer/RefFreeDMA] Is R version important? (Issue #4) Hi, my best guess is that the pre-compiled packages in the tools bundle that don't work (one of them does) are not compatible with the exact python2.7 version you are using. I would recommend trying an older version of python2.7 like python2.7.8. If you find the version that works, please let me know and I will update in the readme. It is fine that the python subfolder is not specified in the tools path - it is taken care of by RefFreeDMA. — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHPFJMRSI6H4GMQDHF26SDXNKSL7ANCNFSM6AAAAAAZT7QXNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jklughammer]

Have you tried running it in local mode first? Scheduling systems can be different for different cluster and it might be necessary to adjust the scheduling commands to your cluster's setup if you want to use the parallel mode.
If the local mode works then you know that RefFreeDMA works but you'll need to adjust the scheduling commands.

[jklughammer]

If the local mode also doesn't work it might be that your python2.7 hack fixed the initial error but messed with some later steps. In that case I would recommend to create an environment where pyhton2.7 is the only python and gets called as python.

[robertw-hihm]

Local mode was worse, not making it past the library and tool checks. I’ve created a new py2_7 env that calls python 2.7.16 as the only python, then used the original RefFreeDMA.sh script, which works up to the same (second) spot (but in local mode doesn’t make it past the first spot, the library and tool checks). In the log, the (second) error is described thusly:

>> Now performing quality (cutoff 20) and adapter trimming in a single pass for the adapter sequence: 'AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC' from file /data/Rob/RefFreeDMA_test/fastq/BSF_0116_C55NLACXX_3__Hum_2_L.fastq <<<

Traceback (most recent call last): File "/data/Rob/tools//cutadapt_1.8.3//bin/cutadapt", line 9, in <module> from cutadapt.scripts import cutadapt File "/data/Rob/tools/cutadapt_1.8.3/lib/python2.7/site packages/cutadapt/scripts/cutadapt.py", line 63, in <module> check_importability() File "/data/Rob/tools/cutadapt_1.8.3/lib/python2.7/site-packages/cutadapt/__init__.py", line 9, in check_importability import cutadapt._align ImportError: /data/Rob/tools/cutadapt_1.8.3/lib/python2.7/site-packages/cutadapt/_align.so: undefined symbol: _Py_ZeroStruct RUN STATISTICS FOR INPUT FILE: /data/Rob/RefFreeDMA_test/fastq/BSF_0116_C55NLACXX_3__Hum_2_L.fastq ============================================= 2 sequences processed in total Sequences removed because they became shorter than the length cutoff of 16 bp: 0 (0.0%) Hum_2_L |Hum_1_G||Hum_1_L||Hum_1_M||Hum_2_G||Hum_2_L||Hum_2_M||Hum_3_G||Hum_3_L||Hum_3_M||Hum_4_G||Hum_4_L||Hum_4_M| Hum_2_L is selected for reference generation! ^TGG Hum_2_L_uniq.ref had filesize 0. Exiting! I couldn’t find python 2.7.8 even on the conda-forge channel. Any advice? Thanks again for your time. Best, Rob Fra: jklughammer ***@***.***> Dato: tirsdag, 27. juni 2023 kl. 13:09 Til: jklughammer/RefFreeDMA ***@***.***> Kopi: Rob Wilson ***@***.***>, Author ***@***.***> Emne: Re: [jklughammer/RefFreeDMA] Is R version important? (Issue #4) If the local mode also doesn't work it might be that your python2.7 hack fixed the initial error but messed with some later steps. In that case I would recommend to create an environment where pyhton2.7 is the only python and gets called as python. — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHPFJPJU2ED5YNXVZPQGRLXNK5QBANCNFSM6AAAAAAZT7QXNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jklughammer]

here is a link to python2.7.8: https://www.python.org/downloads/release/python-278/

@dariarom94 ran RefFreeDMA more recently and still has working environment. She will comment as soon as she gets to it.

[robertw-hihm]

Dear Johanna, According to ”google,” the cutadapt error was attributed to having not built the extensions locally, so I decided to trash the conda env and start fresh with locally installed tools. A conda environment running python 2.7.18 with the libraries present in the RefFreeDMA tools directory and conda-installed cutadapt 1.18, samtools 1.2, picard 2.7.1 (although I had to use the picard_path=$tool_path/picard-tools_1.118/ since samToFastq.jar was renamed to picard.jar in v.2.7.1 and I couldn’t find a conda repo with picard 1.118), trim-galore 0.4.1, bowtie 2.24, bsmap 2.90, bwa 0.7.8 and bwameth 0.20 (with all paths pointing to /miniconda3/envs/py27_18/bin/) allowed RefFreeDMA.sh to process files through to here: -------Differential methylation analysis for gpg motifs------- ls: klarte ikke å åpne '/data/Rob/RefFreeDMA_test/toSelf_filtered_0.08mm_final_concat/diffMeth_*/*_diff_meth.tsv': Ingen slik fil eller filkatalog (= No such file or directory) Differential methylation analysis failed. Didn't find the expected number of diff_meth.tsv files (1). Exiting! This surprises me as the bsmap version 2.90 is the same as provided by tools… I am, however, very pleased that the deduced reference genome was generated as this allows me to use it in the Bismark and DMRcaller pipeline with which I am most familiar. One question: I have already trimmed (trim-galore) and fastQC’d validated .fq.gz read files (unmerged). Can I simply comment out the “READ_PREPARATION_START “section in RefFreeDMA.sh and rather place these fq.gz files in the appropriate folder of the working directory for further processing to generate the deduced reference in section REFERENCE_GENERATION_START through Sample CrossMapping_END ? Another question: My R1 and R2 fq.gz files are large (ca. 10 GB each) as they stem from RRBS of the huge Scots Pine genome (ca. 24 Gb) using MspI/DpnII (I’ll alter the config file accordingly). Is it paramount to include ALL the samples (28) in the ref. genome generation, or can this be produced using a subset for then analyzing the entire set with Bismark and DMRcaller. Thanks again for your time and help! Best, Rob Fra: jklughammer ***@***.***> Dato: tirsdag, 27. juni 2023 kl. 15:20 Til: jklughammer/RefFreeDMA ***@***.***> Kopi: Rob Wilson ***@***.***>, Author ***@***.***> Emne: Re: [jklughammer/RefFreeDMA] Is R version important? (Issue #4) here is a link to python2.7.8: https://www.python.org/downloads/release/python-278/ — Reply to this email directly, view it on GitHub<#4 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABHPFJISWQ4FYLHSUDJOISTXNLMZPANCNFSM6AAAAAAZT7QXNQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jklughammer]

Hi Rob, great that you managed to get it that far. Are you sure it was bsmap that failed? It might have actually succeeded that step and failed in the R follow-up analysis (maybe again due to version conflicts).

You can for sure adapt RefFreeDMA to your needs and omit certain processes. However, you will have to look carefully at the respective scripts and only disable the specific steps you don't want. For example the read preparation step does more than just trim the reads. If possible in any way it would be best to run as is.

We've successfully run RefFreeDMA for Axolotl (30 GB genome). We had less samples, though and it needs a lot more memory and time. You can indeed specify a subset of samples that you want to use for reference generation: Choose good samples and ideally samples that represent all the different states that you have in your dataset but you can for example remove replicates.

Also, RefFreeDMA is not built to work with paired-end reads and we use bsmap because it is a wildcard aligner instead of 3-letter (like bismark). But if you have relatively long reads that should be fine. We built RefFreeDMA to work with 50bp SE (as a cheaper sequencing setup). And the reference generation will not be able to take the PE setup into account - paired reads will be seen as independent reads and so they also couldn't be mapped in PE mode. WE have never tested it with PE reads so please be extra carful and do some sanity checks on the generated deduced reference.

Best,
Johanna