Skip to content

Commit 6f71966

Browse files
arkalarkal
authored
Merge pull request #277 from /issues/274-add-troubleshooting
Add troubleshooting to MANUAL.md (resolves #274)
2 parents 719a1e2 + 1289790 commit 6f71966

1 file changed

Lines changed: 110 additions & 10 deletions

File tree

‎MANUAL.md‎

Lines changed: 110 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -65,14 +65,14 @@ Install Toil
6565

6666
pip install toil[aws]==3.5.2
6767

68-
Install ProTECT and all dependencies in the virtualenv
69-
70-
pip install protect
71-
7268
Install packaging (required if setuptools>=39.0.1)
7369

7470
pip install packaging
7571

72+
Install ProTECT and all dependencies in the virtualenv
73+
74+
pip install protect
75+
7676
### Method 2 - Installing from Source
7777

7878
This will install ProTECT in an editable mode.
@@ -81,19 +81,21 @@ Obtain the source from Github
8181

8282
git clone https://www.github.com/BD2KGenomics/protect.git
8383

84-
Create a virtualenv in the project folder
84+
Create and activate a virtualenv in the project folder (Important since the Makefile checks for
85+
this and will fail if it detects that you are not in a virtual environment)
8586

8687
cd protect
8788
virtualenv venv
88-
89-
Activate the virtualenv
90-
9189
source venv/bin/activate
9290

9391
Install Toil and pytest
9492

9593
make prepare
9694

95+
Install packaging (required if setuptools>=39.0.1)
96+
97+
pip install packaging
98+
9799
Install ProTECT
98100

99101
make develop
@@ -263,7 +265,7 @@ These describe options that are used universally by most tools/jobs in the workf
263265
(`False`), or were they encrypted with individual
264266
per-file keys hashed from the master sse_key
265267
(`True`)
266-
gdc_download_token: /path/to/token.txt -> If any of teh input files are being pulled from
268+
gdc_download_token: /path/to/token.txt -> If any of the input files are being pulled from
267269
the NCBI GDC, this token is required to access
268270
the data. This file should be present at the same
269271
location on all workers in the workflow.
@@ -502,4 +504,102 @@ ProTECT is registered on Dockstore.org and can be run using Dockstore with the f
502504
dockstore tool launch --debug --entry quay.io/ucsc_cgl/protect:<version tag> --json protect.json
503505
`
504506

505-
The example protect.json file provided in this repo describes a run where the input reference files are on the local file system at `/home/ubuntu/protect-work/protect-reference-files/hg38-reference-files` and the input fastqs are in `/home/ubuntu/protect-work/samples/`.
507+
The example protect.json file provided in this repo describes a run where the input reference files
508+
are on the local file system at `/home/ubuntu/protect-work/protect-reference-files/hg38-reference-files`
509+
and the input fastqs are in `/home/ubuntu/protect-work/samples/`.
510+
511+
# Troubleshooting
512+
## Installation
513+
#### 1. packaging related issues
514+
If you see
515+
516+
Traceback (most recent call last):
517+
File "<string>", line 1, in <module>
518+
File "/mnt/dstew/protect/setup.py", line 6, in <module>
519+
from packaging.version import LegacyVersion as _LegacyVersion
520+
ImportError: No module named packaging.version
521+
522+
523+
Then it is mostly due to a version of setuptools >= 39.0.1. The fix for this is to run the
524+
following line before reattempting to install protect.
525+
526+
pip install packaging
527+
528+
## Errors during Runtime
529+
### How to read and understand an error arising during/at the end of the run
530+
Since ProTECT is written in TOIL, the error printed on a failed job is the logfile for the worker
531+
that failed to execute the python function that was the failed job.
532+
533+
An example error is shown here with line numbers manually added at the beginning for annotation
534+
purposes:
535+
536+
01: The job seems to have left a log file, indicating failure: 'merge_mhc_peptide_calls' g/f/jobO4yiE4
537+
02: g/f/jobO4yiE4 ---TOIL WORKER OUTPUT LOG---
538+
03: g/f/jobO4yiE4 INFO:toil:Running Toil version 3.8.0-4c83830e4f42594d995e01ccc07b47396b88c9e7.
539+
04: g/f/jobO4yiE4 INFO:toil.fileStore:Starting job ('merge_mhc_peptide_calls' g/f/jobO4yiE4) with ID (972acf421c864831d756dec528bb9cc2a4d3c281).
540+
05: g/f/jobO4yiE4 INFO:toil.fileStore:LOG-TO-MASTER: Merging MHC calls
541+
06: g/f/jobO4yiE4 Traceback (most recent call last):
542+
07: g/f/jobO4yiE4 File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/worker.py", line 340, in main
543+
08: g/f/jobO4yiE4 job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
544+
09: g/f/jobO4yiE4 File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1289, in _runner
545+
10: g/f/jobO4yiE4 returnValues = self._run(jobGraph, fileStore)
546+
11: g/f/jobO4yiE4 File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1234, in _run
547+
12: g/f/jobO4yiE4 return self.run(fileStore)
548+
13: g/f/jobO4yiE4 File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1406, in run
549+
14: g/f/jobO4yiE4 rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
550+
15: g/f/jobO4yiE4 File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/binding_prediction/common.py", line 566, in merge_mhc_peptide_calls
551+
16: g/f/jobO4yiE4 raise RuntimeError('No peptides available for ranking')
552+
17: g/f/jobO4yiE4 RuntimeError: No peptides available for ranking
553+
18: g/f/jobO4yiE4 ERROR:toil.worker:Exiting the worker because of a failed job on host sjcb10st7
554+
19: g/f/jobO4yiE4 WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'merge_mhc_peptide_calls' g/f/jobO4yiE4 with ID g/f/jobO4yiE4 to 0
555+
20: g/f/jobO4yiE4 WARNING:toil.jobGraph:We have increased the default memory of the failed job 'merge_mhc_peptide_calls' g/f/jobO4yiE4 to 2147483648 bytes
556+
557+
* Line 1 tells you which function failed.
558+
* Line 6 is where the actual error starts (the line starting with `Traceback`)
559+
* Line 16 tells you the tool failed with a RuntimeError and contains the actual error message. In
560+
this case, the tool found no peptides that were available for ranking.
561+
* The line numbers will vary with different tools and dependin gon how jobs were scheduled, however
562+
a rule of thumb is to look for the string `RuntimeError` (the most common error thrown by ProTECT)
563+
or just `Error` and then read the error message.
564+
565+
### A note on non-deterministic errors with RADIA
566+
It has been noticed that RADIA sometimes fails non-deterministically on one (or rarely more)
567+
chromosome in a sample during a ProTECT run. The error is highly non-informative since we currently
568+
do not store logs from tools (see BD2KGenomics/protect#275). The error looks similar to this:
569+
570+
Z/O/job1uH92D ---TOIL WORKER OUTPUT LOG---
571+
Z/O/job1uH92D INFO:toil:Running Toil version 3.8.0-4c83830e4f42594d995e01ccc07b47396b88c9e7.
572+
Z/O/job1uH92D INFO:toil.fileStore:Starting job ('run_filter_radia' Z/O/job1uH92D) with ID (0e8bb5ea5d785324b5abce7b384bf5b99140aa6c).
573+
Z/O/job1uH92D WARNING:toil.fileStore:LOG-TO-MASTER: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrec
574+
tly requested resources. Job 'run_filter_radia' Z/O/job1uH92D used 104.10% (34.3 GB [36816449536B] used, 32.9 GB [35367908263B] requested) at the end of its run.
575+
Z/O/job1uH92D Traceback (most recent call last):
576+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/worker.py", line 340, in main
577+
Z/O/job1uH92D job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
578+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1289, in _runner
579+
Z/O/job1uH92D returnValues = self._run(jobGraph, fileStore)
580+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1234, in _run
581+
Z/O/job1uH92D return self.run(fileStore)
582+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/dev/toil_clean/src/toil/job.py", line 1406, in run
583+
Z/O/job1uH92D rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
584+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/mutation_calling/radia.py", line 238, in run_filter_radia
585+
Z/O/job1uH92D tool_version=radia_options['version'])
586+
Z/O/job1uH92D File "/home/ucsc/arjun/tools/protect_toil_clean/local/lib/python2.7/site-packages/protect/common.py", line 138, in docker_call
587+
Z/O/job1uH92D 'for command \"%s\"' % ' '.join(call),)
588+
Z/O/job1uH92D RuntimeError: docker command returned a non-zero exit status (1)for command "docker run --rm=true -v /scratch/bio/ucsc/toil-681c097c-61da-4687-b734-c5051f0aa19f/tmped2fnu/f041f939-5c0d-40be-a884-68635e929d09:/data --log-driver=none aarjunrao/filterradia:bcda721fc1f9c28d8b9224c2f95c440759cd3a03 TCGA-CH-5788 17 /data/radia.vcf /data /home/radia/scripts -d /data/radia_dbsnp -r /data/radia_retrogenes -p /data/radia_pseudogenes -c /data/radia_cosmic -t /data/radia_gencode --noSnpEff --noBlacklist --noTargets --noRnaBlacklist -f /data/hg38.fa --log=INFO -g /data/radia_filtered_chr17_radia.log"
589+
Z/O/job1uH92D ERROR:toil.worker:Exiting the worker because of a failed job on host sjcb10st1
590+
Z/O/job1uH92D WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'run_filter_radia' Z/O/job1uH92D with ID Z/O/job1uH92D to 0
591+
592+
The fix for this is to just rerun protect with the same jobstore (if you haven't specified
593+
`--clean always`) with the `--restart` flag. I.e. run exactly the same command as the original run
594+
with `--restart` appended at the end of the command.
595+
596+
For example, if your command was:
597+
598+
ProTECT --config_file test.yaml --workDir /tmp/working/protect_test /tmp/jobstores/protect_test
599+
600+
then your restart command becomes:
601+
602+
ProTECT --config_file test.yaml --workDir /tmp/working/protect_test /tmp/jobstores/protect_test \
603+
--restart
604+
605+

0 commit comments

Comments
 (0)