Setting up the Environment

Using the cwltool in a SLURM Cluster

CWL workflows can be run in a number of platforms, from laptops to HPC clusters. In particular, at Duke GCB we have a large SLURM cluster for computing tasks. Ideally, the corresponding engine to run CWL pipelines in HPC clusters is Toil. However, by the time of writing this wiki, there are still open issues (e.g. 1261, 886) that jeopardize its usage. Once those issues get solved, we will update this wiki accordingly.

Install forked cwltool

Currently, we are using a forked and patched version of cwltool. The few patches fix a problem with temporary directories PR 60 as well as a problem with parsing draft-3 workflows PR 168. It can be installed as follows:

git clone https://github.com/alexbarrera/cwltool.git
cd cwltool
git checkout slurm
python setupy.py install

If the installation went OK, you should be able to run cwltool:

cwltool -h

Software and programs used in the workflows

In CWL, software dependencies can be specified in the Requirements or Hints (non-blocking requirements) sections of the documents. In GGR pipelines, each workflow and command-line tool declares which programs are needed as DockerRequirement in the Hints section. Here is an example extracted from macs2-callpeak.cwl:

hints:
  - class: DockerRequirement
    dockerImageId: 'dukegcb/macs2'

If you are using a cloud-based runner for CWL (e.g. Arvados), this hint can be used to pull the dukegcb/macs2 Docker image and run a container with it.

If you are running CWL workflows in a non cloud-based system like a SLURM cluster, those programs have to be already installed and available (e.g. in the $PATH, more on this later). The majority of the software used in these pipelines are well known and stablished programs, which are listed in the required software section. Additionally, there are a few in-house scripts that can be found in the GGR-Docker project.

git clone https://github.com/alexbarrera/GGR-Docker.git

From this directory, the following files should be added to the $PATH:

• windowtrimmer/src/windowTrimmer.py: python script that intends to crawl through bed files to filter out reads that are too concentrated within a specified base-pair window.
• overrepresented_sequence_extract/src/overrepresented_sequence_extract.py: python script that takes as input a fastqc report and outputs a fasta file of overrepresented/adapter sequences for their subsequent trimming.
• workflow-utils/bin/: directory containing a miscellanea of executable bash scripts to compute basic shell commands.