Skip to content

Commit fcac942

Browse files
Merge branch 'release/v6.0.0'
2 parents ec3940d + 32bd01d commit fcac942

File tree

764 files changed

+150323
-59915
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

764 files changed

+150323
-59915
lines changed

.travis.yml

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
language: perl
22
perl:
3-
- "5.18"
3+
- "5.26"
44
sudo: required
55

66
install:
@@ -20,16 +20,25 @@ install:
2020
- conda config --set always_yes yes --set changeps1 no
2121
- conda update -q conda
2222
- conda info -a
23+
- conda config --add channels defaults
24+
- conda config --add channels conda-forge
25+
- conda config --add channels bioconda
26+
27+
## Download cpan libraries
28+
- cd definitions; cpanm --quiet --notest --installdeps .
29+
- cd -
2330

2431
script:
2532
## Run MIP install script
2633
- cd t; prove mip_install.t; cd -
27-
- perl mip_install.pl -env test_environment --quiet -pei --perl_skip_test --perl_modules_force -vep 87 --vep_auto_flag al --vep_plugin 0 --cnvnator_root_binary root_v5.34.34.Linux-ubuntu12-x86_64-gcc4.6.tar.gz
34+
- perl mip_install.pl --quiet --bash_set_errexit -env test_environment -vep 91 --vep_auto_flag al --vep_plugin 0 -snpg GRCh37.75 -skip gatk --bioconda fastqc=0.11.4
35+
# Set user specific path to data
36+
- echo 'cluster_constant_path:' /home/travis/build/${TRAVIS_REPO_SLUG}/t/data >> /home/travis/build/${TRAVIS_REPO_SLUG}/templates/mip_travis_config.yaml
2837
- bash mip.sh
2938
- source ~/.bash_profile
3039
- source ~/.bashrc
3140
- source activate test_environment
3241
- cd /lib/x86_64-linux-gnu
3342
- sudo ln -s "$ROOTSYS"/lib/*so .
3443
- cd -
35-
- cd t; perl run_tests.t -c ../templates/mip_travis_config.yaml
44+
- cd t; prove -r

Documentation/API/Modules.md

Lines changed: 0 additions & 2 deletions
This file was deleted.

Documentation/Api.md

Lines changed: 0 additions & 2 deletions
This file was deleted.

README.md

Lines changed: 110 additions & 104 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,15 @@ PMID:25495354
1717

1818
## Overview
1919

20-
MIP performs whole genome or target region analysis of sequenced single-end and/or paired-end reads from the Illumina plattform in fastq\(.gz\) format to generate annotated ranked potential disease causing variants.
20+
MIP performs whole genome or target region analysis of sequenced single-end and/or paired-end reads from the Illumina platform in fastq\(.gz\) format to generate annotated ranked potential disease causing variants.
2121

2222
MIP performs QC, alignment, coverage analysis, variant discovery and annotation, sample checks as well as ranking the found variants according to disease potential with a minimum of manual intervention. MIP is compatible with Scout for visualization of identified variants. MIP analyses snv, indels and SV.
2323

2424
MIP has been in use in the clinical production at the Clinical Genomics facility at Science for Life Laboratory since 2014.
2525

2626
## Example Usage
2727

28-
```
28+
```Bash
2929
$ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --pedigree_file [family_id_pedigree.yaml]
3030
```
3131

@@ -34,6 +34,7 @@ $ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --ped
3434
* Installation
3535
* Simple automated install of all programs using conda/SHELL via supplied install script
3636
* Downloads and prepares references in the installation process
37+
* Handle conflicting tool dependencies
3738
* Autonomous
3839
* Checks that all dependencies are fulfilled before launching
3940
* Builds and prepares references and/or files missing before launching
@@ -45,7 +46,7 @@ $ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --ped
4546
* Creates internal queues at nodes to optimize processing
4647
* Minimal IO between nodes and login node
4748
* Flexible:
48-
* Design your own workflow by turning on/off relevant modules
49+
* Design your own workflow by turning on/off relevant modules
4950
* Restart an analysis from anywhere in your workflow
5051
* Process one, or multiple samples using the module\(s\) of your choice
5152
* Supply parameters on the command line, in a pedigree.yaml file or via config files
@@ -68,7 +69,7 @@ $ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --ped
6869
* Test data output existens and integrity using automated tests
6970
* Annotation
7071
* Gene annotation
71-
* Summarise over all transcript and output on gene level
72+
* Summarize over all transcript and output on gene level
7273
* Transcript level annotation
7374
* Separate pathogenic transcripts for correct downstream annotation
7475
* Annotate all alleles for a position
@@ -80,127 +81,124 @@ $ mip --family_id [family_id] --pbwa_mem 1 --config_file [mip_config.yaml] --ped
8081
* Use standard formats whenever possible
8182
* Visualization
8283
* Ranks variants according to pathogenic potential
83-
* Output is directly compatibel with [Scout](https://github.com/Clinical-Genomics/scout)
84+
* Output is directly compatible with [Scout](https://github.com/Clinical-Genomics/scout)
8485

8586
## Getting Started
8687

8788
### Installation
8889

89-
MIP is written in perl and therfore requires that perl is installed on your OS.
90+
MIP is written in perl and therefore requires that perl is installed on your OS.
9091

91-
#### Automated Installation \(Linux x86\_64\)
92-
This installation procedure assumes that you have a working perl version (>= 5.10) and a [Miniconda]
93-
installation.
92+
#### Prerequisites
93+
* [Perl], version 5.26.0 or above
94+
* [Cpanm](http://search.cpan.org/~miyagawa/App-cpanminus-1.7043/lib/App/cpanminus.pm)
95+
* [Miniconda]
9496

95-
1. Install MIP
96-
Clone the official git repository
97-
```
98-
$ git clone https://github.com/henrikstranneheim/MIP.git
97+
We recommend perlbrew for installing and managing perl and cpanm libraries. Installation instructions and setting up specific cpanm libraries can be found [here](https://github.com/Clinical-Genomics/development/blob/master/perl/installation/installation.md).
98+
99+
#### Automated Installation \(Linux x86\_64\)
100+
##### 1.Clone the official git repository
101+
102+
```Bash
103+
$ git clone https://github.com/Clinical-Genomics/MIP.git
99104
$ cd MIP
100105
```
101-
#### *Optional*
102-
Test conda and mip_install
103-
```
106+
##### 2.Install required modules from cpan
107+
108+
```Bash
109+
$ cd definitions
110+
$ cpanm --installdeps .
111+
$ cd -
112+
```
113+
###### *Note:*
114+
- Some versions of the cpan module List::MoreUtils has conflicts with certain versions of Perl. Perl v 5.18.2 has been confirmed to be compatible with version 0.4.13 of the List::MoreUtils module.
115+
If you experience issues with that module, try to reinstall it by running: ``cpanm --reinstall List::[email protected] ``
116+
117+
##### 3.Test conda and mip_installation (optional)
118+
119+
```Bash
104120
$ cd t; prove mip_install.t
105121
$ cd -
106122
```
107123

108-
2. Create the install instructions for MIP
109-
```
124+
##### 4.Create the install instructions for MIP
125+
```Bash
110126
$ perl mip_install.pl
111127
```
112-
This will generate a batch script "mip.sh" for the install in your working directory. Use ``--help`` to see
113-
parameters that can be used in the installation process.
114-
#### *Conda*
115-
You can decide to install in the conda default environment or use a conda environment with ``--env [env_name]``.
116-
If you have installed conda in another location than the default you have to supply the path to the location
117-
using ``--conda_dir_path [conda_directory_path]``.
118-
#### *Perl*
119-
MIP requires perl version (>=5.18) and the installation process will upgrade the perl version to at least 5.18 for the user
120-
if you enable ``--perl_install``. Cpanm will be installed if you install a new perl version and used to download required
121-
perl modules. Currently MIP does not use the conda perl installation, but installs perl and cpanm outside of conda.
122-
##### *NOTE*
123-
This will add the following lines to bashrc and bash_profile if the install perl version is not found in your path:
124-
```
125-
'export PATH=$HOME/perl-PERLVERSION/:$PATH' >> ~/.bashrc
126-
'eval `perl -I ~/perl-PERLVERSION/lib/perl5/ -Mlocal::lib=~/perl-PERLVERSION/`' >> ~/.bash_profile
127-
'export PERL_UNICODE=SAD' >> ~/.bash_profile
128-
```
129-
#### *References*
130-
MIP requires many references depending on what modules in MIP you decide to run. MIP ships with a download script
131-
that will attempt to download references that are available in public repositories. This feature can be enables with
132-
by supplying a ``--reference_dir [reference_dir]`` in the installation process.
133-
##### *NOTE*
134-
Some references are quite large and will take time to download. You might want to run this using screen or tmux.
135-
136-
3. Run the bash script
137-
```
128+
This will generate a batch script "mip.sh" for the install in your working directory.
129+
130+
###### *Note:*
131+
- The batch script will install the MIP dependencies in Conda's root environment. Often it is beneficial to create a separate environment for each of your applications. In order to create a separate environment for MIP supply the ``-env [env_name]`` flag when running *mip_install.pl*.
132+
133+
- There is currently an issue with installing GATK via Conda causing the installation process to freeze. The solution at the moment is to turn of the GATK installation by running the install script together with the ``-skip gatk`` flag and install [GATK] manually.
134+
135+
- For a full list of available options and parameters, run: ``$ perl mip_install.pl --help``
136+
137+
- For a full list of parameter defaults, run: ``$ perl mip_install.pl -ppd``
138+
139+
##### 5.Run the bash script
140+
141+
```Bash
138142
$ bash mip.sh
139143
```
140-
This will install all the dependencies of MIP and other modules included in MIP into a conda environment.
141-
However a fresh version of perl and cpanm is installed, if enabled, outside of the conda environment, but are activated through bashrc and bash_profile.
142-
#### *Optional*
143-
Make sure to activate your conda environment if that option was used above.
144-
Test Perl modules and MIP
145-
```
146-
$ cd t; prove run_tests.t
144+
This will install MIP and most of its dependencies into a conda environment.
145+
146+
###### *Note:*
147+
- Some references are quite large and will take time to download. You might want to run this using screen or tmux.
148+
149+
##### 6.Test your MIP installation (optional)
150+
151+
Make sure to activate your conda environment if that option was used above.
152+
153+
```Bash
154+
$ cd t; prove -r
147155
$ cd -
148156
```
157+
##### 7.Install tools with conflicting dependencies
158+
Tools that have conflicting dependencies needs to be installed in separate conda environments. Currently these programs requires separate environments:
149159

150-
4. Run MIP
160+
* Genmod, Chanjo, Multiqc and Variant_integrity
161+
- requires python 3
162+
* Peddy
163+
- conflicts with SVDB dependencies
164+
* CNVnator
165+
- Requires access to ROOT which disturbs the normal linking of C libraries
151166

152-
*Conda default environment*
153-
```
154-
$ mip
155-
```
156-
*Conda environment*
157-
```
158-
$ source activate conda_env
159-
$ mip
160-
```
161167

162-
### Prerequisites
163-
164-
##### Programs/Modules
165-
166-
* Perl modules: There are several perl modules that are used by MIP which are not included in the Perl standard distribution. These need to be downloaded from [CPAN]. The install script will attempt to download all required perl modules and you can also list them and programs using ``perl mip_install -ppd``.
167-
* Simple Linux Utility for Resource Management \(SLURM\)
168-
* Fastqc
169-
* Bwa
170-
* Sambamba
171-
* Samtools
172-
* Bedtools
173-
* Picardtools
174-
* Chanjo
175-
* Plink
176-
* Peddy
177-
* GATK
178-
* Freebayes
179-
* Manta
180-
* Delly
181-
* Cnvnator
182-
* TIDDIT
183-
* Svdb
184-
* Vt
185-
* VEP
186-
* vcfParser \(Supplied with MIP\)
187-
* Snpeff
188-
* Snpsift
189-
* Annovar
190-
* Vcfanno
191-
* Genmod
192-
* Multiqc
193-
* Tabix
194-
* Gzip
168+
```bash
169+
## Python 3 tools
170+
$ perl mip_install.pl -env mip_pyv3.6 --python_version 3.6 --select_program genmod --select_program chanjo --select_program variant_integrity --select_program multiqc
171+
$ bash mip.sh
195172

196-
##### Meta-Data
173+
## Peddy
174+
$ perl mip_install.pl -env mip_peddy --select_program peddy
175+
$ bash mip.sh
197176

198-
* [Pedigree file] \(YAML-format\)
199-
* [Configuration file] \(YAML-format\)
177+
## CNVnator
178+
$ perl mip_install.pl -env mip_cnvnator --select_program cnvnator
179+
$ bash mip.sh
180+
```
181+
182+
In your config yaml file or on the command line you will have to supply the ``module_source_environment_command`` parameter to activate the conda environment specific for the tool. Here is an example with three Python 3 tools in their own environment and Peddy and VEP and CNVnator in each own, with some extra initialization:
183+
184+
```Yml
185+
module_source_environment_command:
186+
pchanjo_sexcheck: "source activate mip_pyv3.6"
187+
pcnvnator: "LD_LIBRARY_PATH=[CONDA_PATH]/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source [CONDA_PATH]/envs/mip_cnvnator/root/bin/thisroot.sh; source activate mip_cnvnator"
188+
pmultiqc: "source activate mip_pyv3.6"
189+
ppeddy: "source activate mip_peddy"
190+
prankvariant: "source activate mip_pyv3.6"
191+
psv_rankvariant: "source activate mip_pyv3.6"
192+
psv_varianteffectpredictor: "LD_LIBRARY_PATH=[CONDA_PATH]/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source activate mip_vep"
193+
pvarianteffectpredictor: "LD_LIBRARY_PATH=[CONDA_PATH]/lib/:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; source activate mip_vep"
194+
pvariant_integrity: "source activate mip_pyv3.6"
195+
```
196+
197+
MIP will execute this on the node before executing the program and then revert to the ``--source_main_environment_command`` if set. Otherwise ``source deactivate`` is used to return to the conda root environment.
200198
201199
### Usage
202200
203-
MIP is called from the command line and takes input from the command line \(precedence\ or falls back on defaults where applicable.
201+
MIP is called from the command line and takes input from the command line \(precedence\) or falls back on defaults where applicable.
204202
205203
Lists are supplied as repeated flag entries on the command line or in the config using the yaml format for arrays.
206204
Only flags that will actually be used needs to be specified and MIP will check that all required parameters are set before submitting to SLURM.
@@ -209,18 +207,24 @@ Program parameters always begins with "p" followed by a capital letter. Program
209207
210208
MIP will overwrite data files when reanalyzing, but keeps all "versioned" sbatch scripts for traceability.
211209
212-
You can always supply `perl mip.pl -h` to list all availaible parameters and defaults.
210+
You can always supply `perl mip.pl --help` to list all available parameters and defaults.
213211

214212
Example usage:
215-
```
216-
$ mip -f 3 -sampleid 3-1-1A,3-2-1U -sampleid 3-2-2U -pfqc 0 --pbwa_mem 2 -c 3_config.yaml
213+
```Bash
214+
$ mip -f 3 --sample_ids 3-1-1A --sample_ids 3-2-1U --sample_ids 3-2-2U -pfqc 0 --pbwa_mem 2 -c 3_config.yaml
217215
```
218216

219217
This will analyse family 3 using 3 individuals from that family and begin the analysis with programs after Bwa mem and use all parameter values as specified in the config file except those supplied on the command line, which has precedence.
220218

221219
#### Input
222220

223-
All references and template files should be placed directly in the reference directory specified by `--reference_dir`, except for ANNOVAR db files, which should be located in annovar/humandb.
221+
All references and template files should be placed directly in the reference directory specified by `--reference_dir`.
222+
223+
##### Meta-Data
224+
225+
* [Pedigree file] \(YAML-format\)
226+
* [Configuration file] \(YAML-format\)
227+
224228

225229
#### Output
226230

@@ -236,5 +240,7 @@ MIP will place any generated datafiles in the output data directory specified by
236240

237241
[Miniconda]: http://conda.pydata.org/miniconda.html
238242
[CPAN]: https://www.cpan.org/
239-
[Pedigree file]: https://github.com/henrikstranneheim/MIP/tree/master/templates/643594-miptest_pedigree.yaml
240-
[Configuration file]: https://github.com/henrikstranneheim/MIP/blob/master/templates/mip_config.yaml
243+
[Pedigree file]: https://github.com/Clinical-Genomics/MIP/tree/master/templates/643594-miptest_pedigree.yaml
244+
[Configuration file]: https://github.com/Clinical-Genomics/MIP/blob/master/templates/mip_config.yaml
245+
[GATK]:https://software.broadinstitute.org/gatk/
246+
[Perl]:https://www.perl.org/

SUMMARY.md

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,16 @@
11
# Summary
22

33
* [Introduction](README.md)
4-
* [Documentation](Documentation/README.md)
5-
* [Setup](Documentation/Setup.md)
6-
* [Analysis](Documentation/Analysis.md)
7-
* [API](Documentation/Api.md)
8-
* [Identifiers](Documentation/API/Identifiers.md)
9-
* [Modules](Documentation/API/Modules.md)
4+
* [Documentation](documentation/README.md)
5+
* [Setup](documentation/Setup.md)
6+
* [Analysis](documentation/Analysis.md)
7+
* [Code](documentation/Code.md)
8+
* [Best Practise](documentation/Code/Best_practise.md)
9+
* [API](documentation/Api.md)
10+
* [Define Parameters](documentation/API/define_parameters.md)
11+
* [Identifiers](documentation/API/Identifiers.md)
12+
* [Modules](documentation/API/Modules.md)
13+
* [Pedigree](documentation/API/Pedigree.md)
14+
* [Tests](documentation/API/Tests.md)
15+
* [Log](documentation/API/Tests/Log.md)
16+
* [Subroutines](documentation/API/Subroutines.md)

definitions/cpanfile

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
## MIP dependencies
2+
# Install by running "cpanm --installdeps ." in this directory
3+
# Module followed by minimum version required
4+
requires qw{ Archive::Extract 0.80 }; # VEP
5+
requires qw{ Archive::Zip 1.59 }; # VEP
6+
requires qw{ Array::Utils 0.5 }; # MIP
7+
requires qw{ Bio::Root::Version 1.007002 }; # VEP
8+
requires qw{ CGI 4.37 }; # VEP
9+
requires qw{ DBD::mysql 4.043 }; # VEP
10+
requires qw{ DBI 1.637 }; # VEP
11+
requires qw{ File::Copy::Recursive 0.38 }; # VEP
12+
requires qw{ HTML::Lint 2.26 }; # VEP
13+
requires qw{ IO::Uncompress::Gunzip 2.074 }; # VEP
14+
requires qw{ JSON 2.97000 }; # VEP
15+
requires qw{ IPC::System::Simple 1.25 }; # MIP
16+
requires qw{ List::MoreUtils 0.413 }; # MIP
17+
requires qw{ List::Util 1.49 }; # MIP
18+
requires qw{ Log::Log4perl 1.49 }; # MIP
19+
requires qw{ Modern::Perl 1.20170117 }; # MIP
20+
requires qw{ Module::Build 0.4224 }; # VEP
21+
requires qw{ Module::CPANfile 1.1002 }; # MIP
22+
requires qw{ Path::Iterator::Rule 1.012 }; # MIP
23+
requires qw{ PerlIO::gzip 0.20 }; # VEP
24+
requires qw{ Readonly 2.05 }; # MIP
25+
requires qw{ Sereal::Encoder 4.004 }; # VEP
26+
requires qw{ Sereal::Decoder 4.004 }; # VEP
27+
requires qw{ Set::IntervalTree 0.11}; # MIP
28+
requires qw{ Try::Tiny 0.28 }; # MIP
29+
requires qw{ YAML 1.24 }; # MIP

0 commit comments

Comments
 (0)