Starting to move to oncotator MAF instead of VCF (#3145)

* Starting to move to oncotator MAF instead of VCF.

Additions to support MAF generation instead of VCF.

Correcting typo

Reducing requirements for running Oncotator

Removing infer ONPs

Adding TODO

Put back infer-onps

PR comments

* Simple doc change to induce another automated test run.

Author: LeeTL1220

scripts/mutect2_wdl/README.md

@@ -18,6 +18,14 @@ This file has reasonable default parameters.
 - "broadinstitute/gatk-protected:1.0.0.0-alpha1.2.4" (This is a private image!  Recommended use ``gatk_jar`` as ``/root/gatk.jar``)
 - "broadinstitute/genomes-in-the-cloud:2.2.4-1469632282" (You must specify a ``gatk4_jar_override``)
 
+### Functional annotation (Oncotator)
+
+The M2 WDL can optionally run oncotator for functional annotation and produce a TCGA MAF from the M2 VCF.  *Oncotator is not a GATK4 tool and is provided in the M2 WDL as a convenience.*  There are several notes and caveats
+- Several parameters should be passed in to populate the TCGA MAF metadata fields.  Default values are provided, though we recommend that you specify the values.  These parameters are ignored if you do not run oncotator.
+- Several fields in a TCGA MAF cannot be generated by M2 and oncotator, such as all fields relating to validation alleles.  These will need to be populated by a downstream process created by the user.
+- Oncotator does not enforce the TCGA MAF controlled vocabulary, since it is often too restrictive for general use.  This is up to the user to specify correctly.
+  *Therefore, we cannot guarantee that a TCGA MAF generated here will pass the TCGA Validator*.  If you are unsure about the ramifications of this statement, then it probably does not concern you.
+- More information about Oncotator can be found at:  http://archive.broadinstitute.org/cancer/cga/oncotator
 
 ### Parameter descriptions
 
@@ -44,13 +52,9 @@ Recommended default values (where possible) are found in ``mutect2_multi_sample_
 - ``Mutect2_Multi.gatk4_jar_override`` -- (optional)  A GATK4 jar file to be used instead of the jar file in the docker image.  (See ``Mutect2_Multi.gatk4_jar``)  This can be very useful for developers.  Please note that you need to be careful that the docker image you use is compatible with the GATK4 jar file given here -- no automated checks are made.
 - ``Mutect2_Multi.preemptible_attempts`` -- Number of times to attempt running a task on a preemptible VM.  This is only used for cloud backends in cromwell and is ignored for local and SGE backends.
 - ``Mutect2_Multi.artifact_modes`` -- List of artifact modes to search for in the orientation bias filter.  For example to filter the OxoG artifact, you would specify ``["G/T"]``.  For both the FFPE artifact and the OxoG artifact, specify ``["G/T", "C/T"]``.  If you do not wish to search for any artifacts, please set ``Mutect2_Multi.is_run_orientation_bias_filter`` to ``false``.
-- ``Mutect2_Multi.onco_ds_tar_gz`` -- (optional)  A tar.gz file of the oncotator datasources -- often quite large (>15GB).  This will be uncompressed as part of the oncotator task.  Depending on backend used, this can be specified as a path on the local filesystem of a cloud storage container (e.g. gs://...).  Typically the Oncotator default datasource can be downloaded at ``ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/oncotator/``.  Do not put the FTP URL into the json file.
-- ``Mutect2_Multi.onco_ds_local_db_dir`` -- (optional)  A direct path to the Oncotator datasource directory (uncompressed).  While this is the fastest approach, it cannot be used with docker unless your docker image already has the datasources in it.  For cromwell backends without docker, this can be a local filesystem path.  *This cannot be a cloud storage location*
 - ``Mutect2_Multi.picard_jar`` -- A direct path to a picard jar for using ``CollectSequencingArtifactMetrics``.  This parameter requirement will be eliminated in the future.
 - ``Mutect2_Multi.m2_extra_args`` -- (optional) a string of additional command line arguments of the form "-argument1 value1 -argument2 value2" for Mutect 2.  Most users will not need this.
 - ``Mutect2_Multi.m2_extra_filtering_args`` -- (optional) a string of additional command line arguments of the form "-argument1 value1 -argument2 value2" for Mutect 2 filtering.  Most users will not need this.
- Note:  If neither ``Mutect2_Multi.onco_ds_tar_gz`` nor ``Mutect2_Multi.onco_ds_local_db_dir`` are specified, the Oncotator task will download and uncompress for each execution.
-
 - ``Mutect2_Multi.pair_list`` -- a tab-separated table with no header in the following formats.  For tumor-normal mode:
  ```
  TUMOR_1_BAM</TAB>TUMOR_1_BAM_INDEX</TAB>TUMOR_1_SAMPLE</TAB>NORMAL_1_BAM</TAB>NORMAL_1_BAM_INDEX</TAB>NORMAL_1_SAMPLE</TAB>
@@ -64,6 +68,21 @@ TUMOR_2_BAM</TAB>TUMOR_2_BAM_INDEX</TAB>TUMOR_2_SAMPLE
 . . .
 ```
 
+- ``Mutect2_Multi.onco_ds_tar_gz`` -- (optional)  A tar.gz file of the oncotator datasources -- often quite large (>15GB).  This will be uncompressed as part of the oncotator task.  Depending on backend used, this can be specified as a path on the local filesystem of a cloud storage container (e.g. gs://...).  Typically the Oncotator default datasource can be downloaded at ``ftp://gsapubftp-anonymous@ftp.broadinstitute.org/bundle/oncotator/``.  Do not put the FTP URL into the json file.
+- ``Mutect2_Multi.onco_ds_local_db_dir`` -- (optional)  A direct path to the Oncotator datasource directory (uncompressed).  While this is the fastest approach, it cannot be used with docker unless your docker image already has the datasources in it.  For cromwell backends without docker, this can be a local filesystem path.  *This cannot be a cloud storage location*
+
+ Note:  If neither ``Mutect2_Multi.onco_ds_tar_gz``, nor ``Mutect2_Multi.onco_ds_local_db_dir``, is specified, the Oncotator task will download and uncompress for each execution.
+
+The following three parameters are useful for rendering TCGA MAFs using oncotator.  These parameters are ignored if ``is_run_oncotator`` is ``false``.
+- ``Mutect2_Multi.sequencing_center`` -- (optional) center reporting this variant.     Please see ``https://wiki.nci.nih.gov/display/TCGA/Mutation+Annotation+Format+%28MAF%29+Specification+-+v2.4`` for more details.   
+- ``Mutect2_Multi.sequence_source`` -- (optional)  ``WGS`` or ``WXS`` for whole genome or whole exome sequencing, respectively.  Please note that the controlled vocabulary of the TCGA MAF spec is *not* enforced.  Please see ``https://wiki.nci.nih.gov/display/TCGA/Mutation+Annotation+Format+%28MAF%29+Specification+-+v2.4`` for more details.
+- ``Mutect2_Multi.default_config_file`` -- (optional)  A configuration file that can direct oncotator to use default values for unspecified annotations in the TCGA MAF.  This help prevents having MAF files with a lot of "__UNKNOWN__" values.  An usable example is given below.  Here is an example that should work for most users:
+
+```
+[manual_annotations]
+override:NCBI_Build=37,Strand=+,status=Somatic,phase=Phase_I,sequencer=Illumina,Tumor_Validation_Allele1=,Tumor_Validation_Allele2=,Match_Norm_Validation_Allele1=,Match_Norm_Validation_Allele2=,Verification_Status=,Validation_Status=,Validation_Method=,Score=,BAM_file=,Match_Norm_Seq_Allele1=,Match_Norm_Seq_Allele2=
+```
+
 #### mutect2 (single pair/sample)
 
 Recommended default values (where possible) are found in ``mutect2_template.json``
@@ -98,6 +117,9 @@ Recommended default values (where possible) are found in ``mutect2_template.json
 - ``Mutect2.picard_jar`` -- Please see parameter description above in the mutect2_multi_sample.
 - ``Mutect2.m2_extra_args`` -- Please see parameter description above in the mutect2_multi_sample.
 - ``Mutect2.m2_extra_filtering_args`` -- Please see parameter description above in the mutect2_multi_sample.
+- ``Mutect2.sequencing_center`` -- Please see parameter description above in the mutect2_multi_sample.   
+- ``Mutect2.sequence_source`` -- Please see parameter description above in the mutect2_multi_sample.
+- ``Mutect2.default_config_file`` -- Please see parameter description above in the mutect2_multi_sample.
 
 #### mutect2-replicate-validation
 
@@ -174,7 +196,7 @@ gs://broad-dsde-methods/takuto/na12878-crsp-ice/SM-612V3.bam    gs://broad-dsde-
   "Mutect2_Multi.is_run_orientation_bias_filter": true,
   "Mutect2_Multi.is_run_oncotator": true,
   "Mutect2_Multi.m2_docker": "broadinstitute/gatk:1.0.0.0-alpha1.2.4",
-  "Mutect2_Multi.oncotator_docker": "broadinstitute/oncotator:1.9.2.0",
+  "Mutect2_Multi.oncotator_docker": "broadinstitute/oncotator:1.9.3.0",
   "Mutect2_Multi.preemptible_attempts": 2,
   "Mutect2_Multi.onco_ds_tar_gz": "/data/onco_dir/oncotator_v1_ds_April052016.tar.gz",
   "Mutect2_Multi.m2_extra_args": "--maxNumHaplotypesInPopulation 50 --tumor_lod_to_emit 4.0",

scripts/mutect2_wdl/mutect2.wdl

@@ -11,7 +11,7 @@
 #  gnomad, gnomad_index: optional database of known germline variants, obtainable from http://gnomad.broadinstitute.org/downloads
 #  variants_for_contamination, variants_for_contamination_index: vcf of common variants with allele frequencies fo calculating contamination
 #  is_run_orientation_bias_filter: if true, run the orientation bias filter post-processing step
-#  is_run_oncotator: if true, annotate the M2 VCFs using oncotator.  Important:  This requires a docker image and should
+#  is_run_oncotator: if true, annotate the M2 VCFs using oncotator (to produce a TCGA MAF).  Important:  This requires a docker image and should
 #   not be run in environments where docker is unavailable (e.g. SGE cluster on a Broad on-prem VM).  Access to docker
 #   hub is also required, since the task will download a public docker image.
 #
@@ -54,6 +54,9 @@ workflow Mutect2 {
   File picard_jar
   String? m2_extra_args
   String? m2_extra_filtering_args
+  String? sequencing_center
+  String? sequence_source
+  File? default_config_file
 
   call ProcessOptionalArguments {
     input:
@@ -157,7 +160,10 @@ workflow Mutect2 {
                 preemptible_attempts = preemptible_attempts,
                 oncotator_docker = oncotator_docker,
                 onco_ds_tar_gz = onco_ds_tar_gz,
-                onco_ds_local_db_dir = onco_ds_local_db_dir
+                onco_ds_local_db_dir = onco_ds_local_db_dir,
+                sequencing_center = sequencing_center,
+                sequence_source = sequence_source,
+                default_config_file = default_config_file
         }
   }
 
@@ -167,8 +173,8 @@ workflow Mutect2 {
         File filtered_vcf = Filter.filtered_vcf
         File filtered_vcf_index = Filter.filtered_vcf_index
 
-        # select_first() fails if nothing resulve to non-null, so putting in "/dev/null" for now.
-        File? oncotated_m2_vcf = select_first([oncotate_m2.oncotated_m2_vcf, "null"])
+        # select_first() fails if nothing resolves to non-null, so putting in "null" for now.
+        File? oncotated_m2_maf = select_first([oncotate_m2.oncotated_m2_maf, "null"])
   }
 }
 
@@ -426,6 +432,10 @@ task oncotate_m2 {
     String oncotator_docker
     File? onco_ds_tar_gz
     String? onco_ds_local_db_dir
+    String? oncotator_exe
+    String? sequencing_center
+    String? sequence_source
+    File? default_config_file
     command {
 
           # fail if *any* command below (not just the last) doesn't return 0, in particular if wget fails
@@ -439,8 +449,8 @@ task oncotate_m2 {
 
           elif [[ "${onco_ds_tar_gz}" == *.tar.gz ]]; then
               echo "Using given tar file: ${onco_ds_tar_gz}"
-              tar zxvf ${onco_ds_tar_gz}
-              ln -s oncotator_v1_ds_April052016 onco_dbdir
+              mkdir onco_dbdir
+              tar zxvf ${onco_ds_tar_gz} -C onco_dbdir --strip-components 1
 
           else
               echo "Downloading and installing oncotator datasources from Broad FTP site..."
@@ -450,20 +460,22 @@ task oncotate_m2 {
               ln -s oncotator_v1_ds_April052016 onco_dbdir
           fi
 
-
-        /root/oncotator_venv/bin/oncotator --db-dir onco_dbdir/ -c $HOME/tx_exact_uniprot_matches.AKT1_CRLF2_FGFR1.txt  \
-            -v ${m2_vcf} ${entity_id}.oncotated.vcf hg19 -i VCF -o VCF --infer-onps --collapse-number-annotations --log_name oncotator.log
+        ${default="/root/oncotator_venv/bin/oncotator" oncotator_exe} --db-dir onco_dbdir/ -c $HOME/tx_exact_uniprot_matches.AKT1_CRLF2_FGFR1.txt  \
+            -v ${m2_vcf} ${entity_id}.maf.annotated hg19 -i VCF -o TCGAMAF --skip-no-alt --infer-onps --collapse-number-annotations --log_name oncotator.log \
+            -a Center:${default="Unknown" sequencing_center} \
+            -a source:${default="Unknown" sequence_source} \
+            ${"--default_config " + default_config_file}
     }
 
     runtime {
         docker: "${oncotator_docker}"
-        memory: "5 GB"
-        bootDiskSizeGb: 10
-        disks: "local-disk 150 SSD"
+        memory: "3 GB"
+        bootDiskSizeGb: 12
+        disks: "local-disk 100 HDD"
         preemptible: "${preemptible_attempts}"
     }
 
     output {
-        File oncotated_m2_vcf="${entity_id}.oncotated.vcf"
+        File oncotated_m2_maf="${entity_id}.maf.annotated"
     }
 }

scripts/mutect2_wdl/mutect2_multi_sample.wdl

@@ -76,6 +76,9 @@ workflow Mutect2_Multi {
     File picard_jar
     String? m2_extra_args
     String? m2_extra_filtering_args
+    String? sequencing_center
+    String? sequence_source
+    File? default_config_file
 
 	scatter( row in pairs ) {
 	    #      If the condition is true, variables inside the 'if' block retain their values outside the block.
@@ -117,7 +120,10 @@ workflow Mutect2_Multi {
                     artifact_modes = artifact_modes,
                     picard_jar = picard_jar,
                     m2_extra_args = m2_extra_args,
-                    m2_extra_filtering_args = m2_extra_filtering_args
+                    m2_extra_filtering_args = m2_extra_filtering_args,
+                    sequencing_center = sequencing_center,
+                    sequence_source = sequence_source,
+                    default_config_file = default_config_file
             }
     }
 
@@ -142,5 +148,7 @@ workflow Mutect2_Multi {
         Array[File] unfiltered_vcf_files = Mutect2.unfiltered_vcf
         Array[File] filtered_vcf_files = Mutect2.filtered_vcf
         Array[File] filtered_vcf_index_files = Mutect2.filtered_vcf_index
+
+        Array[File?] oncotated_m2_mafs = Mutect2.oncotated_m2_maf
     }
 }

scripts/mutect2_wdl/mutect2_template.json

@@ -20,7 +20,7 @@
   "Mutect2.is_run_orientation_bias_filter": "$__is_run_orientation_bias_filter__",
   "Mutect2.is_run_oncotator": "$__is_run_oncotator__",
   "Mutect2.m2_docker": "broadinstitute/gatk-protected:1.0.0.0-alpha1.2.4",
-  "Mutect2.oncotator_docker": "broadinstitute/oncotator:1.9.2.0",
+  "Mutect2.oncotator_docker": "broadinstitute/oncotator:1.9.3.0",
   "Mutect2.preemptible_attempts": 2,
   "Mutect2.artifact_modes": ["G/T", "C/T"],
   "Mutect2.picard_jar": "$__picard_jar__"

src/test/java/org/broadinstitute/hellbender/tools/walkers/mutect/Mutect2IntegrationTest.java

@@ -40,8 +40,8 @@ public class Mutect2IntegrationTest extends CommandLineProgramTest {
      * DREAM challenge vcfs):
      *
      * Sample 1: pure monoclonal sample, SNVs only
-     * Sample2: 80% pure monoclonal sample, SNVs only
-     * Sample3: pure triclonal sample, subclone minor allele frequencies are 1/2, 1/3, and 1/5, SNVs and indels
+     * Sample 2: 80% pure monoclonal sample, SNVs only
+     * Sample 3: pure triclonal sample, subclone minor allele frequencies are 1/2, 1/3, and 1/5, SNVs and indels
      * Sample 4: 80% biclonal sample, subclone minor allele fractions are 50% and 35%, SNVs and indels
      *
      * @throws Exception