v1.1

GATK-SV v1.1 includes major single-sample pipeline upgrades such as improved filtering and a new reference panel, alongside updates to the joint calling pipeline.

Updates include

Single-sample pipeline

• Single-sample pipeline updates
  • Add filtering
  • New reference panel
  • Run Scramble by default
  • Reference panel creation notebook
  • and more
• Revert format detection changes to MakeBincovMatrix

Joint-calling pipeline functionality

• Enable DRAGEN-SV as a caller in cohort mode
• Enable outlier sample exclusion from training in TrainGCNV and GenerateBatchMetrics
• Add denoised copy number plots and stratified variant counts to EvidenceQc outputs

Minor changes and bug fixes

• Zip regeno coverage medians file
• Update SVConcordance.wdl to enable stratified matching
• Update reference genome in depth plots
• Fix code path in EvidenceQc when run_ploidy is false
• Bug fix missing sample ID in median coverage
• Enable passing a threshold to govern the window for detection of dispersed deletions
• Handle zero-length arrays in JoinRawCalls

sv-shell (WDL-independent pipeline-in-progress)

• WDL-independent version of GatherSampleEvidence
• Log stdout and stderr of tasks in sv_shell
• Implement a bash version of the Evidence QC workflow

CI/CD

• Disable automated sv-shell and scramble docker updates temporarily while addressing R package installation issues

Full Changelog: v1.0.5...v1.1

v1.0.5

What's Changed

Bug fixes

• Remove sorting from SVConcordance to align with the updated GATK tool
• Keep EV as int when reformatting legacy genotyped VCFs

Miscellaneous

• Update image links to address page scrolling issues on our website
• Fix typo in argument in stitch_fragmented_CNVs.sh
• Update issue templates
• Update shell flags for consistency and deprecate outdated bincov code path in MakeBincovMatrix
• Fix BND header description
• Deprecate ShardedCluster.wdl

Full Changelog: v1.0.4...v1.0.5

v1.0.4

What's Changed

Critical

The AnnotateVcf workflow configuration in Terra has been corrected to take the filtered VCF as input.

• If you have processed a cohort with GATK-SV in Terra and you have run FilterGenotypes, you should be aware that the AnnotateVcf output may not be filtered.
• You can tell whether a VCF has been through filtering by looking for the NCR INFO field in the header - if it’s there, FilterGenotypes has been applied.
• To address this, copy the updated AnnotateVcf workflow from the featured workspace to get the new input configuration, and update the gatk_docker in Workspace Data to match the featured workspace.

Bug fixes

• Update GATK docker with CPX interval sorting fix for SVAnnotate
• CPU configuration workaround for RealignSoftClippedReads
• Enable non-existent INFO/EVIDENCE fields in input VCF for breakpoint overlap filtering in ResolveComplexVariants

Miscellaneous

• Remove hail-based VCF merging
• Update website regarding GATK forum

Full Changelog: v1.0.3...v1.0.4

v1.0.3

What's changed

Critical

This release includes a fix for a bug introduced in v1.0.2 that affects GQ values and may have impacts on filtering. We recommend that users who ran the CombineBatches workflow with version v1.0.2 or v1.0.2-hotfix should update CombineBatches and CleanVcf to v1.0.3 and rerun the workflows from CombineBatches until the end of the pipeline. More details in #796.

Pipeline resources

• Update GTF to GENCODE v47, and add reusable GTF preprocessing script

Bug fixes

• Address rare edge cases in RefineComplexVariants
• Fix MainVcfQc.IdentifyDuplicates to resolve NoneType mismatches

CI/CD

• Update the OS version of the Action runners, and make the corresponding changes to the docker build CI/CD action
• Add the de novo pipeline to Dockstore integration
• Remove carrot tests
• Use communicate() with Python subprocess

Full Changelog: v1.0.2-hotfix...v1.0.3

v1.0.2-hotfix

This release is a hotfix addressing a bug introduced in v1.0.2 that caused ClusterBatch to fail.

Full Changelog: v1.0.2...v1.0.2-hotfix

v1.0.2

What's Changed

Core updates

• Update CombineBatches workflow
• Update to gnomAD-SV-v4 for external AF annotation
• Create SampleQC and Batching notebooks
• Remove internal workflows and scripts (migrated to gatk-sv-internal repo)

Bug fixes

• Fix handling of multiallelic GTs with low copy number
• Fix bug that results in an empty Vapor report
• Fix IntegrateGQ.sh errors due to presence of variants of just one type
• Reorder PED file columns in SampleQC notebook

Ease of use

• Add VCF input option to Vapor WDL
• Make functional annotation optional if no GTF/BED provided
• Small fixes to sample QC notebook

Documentation and input templates

• Updates to EvidenceQC templates
• Add joint calling link to documentation
• Remove mention of variant filtering from GenerateBatchMetrics documentation
• Update gatk docker on azure

Full Changelog: v1.0.1...v1.0.2

v1.0.1

What's Changed

Bug fixes

• Fix for normalize_counts() in GenerateBatchMetrics
• Modify EvidenceQC to not create/expect additional column in median coverage file
• Fix multiallelic genotypes for biallelic CNVs

Documentation

• Add links to GATK forums and joint calling workspace
• Replace relative paths with abs paths to JS components

Full Changelog: v1.0...v1.0.1

v1.0

Welcome to v1.0 of GATK-SV! With this release, we are deprecating MELT in favor of Scramble, adding downstream filtering and refinement steps to the core pipeline and Terra workspace, moving our documentation to our website, and getting ready to release our public Terra workspace for the joint calling mode of GATK-SV.

Updates include

Pipeline functionality

• Improve Scramble accuracy for BWA and Dragen 3.7.8
• Refine complex variants and translocations
• Filter wham-only DELs and scramble-only SVAs in CleanVcf
• Update GATK to 4.6.0.0 in sv_pipeline_docker
• Resolve incorrect mCNV genotype counts for hom-alt
• Updates to EvidenceQC for sample QC analysis
  • Enhance EvidenceQC outputs to better align with sample QC
  • Add Scramble metrics to the qc table generated in EvidenceQC
  • Rename high & low overall outliers columns generated via EvidenceQC
• Leave empty genotypes in CleanVcf part 5
• Add check for duplicate records to MainVcfQc
• Update type-checking in RdTestV2 to be class-based

Documentation

• Move documentation to website
• Update docs on creating sample_set_set
• Update GatherSampleEvidence & TrainGCNV docs
• Update the landing page of the docs
• Add genotype filtering Terra workflow configs and documentation
• Include SVTK documentation in README

For developers

• Deprecate Google Cloud configuration JSONs
• Add the single-sample pipeline to the Dockstore automation

Full Changelog: v0.29-beta...v1.0

v0.29-beta

What's Changed

Critical

This release includes a critical bug fix to the SplitVariants task in GenotypeBatch. The affected workflow was GenotypeBatch, and the affected version was v0.28.5-beta. We recommend immediately updating to v0.29-beta. If you ran GenotypeBatch with v0.28.5-beta, please check if any records were dropped. If in doubt, re-run with v0.29-beta. More details in #712

Pipeline functionality updates

• Integrate ReshardVcf into ResolveComplexVariants
• Remove CHR2 and END2 from INS in CleanVcf
• Fix --par arg to compute_AFs.py in ShardedAnnotateVcf
• Updates to allele frequency annotation fields
• Grouped MEIs with insertions in splitvariants.py

Performance improvements

• Reimplement ParseGenotypes in GenotypeComplexVariants
• Reduce memory usage in GenotypeSRPart1
• Set the default disk size in AnnotateIntervals as a function of input files size
• Making the Vapor plots optional to store as a final output
• Make per-sample QC plots optional in MainVcfQc

Fixing bugs and small annoyances

• Skip subsampling if batch size is less than n_samples_subsample
• Add NonZeroReferenceLengthAlignmentReadFilter read filter to CollectSVEvidence
• Prevent sample ID mangling in WGD computation
• Update gatk docker with changes to handle CPX_TYPE for CTX
• Fix UnboundLocalError in EvidenceQc
• Update gnomad-v2 sample-level benchmarking data path

Documentation

• Terra dashboard updates
• Reorganize docs on running the pipeline
• Add CONTRIBUTING.md
• Update Docusaurus and its dependencies to v3.3.2
• Update docs on building and hosting Docker images
• Update docs on building inputs
• Add documentation for rename_samples in GatherBatchEvidence

CI/CD, workflow organization, and auxiliary scripts

• Deprecate single-batch Terra configs
• Add support for incomplete workflows in get_inputs_outputs.py
• Trigger WDL tests on changes to /inputs and fix Terra config tests
• Update deprecated Pandas append operation for monitoring log analysis
• Extend the list supported syntax-highlighting languages

Full Changelog: v0.28.5-beta...v0.29-beta

v0.28.5-beta

Updates

GatherSampleEvidence

• Copy or move the files instead of creating symlinks in the LocalizeReads task for compatibility with CoA/TES
• Make LocalizeReads optional
• Make manta region bed index required
• Make MELT scripts independent of the execution path

Main pipeline

• Rewrite SplitVariants in TasksGenotypeBatch.wdl
• Disable MakeCohortVcfMetrics by default
• Find CN field for mCNVs in MainVcfQc without hard-coding order of format fields
• Update BEDTools version to 2.31.0
• Two small bugfixes to EvidenceQc
• Update gatk_docker with CPX annotation changes to SVAnnotate
• Reduce redundant docker image inputs: sv_pipeline_base_docker, sv_pipeline_hail_docker, sv_pipeline_updates_docker, sv_pipeline_rdtest_docker all now just sv_pipeline_docker

Downstream filtering

• Reduce SVConcordance memory usage
• Release AoU filtering model
• Make the ploidy table a workflow output instead of an intermediate
• Change Vapor bed preprocessing errors to warnings
• Genotype filtering training labels and cutoff optimization

Misc.

• Add ReshardVcf workflow
• Setup automatic updates to WDLs on Dockstore
• Trivial change to sv-pipeline-virtual-env
• Add a preview version of the de-novo pipeline

Full Changelog: v0.28.4-beta...v0.28.5-beta