Skip to content

TECHNICAL.md

Latest commit

 

History

History
370 lines (259 loc) · 10.3 KB

TECHNICAL.md

File metadata and controls

370 lines (259 loc) · 10.3 KB
Raw

Technical Documentation: Variant Support Pipeline

This document provides detailed technical information about each process in the Variant Support Pipeline, including inputs, outputs, parameters, and implementation details.

Table of Contents

  1. Core Workflow
  2. Read Processing Processes
  3. Amplicon Matching Processes
  4. Variant Analysis Processes
  5. Utility Processes
  6. Python Script: build_variant_support_df.py

Core Workflow

The core workflow coordinates the execution of all the individual processes. It performs the following major steps:

  1. Input Preparation

    • Collects FastQ files and filters out Undetermined files
    • Optionally limits processing to 3 samples (dev mode)
    • Optionally randomizes processing order

  2. Read Processing

    • Trims adapters from paired reads
    • Aligns reads to reference genome
    • Processes BAM files (conversion, sorting, marking duplicates)

  3. Variant Analysis

    • Reads variant information from JSON
    • Matches reads to probes and amplicons
    • Calculates coverage at variant positions
    • Generates final output with variant support metrics

Pipeline Overview Diagram

alt pipeline diagram

Read Processing Processes

trim_pair

Purpose: Trim adapters and low-quality sequences from paired-end FastQ files.

Container: quay.io/staphb/fastp:0.23.4

Inputs:

  • Tuple of (sample_id, [read1.fastq.gz, read2.fastq.gz])

Outputs:

  • fastqs: Tuple of (sample_id, read1.trimmed.fq.gz, read2.trimmed.fq.gz)
  • jsons: JSON file with trimming metrics (sample.fastp.json)

Resource Requirements:

  • CPUs: 4
  • Memory: 4 GB

Process:

  • Uses fastp to perform quality control and adapter trimming
  • Retains original file naming pattern with added .trimmed suffix
  • Generates a JSON report with QC metrics

minimap2_align_reads

Purpose: Align paired-end reads to the reference genome.

Container: quay.io/biocontainers/minimap2:2.28--he4a0461_0

Inputs:

  • Tuple of (sample_id, read1.trimmed.fq.gz, read2.trimmed.fq.gz)
  • Reference genome FASTA file
  • Reference genome index file

Outputs:

  • Tuple of (sample_id, sample.sam)

Resource Requirements:

  • CPUs: 4
  • Memory: 16 GB

Process:

  • Uses minimap2 for short-read alignment (-ax sr)
  • Generates MD tags for mismatches
  • Adds read group information to the SAM file
  • Specifies platform as "ONT" (Oxford Nanopore) but platform usage label as "illumina"

sam_to_bam

Purpose: Convert SAM files to the binary BAM format.

Container: quay.io/biocontainers/samtools:1.19.2--h50ea8bc_1

Inputs:

  • Tuple of (sample_id, sample.sam)

Outputs:

  • Tuple of (sample_id, sample.bam)

Resource Requirements:

  • CPUs: 4
  • Memory: 1 GB

Process:

  • Uses samtools to convert from SAM to BAM format
  • Simple conversion with no additional filtering or processing

sort_bam

Purpose: Sort BAM files by genomic coordinates.

Container: quay.io/biocontainers/sambamba:1.0--h98b6b92_0

Inputs:

  • Tuple of (sample_id, sample.bam)

Outputs:

  • Tuple of (sample_id, sample.sorted.bam, sample.sorted.bam.bai)

Resource Requirements:

  • CPUs: 1
  • Memory: 4 GB

Process:

  • Uses sambamba to sort the BAM file by coordinates
  • Generates a BAM index file
  • Publishes sorted BAMs to the output directory under samples/{sample}/

mark_duplicates

Purpose: Identify and mark duplicate reads in BAM files.

Container: quay.io/biocontainers/sambamba:1.0--h98b6b92_0

Inputs:

  • Tuple of (sample_id, sample.sorted.bam, sample.sorted.bam.bai)

Outputs:

  • Tuple of (sample_id, sample.dup.bam, sample.dup.bam.bai)

Resource Requirements:

  • CPUs: 2
  • Memory: 8 GB

Process:

  • Uses sambamba to mark duplicate reads
  • Uses local /var/tmp directory for temporary files
  • Configures hash table size and overflow list size for efficient processing
  • Publishes duplicate-marked BAMs to the output directory

Amplicon Matching Processes

amplicon_matching

Purpose: Combined process that performs sub-BAM creation, BED conversion, and amplicon matching.

Container: pedrofeijao/amplicon_matching:0.1

Inputs:

  • Tuple of (sample_id, probe_hits_file, bam, bai, amplicon_bed, hit_string, variant, variant_safename)

Outputs:

  • match_counts: Tuple with amplicon match counts
  • unmatched_coords: Tuple with unmatched coordinates

Process:

  • Creates a sub-BAM containing only reads with probe hits
  • Converts the BAM to BED format (keeping only forward reads in proper pairs)
  • Sorts the BED file
  • Intersects with amplicon BED file to count matching reads
  • Requires 95% reciprocal overlap for an amplicon match

Note: This process consolidates the functionality of create_sub_bam, bam_to_bed, sort_bed, and count_reads processes into a single process for efficiency.

create_sub_bam

Purpose: Extract probe-matching reads from the main BAM file.

Container: quay.io/biocontainers/samtools:1.19.2--h50ea8bc_1

Inputs:

  • Tuple of (sample_id, probe_hits_file, bam, bai, hit_string, variant, variant_safename)

Outputs:

  • Tuple with sub-BAM file containing only reads that match probes

Resource Requirements:

  • CPUs: 4
  • Memory: 1 GB

Process:

  • Extracts read names from probe hits file that match a specific variant
  • Uses samtools to create a sub-BAM with only those reads
  • Only published to output if debug is enabled

bam_to_bed

Purpose: Convert BAM to BED format for amplicon matching.

Container: quay.io/biocontainers/sambamba:1.0--h98b6b92_0

Inputs:

  • Tuple with sub-BAM file from probe-matching reads

Outputs:

  • Tuple with BED file of read positions

Resource Requirements:

  • CPUs: 4
  • Memory: 1 GB

Process:

  • Uses sambamba to extract forward strand reads in proper pairs
  • Calculates the read start and inferred end position based on insert size
  • Converts to BED format
  • Only published to output if debug is enabled

sort_bed

Purpose: Sort the BED file for efficient intersection.

Container: quay.io/biocontainers/bedtools:2.31.1--hf5e1c6e_1

Inputs:

  • Tuple with unsorted BED file

Outputs:

  • Tuple with sorted BED file

Resource Requirements:

  • CPUs: 4
  • Memory: 1 GB

Process:

  • Uses bedtools sortBed to sort the BED file by genomic coordinates
  • Only published to output if debug is enabled

count_reads

Purpose: Count reads overlapping with amplicons.

Container: quay.io/biocontainers/bedtools:2.31.1--hf5e1c6e_1

Inputs:

  • Tuple with sorted BED file and amplicon BED file

Outputs:

  • match_counts: Tuple with amplicon match counts
  • unmatched_coords: Tuple with unmatched coordinates

Resource Requirements:

  • CPUs: 4
  • Memory: 1 GB

Process:

  • Uses bedtools intersect with 95% reciprocal overlap requirement
  • Counts how many reads match each amplicon
  • Identifies reads that don't match any amplicons
  • Only published to output if debug is enabled

Variant Analysis Processes

bam_readcount

Purpose: Calculate per-base coverage at variant positions.

Container: mgibio/bam-readcount:1.0.0

Inputs:

  • Tuple of (sample_id, bam, bai, amplicon_bed)
  • Reference genome FASTA file
  • Reference genome index file

Outputs:

  • Sample allele coverage text file

Process:

  • Uses bam-readcount to calculate the coverage of each base at variant positions
  • Applies a mapping quality filter of 20
  • Outputs a file with coverage for each base (A, C, G, T)
  • Published to output if debug is enabled

build_variant_support_output

Purpose: Generate the final variant support spreadsheet.

Container: pedrofeijao/pandas-openpyxl:v1.0

Inputs:

  • Variant coverages file
  • Read counts file
  • Variant info JSON file
  • Cancer types file (optional)

Outputs:

  • Excel spreadsheet with variant support metrics
  • Tab-separated text version of the same data

Process:

  • Calls the Python script build_variant_support_df.py
  • Combines variant coverage data with amplicon read counts
  • Calculates Variant Allele Frequency (VAF)
  • Optionally separates output by cancer type
  • Published to the main output directory

Utility Processes

write_fastq_ordering

Purpose: Record the order of FastQ processing.

Inputs:

  • List of FastQ sample names

Outputs:

  • Text file with FastQ processing order

Process:

  • Creates a simple text file with the order of sample processing
  • Useful for debugging and reproducibility
  • Published to the main output directory

Python Script: build_variant_support_df.py

This Python script processes the outputs from previous steps to create the final variant support tables.

Key Functions:

get_variant_coverage

  • Parses bam-readcount output
  • Creates a dictionary with coverage for each base at each variant position

parse_variant_info

  • Reads the variant info JSON file
  • Filters out duplicate variants (from alternative transcripts)
  • Creates a dictionary of variant information

build_variant_df

  • Builds a DataFrame from amplicon match counts
  • Merges with variant information
  • Ensures all sample-variant combinations are present (filling missing values with zeros)
  • Calculates the number of supporting amplicons for each variant

add_variant_metadata

  • Adds genomic coordinates and other metadata to the variant DataFrame

add_bam_coverage_and_vaf

  • Adds coverage data from bam-readcount
  • Calculates coverage for reference and alternate alleles
  • Calculates Variant Allele Frequency (VAF)

add_cancer_types

  • Adds cancer type information if provided
  • Matches sample names even with different suffixes

write_output

  • Generates Excel spreadsheet with variant support metrics
  • Creates a tab-separated text version of the same data
  • Optionally separates output by cancer type