Skip to content

Sorted BAM files and their indexes are published regardless of --save_bam #474

Description

@erikrikarddaniel

What happens

--save_bam defaults to false, but every run still publishes a sorted BAM and its index to results/samtools/.
On the full test that is ~700 MB per sample across four samples, i.e. several GB of alignments and indexes in the published output of a run whose results are linked from the documentation.

Why

There are two BAM publish paths, and only one of them is gated.

BBMAP_ALIGN (conf/modules.config#L129-L144) publishes its unsorted *.bam with enabled: params.save_bam, so with the default nothing lands in bbmap/bbmap/.

The .*SAMTOOLS.* block (conf/modules.config#L150-L171) then publishes *.bam and *.bam.bai with no enabled: guard at all:

withName: ".*SAMTOOLS.*" {
    publishDir = [
        [
            pattern: "*.bam",
            path: { "${params.outdir}/samtools/" },
            mode: params.publish_dir_mode

        ],
        [
            pattern: "*.bam.bai",
            path: { "${params.outdir}/samtools/" },
            mode: params.publish_dir_mode
        ],
        [
            pattern: "*.{flagstat,idxstats}",
            ...
            enabled: params.save_samtools,
        ]
    ]
}

Only the {flagstat,idxstats} rule in that block is gated, on params.save_samtools (default true).
So suppressing the unsorted BBMap BAM just means a sorted copy of the same alignments gets published instead, which is not what the param's own description ("Save the bam files from mapping") says.

This is long-standing rather than a recent regression — git log -L 150,171:conf/modules.config puts the block at ecd5f42 ("Merge 3.2.1 template plus some more").

Documentation gap

docs/output.md#L242 documents bbmap/bbmap/*.bam as being written only "if enabled with --save_bam", which is the misleading half of the story.
There is no samtools/ section in docs/output.md at all, so the directory that actually receives the BAMs and BAIs is undocumented.

Suggested fix

  • Add enabled: params.save_bam to the *.bam and *.bam.bai rules in the .*SAMTOOLS.* block.
  • Add a samtools/ section to docs/output.md covering the BAM/BAI (gated on --save_bam) and the flagstat/idxstats files (gated on --save_samtools).

Leave save_samtools alone; it governs small text files and defaulting it to on is fine.

Worth deciding while in there: SAMTOOLS_STATS emits *.stats, which the *.{flagstat,idxstats} pattern does not match, so those are never published even with --save_samtools.
MultiQC still picks them up from the work directory, so this may well be deliberate — but it is inconsistent with the param's description.

Note on cost

This changes published output for every profile, so all twelve pipeline snapshots need regenerating, two of which (diamond and kofamscan) need a machine with the databases.
Worth keeping out of any branch that is mid-testing.


Drafted with help from Claude Code.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions