RELEASE: staging -> master - #1943
Merged
Merged
Conversation
Picard CompareMetrics diffs metric rows by index across every column. The CrosscheckMetric emits read-group rows in non-deterministic order and its LEFT_FILE/RIGHT_FILE columns hold absolute BAM paths (submission id + SortSampleBam attempt-N), so verification flags spurious diffs on runs where nothing scientific changed. Normalize (drop the two path columns, sort rows) and diff for this metric type only; all other metrics keep the exact Picard comparison. Real value drift is still caught. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CompareAtacLibraryMetrics flagged sequenced_reads at 6.6e-9 (allowed diff ~4 reads on a 614M-read library); an observed 9-read run-to-run drift tripped it. Raise to 1e-7 (allowed ~61) — still 1e-5%, far below any real regression. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ATAC BAMs carry known run-to-run nondeterminism from the BWA aligner (per the WARP metrics nondeterminism catalog), so Picard CompareSAMs' strict pass/fail flags healthy runs. Add an optional mappings_diff_threshold to the shared CompareBams: default 0.0 preserves the exact strict behavior for all existing callers (Optimus, SlideSeq, snm3C, SmartSeq2, ...); when set, parse Picard's comparison.tsv (SamComparisonMetric, same schema VerifyRNAWithUMIs uses) and pass when the differing-record fraction is within threshold. Set 0.05 on the three ATAC callers (Multiome, ATAC, PairedTag). Missing-column and missing-tsv guards fail loud rather than silently pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CompareH5adFilesGEX compares every obs column with a strict .equals(), exempting only doublet_score. The emptydrops_* columns come from EmptyDrops, which is Monte-Carlo stochastic and is catalogued as nondeterministic within 1% in the WARP metrics catalog; the cell-metrics CSV compare does not cover them (they live only in the h5ad obs), so this strict check was their only gate and it flagged a healthy 0.03% drift in emptydrops_Limited. Allow emptydrops_* columns to differ within a 1% column-sum tolerance, mirroring the doublet_score exemption. Applies to all GEX h5ad callers (Optimus, Multiome, PairedTag, SlideSeq, SlideTags, SmartSeq2). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CompareH5adFilesGEX and CompareH5adFilesATAC ran python:3.10.0-buster and pip-installed anndata on every invocation, floating to whatever anndata/numpy/ pandas is latest (this run pulled anndata 0.11.4 / numpy 2.2.6 / pandas 2.3.3). Switch both to us.gcr.io/broad-gotc-prod/warp-tools:2.6.1 - the same image that writes the h5ad in H5adUtils.wdl, so anndata/numpy/pandas are already present - and drop the pip install. Pins the compare env and removes a network dependency per run. No image rebuild required. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make explicit what was convention: a pipeline WDL (pipelines/wdl/**) or shared tasks/wdl/ change takes a patch +0.0.1 bump + changelog; a change confined to verification/** (test wrappers, compare tasks) takes neither, since it ships no pipeline artifact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CompareTabix (CompareFragment) failed a fragment file over a hard 100-line difference (test 215,626,271 vs truth 215,626,137 = 134 lines, 6e-7 of the file) — expected wobble from the known BWA-aligner nondeterminism that cascades into fragment counts. Replace the absolute 100-line rule with max(100, truth_lines * fraction) at fraction 0.0001 (0.01%): big fragment files absorb the wobble, small plumbing files stay at the original 100-line floor, and gross count changes still fail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
genomic_read_quality_variance (and _mean) are per-gene float aggregates whose rounding depends on summation order, so they drift run-to-run (~6e-8 relative, single gene observed in SlideSeq). They were compared two ways with no tolerance: the exact gene-metrics CSV diff (CompareCompressedTextFiles) and the strict .equals() var-column check in CompareH5adFilesGEX. Add CompareGeneMetricsWithTolerance: a wide-CSV compare keyed by id column, order-insensitive, exact on every column except a configurable tolerated set (default genomic_read_quality_variance/_mean) which uses a 1e-4 relative tolerance. Swap it in for the gene-metrics compare in Optimus/Multiome/ PairedTag/SlideSeq/SlideTags (cell/UMI metrics keep the exact diff). Mirror the same tolerance in CompareH5adFilesGEX's post-unique var loop. Catalog note: these were marked 'assumed deterministic' (N); they are not. Reclassify to nondeterministic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SamToFastqAndDragmapAndMba ran dragen-os on a 40 GiB VM (custom-16-40960). The hg38 dragmap hash table (~40 GiB resident) plus samtools/OS overhead OOM-killed dragen-os on full scientific whole-genome read groups, surfacing as exit code 1 (empty pipe -> 'samtools view: fail to read header from -') and, on sibling shards, VMReportingTimeout(50002) / VMRecreatedDuringExecution(50006) as the OOM-killer froze the VM. Plumbing (downsampled) fit; scientific did not. Raise memory_mb 40960 -> 65536 (64 GiB) in DragmapAlignment.wdl. Cascade patch bumps to the transitive importers: WholeGenomeGermlineSingleSample 3.3.7->3.3.8, ExomeGermlineSingleSample 3.2.7->3.2.8, WholeGenomeReprocessing 3.3.7->3.3.8, ExomeReprocessing 3.3.7->3.3.8, with changelogs and pipeline_versions.txt. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…erant compares) - validate_release.sh diffs committed HEAD vs the ref, so uncommitted changes read as all-valid; commit before running. - point failing Verify metric comparisons at the existing tolerant compare tasks rather than adding strict checks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ures The recalibrated-BAM gather (UnmappedBamToAlignedBam.GatherBamFiles -> GatherSortedBamFiles, Picard MergeSamFiles) is a ~100-minute single-threaded block-copy for full scientific WGS. On a preemptible VM it repeatedly got preempted/torn down before finalizing (two RetryableFailures then a teardown exit-125), exhausting retries. Picard itself completed (done at 101.95 min, ~2 GB RSS) - the failure was VM survival, not memory or the command. Set preemptible_tries = 0 at the GatherBamFiles call. Appends to the existing unreleased 3.3.8/3.2.8 changelog entries (version already bumped this branch; bumps are once per branch). Cascade covers WGS/Exome germline + reprocessing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Note that long single-threaded steps (large-BAM gather/merge) should be non-preemptible (preemptible_tries = 0), and how preemption surfaces in a failed test (repeated RetryableFailure, possible teardown exit 125). Points at existing precedent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* get rid of sample name map input * replace sample_name and entity_id with sample_id for inputs --------- Co-authored-by: Jose Soto <jsoto@broadinstitute.org> Co-authored-by: GitHub Action <action@github.com>
* remove unused struct field --------- Co-authored-by: Jose Soto <jsoto@broadinstitute.org> Co-authored-by: GitHub Action <action@github.com>
FIX scientific test failures and nondeterminisms that broke on recent staging to master
* docs(scANVI): explain the SCVI (unsupervised) + SCANVI (semi-supervised) method Add a 'How the label transfer works' section: SCVI first learns an unsupervised, batch-corrected latent embedding over all query+reference cells (no labels), then SCANVI is initialized from it and trained semi-supervised (reference labels + unlabeled query) to predict query cell types. Explains why the two-stage design beats a supervised classifier on the reference alone. The page previously only listed the train-SCVI/train-SCANVI steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(scANVI): refresh page for 2.0.0 (new inputs/outputs + model reuse) Version header 1.2.0 -> 2.0.0; add the new workflow inputs (scanvi_model, output_max_probability, gpu_count/mem_size/nthreads/disk_size) to the inputs table, the scanvi_model_out output to the outputs table, and a note on reusing a trained model (load+predict, skip training; gpu_count=0 for a CPU-only run). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(scANVI): describe minibatch training + batch-vs-minibatch distinction Add a 'How the models are trained' subsection to the SCVI+SCANVI section: minibatch SGD streams batch_size=128 cells to the GPU while the full dataset stays in host RAM (GPU memory ~ minibatch x genes, independent of cell count), and disambiguate the SGD minibatch (batch_size) from the batch covariate (donor/library) that the models correct for. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(scANVI test): wire max_epochs through TestscANVI wrapper The scANVI workflow exposes Int? max_epochs and forwards it to the GPU task, but the TestscANVI wrapper never declared or passed it. The Plumbing JSON's scANVI.max_epochs (re-keyed to TestscANVI.* by the test framework) therefore did not bind, so the training-Plumbing test ignored the epoch cap. Declare Int? max_epochs on TestscANVI and forward it to the scANVI call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * scanvi batch_size * Updated pipeline_versions.txt with all pipeline version information * update docker pin * scanvi * fix gpucount and max_epochs bugs * fix nogpu runs * scanvi refactor for GPU/CPU container logic * scientific test guidelines: * scientific test guidelines: * scanvi rrid * workspace doc --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: GitHub Action <action@github.com>
…nt (SV pipeline) (#1907)
* Adding ReblockGVCF to website * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * add a diagram * update runtime and cost info * minor updates * Update _category_.json * updated readme to change preprint to publication --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Elizabeth Kiernan <55763654+ekiernan@users.noreply.github.com>
…est bcftools (#1906) * update bcftools in lowpass pipeline --------- Co-authored-by: GitHub Action <action@github.com>
* move reheadering commands run in glimpse phase task into a different wdl task --------- Co-authored-by: Jose Soto <jsoto@broadinstitute.org> Co-authored-by: GitHub Action <action@github.com>
TSPS-1039 Add tests for Glimpse2SVImputation
#1935) * documentation clean up to refer to changelogs instead of specific versions * Revert accidental deletion of deprecated changelogs and ignore deprecated paths in version scraper * Updated pipeline_versions.txt with all pipeline version information --------- Co-authored-by: GitHub Action <action@github.com>
…impse2LowPassImputation (#1931)
* updates made while scale testing * clean up boot disk size param cuz it doesnt work as expected * get optional phase base mem object working --------- Co-authored-by: Jose Soto <jsoto@broadinstitute.org> Co-authored-by: GitHub Action <action@github.com>
develop -> staging
|
DO NOT SQUASH MERGE!!! |
🔍Changelog Validation Results: |
🔍Version Validation Results: |
dev to staging
|
DO NOT SQUASH MERGE!!! |
🔍Changelog Validation Results: |
🔍Version Validation Results: |
develop -> staging: update to 582.0.0-slim (#1945)
|
DO NOT SQUASH MERGE!!! |
🔍Changelog Validation Results: |
🔍Version Validation Results: |
Collaborator
|
TestIlluminaGenotypingArray - transient testing error |
Collaborator
|
Nondeterminism PR incoming #1949 |
rsc3
approved these changes
Sep 11, 2026
rsc3
left a comment
rsc3
left a comment
Collaborator
There was a problem hiding this comment.
scANVI is currently broken, that is ok
`Found the smoking gun. Sub3's failure:
FAIL: test produced labels not present in truth: ['L6 IT ENTl']
L6 IT ENTl = "Layer 6 Intratelencephalic, Entorhinal" — an Allen Institute Taxonomy brain cortical neuron label. This is a PBMC (blood) test. Blood does not contain
cortical neurons. The pipeline annotated blood cells with a brain-atlas cell type. Seed nondeterminism cannot invent that. This is a wiring/model bug.
Verdict: substantive change, not drift. The previous agent under-called it.
The decisive evidence:
- Brain labels in a blood sample (sub3). L6 IT ENTl is an AIT brain-taxonomy label. The scANVI 2.0.0 changelog explicitly added "support for AIT (Allen Institute
Taxonomy) schema reference atlases… auto-detected (uns['schema_version'] + uns['hierarchy'])." Either this PBMC test config is now pointed at (or auto-detecting) a
brain reference atlas, or the AIT auto-detection is misfiring. That's a config/model regression, full stop. - The proportion shifts are targeted, not stochastic (sub1, directly observed). Some populations are byte-identical to 4 decimals while others move 40-50% in the
same run: `
jessicaway
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Slack thread discussing failures: https://broadinstitute.slack.com/archives/CESEYJW9W/p1788899389241109
Failures also addressed in PR comments below.
Test failures:
The following 4 pipelines had true failures:
Cell Metric emptydrops_FDR sums differ by 1.0482%, exceeds 1% tolerancein logsCell Metric emptydrops_FDR sums differ by 1.6346%, exceeds 1% tolerancein logsFAIL: cell-type proportion correlation 0.7831 < 0.95in logsFAIL: cell-type proportion correlation 0.8750 < 0.95in logsThe following 6 pipeline tests do not have Scientific tests, and therefore the GHAs fail:
The following 5 pipelines had jobs that were canceled by the GHA (probably a timeout?) but completed successfully in Terra (to view, search for the string below in Submissions and look for the Sep 2 date)
Checklist
If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP team by tagging @broadinstitute/warp-admins in a comment on this PR.