Add mapmycells by rsc3 · Pull Request #1947 · broadinstitute/warp

rsc3 · 2026-09-02T15:47:38Z

Description

Give your PR a concise yet descriptive title.
Please explain the changes you made here.
Explain the motivation for making this change. What existing problem does the pull request solve?
Mention any issues fixed, addressed, or otherwise related to this pull request, including issue numbers or hard links for issues in other repos.
You can delete these instructions once you have written your PR description.

Checklist

If you can answer "yes" to the following items, please add a checkmark next to the appropriate checklist item(s) and notify our WARP team by tagging @broadinstitute/warp-admins in a comment on this PR.

Did you add inputs, outputs, or tasks to a workflow?
Did you modify, delete or move: file paths, file names, input names, output names, or task names?
If you made a changelog update, did you update the pipeline version number?

github-actions · 2026-09-02T15:47:49Z

Remember to squash merge!

github-actions · 2026-09-02T15:48:01Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
MapMyCells.wdl has not had its version updated
grep: /home/runner/work/warp/warp/pipelines/wdl/mapmycells/MapMyCells.changelog.md: No such file or directory
Some WDLs or changelog files need updating. See output for details.
validation_failed

github-actions · 2026-09-02T15:48:03Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/wdl/mapmycells/MapMyCells.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
All changelog files are valid for this release.

github-actions · 2026-09-02T15:59:33Z

Remember to squash merge!

github-actions · 2026-09-02T15:59:44Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
MapMyCells.wdl has not had its version updated
grep: /home/runner/work/warp/warp/pipelines/wdl/mapmycells/MapMyCells.changelog.md: No such file or directory
Some WDLs or changelog files need updating. See output for details.
validation_failed

github-actions · 2026-09-02T15:59:50Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
fatal: ambiguous argument '/home/runner/work/warp/warp/pipelines/wdl/mapmycells/MapMyCells.changelog.md': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
All changelog files are valid for this release.

- Collapse the dead query_markers ternary (Human_MTG and the catch-all both resolved to custom_query_markers) into a single condition - Fail fast via ErrorWithMessage when reference_atlas=="Custom" but custom_precomputed_stats is unset, instead of an opaque select_first error - Repin docker to the tag actually published for this branch (add-mapmycells); the previous default was never built - Add pipeline_version + MapMyCells.changelog.md (0.1.0, initial release) - Register MapMyCells and TestMapMyCells in .dockstore.yml - Move test_inputs into test_inputs/Plumbing per AGENTS.md layout - Add verification/test-wdls/TestMapMyCells.wdl, verification/VerifyMapMyCells.wdl, and .github/workflows/test_mapmycells.yml following the scANVI CI trio as a template Note: five gs://broad-gotc-test-storage/mapmycells/* asset paths referenced by the pipeline (both precomputed_stats files, gene_mapping_db, mouse_markers, and both test query.h5ad files) do not exist yet, so CI cannot pass or seed truth until real reference data is staged. See MapMyCells.changelog.md / follow-up discussion. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

The gs://broad-gotc-test-storage/mapmycells/ paths for precomputed_stats, gene_mapping_db, and mouse markers never actually existed (verified 404 via GCS metadata lookups). Per the upstream cell_type_mapper docs (running_online_taxonomies_locally.md), these are real, publicly hosted Allen Institute assets -- just not on GCS. Move them into the docker image built in warp-tools instead of trying to host a copy ourselves. - Task now selects precomputed_stats/query_markers by reference_atlas via a bash case statement pointing at baked-in container paths for Human_MTG and Mouse_WMB; Custom mode still takes real File inputs, localized normally - Dropped the gene_mapping_db default entirely (its source, gs://.../mmc_gene_mapper.2025-08-04.db, also never existed); building it via mmc_gene_mapper needs a 15GB NCBI taxonomy download, deliberately deferred -- see warp-tools commit - Removed the now-unneeded workflow-level select_first/ternary plumbing Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

gs://broad-gotc-test-storage/mapmycells/mmc_gene_mapper.2025-08-04.db is now populated with Allen Institute's pre-built gene-symbol-to-Ensembl-ID mapping db (~15GB). Default gene_mapping_db to it for every reference_atlas (select_first against custom_gene_mapping_db), matching the original author's intent -- the file just didn't exist at that path until now. Kept it GCS-hosted rather than baked into the mapmycells docker image: audited warp-tools and found no precedent for baking reference data this large into an image (every other large reference asset -- imputation panels, gnomAD data -- is a runtime File input hosted on GCS, never a docker layer), and baking it in would tax every task's docker pull by ~15GB regardless of whether that run needs gene mapping. Bumped disk_size default 100->150 to cover localizing the db on every run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…g tests test_human.json / test_mouse.json pointed at gs://broad-gotc-test-storage/mapmycells/query.h5ad and query_mouse.h5ad, neither of which ever existed. scANVI already has small, known-good gene-expression h5ads in its own test_inputs (verified via GCS metadata: ~31-36MB each, cheap enough for a Plumbing test regardless of which scANVI tier they were filed under). MapMyCells only consumes gene expression, so only each case's gex_h5ad is reused -- not the atac_h5ad or ref_h5ad: - Human_MTG: scANVI/input/scientific/10k_PBMC/10k_PBMC_gex.h5ad - Mouse_WMB: scANVI/input/plumbing/Mouse_Hippocampus_AIT/Mouse_Hippocampus_AIT_plumbing_gex.h5ad Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-02T17:01:59Z

Remember to squash merge!

github-actions · 2026-09-02T17:02:14Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-02T17:02:15Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2026-09-02T17:02:50Z

Remember to squash merge!

github-actions · 2026-09-02T17:03:03Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-02T17:03:12Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Verified via PubMed/Europe PMC (PMID 41958981, PMCID PMC13060854) rather than trusting a web-search snippet: Daniel SF, Lee C, Mollenkopf T, et al., "High-performance mapping of unlabeled cell-by-gene data to reference brain taxonomies," bioRxiv 2026, doi:10.64898/2026.03.06.710160. Also links the cell_type_mapper source repo. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-02T19:19:30Z

Remember to squash merge!

github-actions · 2026-09-02T19:19:39Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

github-actions · 2026-09-02T19:19:44Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-02T20:15:51Z

Remember to squash merge!

github-actions · 2026-09-02T20:16:06Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-02T20:16:06Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

UpdateTestInputs.py infers Plumbing vs Scientific by checking the input filename first, then falling back to substring-searching the whole input JSON for "plumbing"/"scientific" if the filename doesn't say. test_human's query h5ad path (borrowed from scANVI) contains "scientific", which fooled the fallback into seeding its truth under gs://.../MapMyCells/truth/scientific/... instead of .../plumbing/... -- confirmed by inspecting the actual GCS truth layout after the seed run. Renaming to *_plumbing.json short-circuits the fallback instead of relying on an accidental path substring match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

No other WARP pipeline uses a _plumbing filename suffix (checked Optimus, Multiome, ATAC, PairedTag, SlideSeq, SlideTags, PeakCalling) -- scANVI, which this was copied from, is the outlier. Back to test_human.json / test_mouse.json. The tier-mislabeling this was working around (test_human's truth landing under .../truth/scientific/... because its borrowed query h5ad path contains that word) is real but cosmetic -- truth_path is computed identically at seed- and compare-time, so it's self-consistent and doesn't fail the test. Left as a known issue in the changelog for a real fix later. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

New, smaller Plumbing tests to follow separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-03T13:05:50Z

Remember to squash merge!

github-actions · 2026-09-03T13:05:56Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-03T13:05:58Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

Fetched from https://brain-map.org/bkp/analyze/mapmycells/files (verified both URLs resolve): a 1k-cell human MTG example and a 10k-cell whole-mouse- brain example, both small enough for a fast/cheap smoke test. Staged at gs://pd-test-storage-public/MapMyCells/input/plumbing/{human,mouse}/, matching the <Pipeline>/input/{plumbing,scientific}/ layout other WARP pipelines use. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

github-actions · 2026-09-03T13:34:19Z

Remember to squash merge!

github-actions · 2026-09-03T13:34:24Z

🔍Version Validation Results:

Comparing versions and changelogs for pipelines that differ from the versions on 'origin/staging':
All WDLs and changelog files appear to be valid for this release.

github-actions · 2026-09-03T13:34:26Z

🔍Changelog Validation Results:

Comparing changelogs for pipelines that differ from the versions on 'origin/develop':
All changelog files are valid for this release.

rsc3 added 4 commits September 2, 2026 09:45

Add initial MapMyCells pipeline workflow

5666aa1

Add MapMyCells test inputs

434b704

Update MapMyCells.wdl to include reference_atlas selection logic

780a40c

Update test inputs to use reference_atlas

6f6d72d

rsc3 added 2 commits September 2, 2026 11:50

Add preemptible runtime parameter for Terra/Cromwell compatibility

cbac5aa

Add handoff README for MapMyCells pipeline

348f69a

rsc3 and others added 4 commits September 2, 2026 16:15

Updated pipeline_versions.txt with all pipeline version information

c6a2dc2

Merge branch 'develop' into add-mapmycells

b3063d3

rsc3 and others added 4 commits September 3, 2026 12:47

Reclassify test_human/test_mouse as the Scientific tests

25da603

New, smaller Plumbing tests to follow separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/add-mapmycells' into add-mapmycells

d8693f0

Uh oh!

Conversation

rsc3 commented Sep 2, 2026

Description

Checklist

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 2, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

🔍Changelog Validation Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

Uh oh!

github-actions Bot commented Sep 3, 2026

🔍Version Validation Results:

Uh oh!

github-actions Bot commented Sep 3, 2026

🔍Changelog Validation Results:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants