Germline accuracy validation (simulated diploid truth) + eval-germline - #26
Merged
Conversation
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… report Generalizes run_eval_somatic -> a VCF-agnostic run_eval used by both eval-somatic and the new eval-germline subcommand, and adds F1 (2PR/(P+R)) to the printed precision/recall report. eval-germline is the GIAB-ready interface: a real GIAB germline benchmark uses 'rosalind eval-germline --reference --calls --truth --regions <highconf.bed>' against the same compare_callsets + BedIndex path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…/recall/F1) Injects known het/hom SNVs into a reference, builds diploid-sampled sorted BAMs directly (bypassing the toy aligner -> measures the CALLER), calls, and compares PASS + all records against truth via compare_callsets. Deterministic (fixed-seed LCG). Two regimes, measured 2026-06-02: - 40x/0.5% (clean): PASS precision 1.00, recall 1.00, F1 1.00 - 12x/1.5% (stress): PASS precision 0.68, recall 0.90, F1 0.77 (the min_qual 30 / min_depth 8 filter cuts error-driven FPs 98->17; unfiltered 'all' is 0.29/1.00) Also asserts the deep-het probe is recovered as a PASS call under a cap below local depth (the unbiased reservoir fix, in an accuracy frame). Closes the audit's #1 blind spot: the caller's detection accuracy is now measured + gated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Records the measured precision/recall/F1: clean 40x/0.5% = 1.00/1.00/1.00; stress 12x/1.5% PASS = 0.68/0.90/0.77. Honest scope: simulated (not GIAB), detection-only, SNV-only; eval-germline is the GIAB-ready interface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the reflection audit's #1 blind spot: the memory contract was validated, but the variant calls themselves had never been measured against ground truth (only simulated E. coli with no injected variants).
eval-germlineCLI — generalizes the (already VCF-agnostic) somatic comparator into a sharedrun_evaland adds F1. This is the GIAB-ready interface: a real GIAB germline benchmark runsrosalind eval-germline --reference --calls --truth --regions <highconf.bed>against the samecompare_callsets+BedIndexpath.tests/germline_accuracy.rs— a deterministic, download-free harness that injects known het/hom SNVs into a reference, builds diploid-sampled sorted BAMs directly (bypassing the toy aligner, so it measures the caller), calls, and compares PASS + all records to truth. Two regimes:Honest reading: exact on clean data; at low-cov/high-error the caller finds every true variant (all-recall 1.000) but the
min_qual 30/min_depth 8PASS filter is what controls error-driven FPs (98→17). The deep-het probe (alt reads starting at the variant, under a cap below local depth) is recovered as a PASS call — the unbiased reservoir fix in an accuracy frame.Spec:
docs/superpowers/specs/2026-06-02-germline-accuracy-harness-design.md. Findings:docs/findings/2026-06-02-germline-accuracy.md.Honest scope (documented): simulated (not GIAB), detection-only (pos+ref+alt; no genotype concordance), SNV-only. Real GIAB HG002 plugs into the same
eval-germlineinterface and is the natural next increment (needs a downloaded pre-aligned BAM slice; out of scope here).Test plan
cargo test --test germline_accuracy -- --nocapture— both regimes pass; numbers deterministiccargo testgreen;cargo fmt --all -- --checkclean; 0 warnings (debug + release)🤖 Generated with Claude Code