Add dbGaP digest fetcher and adapt-digests pipeline stage

[amc-corey-cox]

Summary

Adds the dm-bip side of dbGaP variable digest ingestion: a fetcher that pulls paired *.data_dict.xml + *.var_report.xml files from dbGaP's public FTP into a local cache, plus Make targets that drive per-pair adaptation via SchemaAutomator's schemauto adapt-dbgap CLI.

After fetch + adapt, downstream stages (schema-create, validate-data, map-data) work against the canonical-DD TSVs without further dm-bip changes.

Refs #204.
Closes #204.

Architecture

[upstream cohorts.yaml] → load_cohorts (Python)
                              ↓
[dbGaP FTP] ────fetch───→ .dbgap-cache/<cohort>/<phs.v>/pheno_variable_summaries/
                              │  paired *.data_dict.xml + *.var_report.xml
                              ↓
                 schemauto adapt-dbgap  (per pair, via Make)
                              ↓
                  output/<cohort>/dd/<phs>.<pht>.dd.tsv
                              ↓
                 schema-create / map-data / ...

What's in this PR

• src/dm_bip/prepare_study/fetch_digests.py — cohort registry loader (sourced from upstream RTIInternational/NHLBI-BDC-DMC-HV/hv-lint/cohorts.yaml), FTP fetcher with local cache, directory-listing scraper. No parsing — that lives in schema-automator.
• dm-bip fetch-digests <cohort-key> CLI: --list, --refresh, --cache-dir.
• New Make targets in pipeline.Makefile:
  • fetch-digests — populates the cache for DM_COHORT.
  • adapt-digests — pattern rule that calls schemauto adapt-dbgap per pair, writes output/<cohort>/dd/<phs>.<pht>.dd.tsv.

History — how we got here

Earlier commits on this branch shipped an inline ad-hoc adapter: parsers for data_dict.xml / var_report.xml, a translator from dbGaP's type vocabulary to schema-automator's canonical 10-type system, and a TSV writer for the canonical DD format. During review we decided that work belonged upstream in the linkml-map-driven adapter ecosystem rather than as bespoke dm-bip code.

That parser/translator/writer chunk now lives in linkml/schema-automator#207 (closes linkml/schema-automator#206), which:

• Adds a LinkML schema describing the dbGaP digest XML form.
• Adds a declarative linkml-map trans-spec doing the type-vocab translation — more comprehensive than what we had (handles dbGaP typos, composite types, empty-type fallback to calculated_type).
• Adds the schemauto adapt-dbgap CLI consumed by our new Make target.

This dm-bip PR has been rewritten to be just the fetcher + orchestration glue. The fetch + Make split also matches the rest of pipeline.Makefile (each stage is a Make target with explicit inputs/outputs).

Blocked on

• dbGaP variable digest adapter (closes #206) schema-automator#207 merging.
• A schema-automator release that includes the dbgap adapter, so the schema-automator pin in pyproject.toml can be bumped.

Things deliberately deferred

• BDC bucket fetching — auth complexity; deferred.
• Variable corpus assembly — the downstream consumer of #204; tracked separately.
• adapt-digests idempotency under repeated runs — depends on schema-automator's output being byte-stable, which in turn depends on linkml-map not introducing nondeterministic ordering. Out of scope here; revisit if Make rebuild churn becomes a problem.

Test plan

• make test — fetcher-layer unit tests pass.
• uv run ruff check . — lint clean.
• dm-bip fetch-digests --list — fetches upstream cohorts.yaml, prints all cohorts.
• make fetch-digests DM_COHORT=jhs — populates the cache.
• After SA release: make adapt-digests DM_COHORT=jhs — produces TSVs under output/jhs/dd/.

[amc-corey-cox]

Follow-up commit (`68731b0`) adds the consumable endpoint we discussed: `dm-bip parse-digests `.

What's new:

• Reads cached `data_dict.xml` files (output of `fetch-digests`)
• Writes one TSV per data table in schema-automator canonical DD format: `output//dd/..dd.tsv`
• All Spec A columns (`name`, `type`, `description`, `codes`, `unit`, `min`, `max`) plus `uri` from Spec B
• dbGaP types translated to canonical vocabulary (`String`/`string` → `string`, `encoded value` → `permissible_values`, etc.); unknowns default to `string`
• Encoded values rendered REDCap-style (`code, label | code, label`) with label sanitization for separator characters
• `uri` column carries `dbgap:` for traceability
• `unit`, `min`, `max` emitted empty for now (deferred until richer var_report parsing)

Why canonical DD output: gives the rest of the pipeline (#307 schema enrichment, eventually schema-automator's #192 ingestion) a real artifact to consume on disk, not just in-memory dataclasses.

Additional smoke test for the test plan:

• `dm-bip parse-digests jhs` — produces `output/jhs/dd/phs000286.pht*.dd.tsv` files; spot-check one TSV opens cleanly in a spreadsheet/editor

#204 still stays open: this gives us a usable endpoint and probably enough for a first pass at #307, but BDC bucket fetching, the legacy-format adapter for `prepare_metadata.py`, and any "richer corpus" work remain.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 76.31579% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.32%. Comparing base (8f4f431) to head (dcf5b3b).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/dm_bip/cli.py	0.00%	18 Missing ⚠️
src/dm_bip/prepare_study/fetch_digests.py	90.62%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #320      +/-   ##
==========================================
+ Coverage   75.52%   79.32%   +3.80%     
==========================================
  Files           9       10       +1     
  Lines         625      740     +115     
==========================================
+ Hits          472      587     +115     
  Misses        153      153              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a foundational “fetch + parse” layer for dbGaP variable digest files (supporting #204) by introducing a new digest module, new CLI commands to fetch/parse into a canonical TSV shape, and fixtures/tests to validate the XML parsers.

Changes:

• Added prepare_study.fetch_digests module with cohort registry loading (from upstream cohorts.yaml), XML parsing for data_dict.xml / var_report.xml, and cached fetching from NCBI.
• Added new CLI commands dm-bip fetch-digests and dm-bip parse-digests.
• Added parser unit tests plus real XML fixtures; added defusedxml dependency and ignored .dbgap-cache/.

Reviewed changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/dm_bip/prepare_study/fetch_digests.py	Implements cohort loading, digest fetching with local cache, XML parsers, and canonical TSV writer.
src/dm_bip/cli.py	Exposes fetch-digests and parse-digests commands via Typer.
tests/unit/test_fetch_digests.py	Adds unit tests covering XML parsing and canonical TSV output.
tests/input/dbgap_digests/JHS_Subject.data_dict.xml	Real-world fixture for data_dict.xml parser tests.
tests/input/dbgap_digests/JHS_Subject.var_report.xml	Real-world fixture for var_report.xml parser tests.
pyproject.toml	Adds direct dependency on defusedxml.
uv.lock	Locks defusedxml into the environment.
.gitignore	Ignores the local .dbgap-cache/ directory created by the new CLI.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[csiege]

Thanks for the useful foundation here. I found three correctness issues that look worth addressing before this becomes a dependable canonical-DD path:

1. Incorrect type translation for many real dbGaP dictionaries

   src/dm_bip/prepare_study/fetch_digests.py only maps a small set of exact dbGaP type strings, then silently defaults unknown types to string. In the local dbGaP cache, common real values include numeric, encoded, decimal, encoded, num, integer, encoded, continuous decimal, enumerated integer, and encoded values. Those would all become string, so parse-digests would emit materially wrong canonical data dictionaries: numeric fields lose numeric type, encoded fields lose permissible_values, and downstream schema generation would be incorrect.

2. Codes encoding mutates labels instead of following the canonical-DD escaping rules

   _sanitize_label() replaces commas with semicolons and pipes with slashes inside labels. The referenced schema-automator spec says labels may contain unescaped commas after the first separator comma, and literal pipe/backslash/comma in code values should be escaped with backslashes. The current implementation changes source metadata content, and the new test locks in that incorrect behavior by asserting comma removal. Labels like Black, non-Hispanic should stay as written, not become Black; non-Hispanic.

3. Generated numeric rows are knowingly non-conformant for unit, min, and max

   _dd_row() always emits empty unit, min, and max. The schema-automator canonical spec distinguishes empty cells from the explicit none token, and says empty cells are conformance issues when these fields apply. This may be acceptable as an interim draft output, but the CLI description says it converts digests into canonical-DD TSVs, so consumers may reasonably expect valid canonical rows. At minimum, numeric fields should probably emit none where genuinely unitless/unbounded, or the command should make clear that the output is incomplete/non-strict.