PTM dataset build path: uniprot-ptm key bug, cptac reprocess wiring, deps & resilience

[enriquea]

Context

While fetching + building all hvantk PTM datasets into a reusable store
($WORK/hvantk-datasets/ptm) on 2026-07-02, I hit several build-path bugs and gaps.
All three datasets were ultimately built (with workarounds); this issue tracks making
the happy path work cleanly so it's reproducible via the documented CLI.

Delivered (state as built, for reference):

• peptideatlas:phospho → peptideatlas_phospho.parquet — AnnotationTable, 259,932 sites
• uniprot-ptm:sites → uniprot_ptm_sites.ht — Hail AnnotationTable, 55,496 sites
• cptac:phospho → cptac_phospho/*.h5ad — 8/10 tumor types (coad + ov failed upstream)

---

Issues to fix

1. hvantk ptm build uses the wrong dataset key (blocker, 1-char fix)

hvantk/tools/ptm/pipeline.py:100 calls reg.get_dataset("uniprot_ptm:sites")
(underscore), but the plugin registers as uniprot-ptm:sites (hyphen;
plugin.yaml name: uniprot-ptm). The GTF download + coordinate mapping succeed,
then it KeyErrors at the final Hail-build step — i.e. it wastes the whole expensive
run before failing.

• Fix: "uniprot_ptm:sites" → "uniprot-ptm:sites".
• Test: add a regression asserting ptm_build_pipeline resolves the spec.
• Workaround used: built the .ht from the produced ptm_sites_mapped.tsv.bgz via
  run_builder_for_spec(get_dataset("uniprot-ptm:sites"), ...).

2. reprocess cptac:phospho can't build end-to-end

The parse stage (parse_raw_dir) returns a consolidated TSV path (str), but
build_cptac_phospho expects a {"expression": matrix.csv, "metadata": metadata.csv}
dict → TypeError at build. Also the builder default site_id_col="SiteID" doesn't
match the downloaded matrix column name, which is "Site" (write_matrix_csv sets
df.index.name="Site").

• Fix options: (a) parse step emits the {expression, metadata} pair per cancer;
  (b) builder accepts the consolidated TSV; (c) if the per-cancer driver is the intended
  path, document it. Reconcile the site_id_col default with "Site".
• Workaround used: per-cancer driver calling
  build_cptac_phospho(..., site_id_col="Site", sample_id_col="SampleID").

3. cptac --all aborts on one bad cancer; coad + ov fail upstream

hvantk download cptac-phospho --all aborts the whole batch on the first failing type
(broad except in phospho/cli.py). coad + ov fail inside cptac 1.5.14 (not
hvantk): umich mapping download unsupported operand type(s) for //: 'NoneType' and 'int',
and the source fallback hits len(generator) at cptac/cancers/cancer.py:682.

• Fix: make --all skip-and-continue per cancer (log failures, report a summary), so
  one bad type doesn't lose the other 9. Document coad/ov as upstream-blocked; revisit on
  a newer cptac release.

4. cptac missing dependency sorted_nearest

import cptac → pyranges → ModuleNotFoundError: No module named 'sorted_nearest'
(pyranges' compiled dep). The cptac skill is unusable until it's installed.

• Fix: add sorted_nearest to the cptac optional extra in pyproject.toml so
  poetry/uv installs pull it.

5. peptideatlas output-extension + --intermediate UX (minor)

peptideatlas:phospho is a pandas AnnotationTable, but core/io.save dispatches by
extension: --output foo.ht silently calls to_hail() (needs a JVM/Hail) — the correct
output is .parquet. Also reprocess errors --intermediate is required because the
plugin declares lifecycle.parse (the download already wrote the parsed TSV).

• Fix: validate/warn when the output extension doesn't match the artifact backend;
  clearer message or a backend-appropriate default.

---

Environment notes (HPC, for whoever retakes this)

• uv venv, no pip: install with uv pip install --python .venv/bin/python <pkg>;
  run via .venv/bin/hvantk.
• Hail builds need Java 11: source local/env/hail_env.sh. Non-Hail builds:
  HVANTK_SKIP_HAIL_INIT=1, and save .parquet/.h5ad (not .ht).
• Built store + full gotcha notes live at $WORK/hvantk-datasets/ptm/{README.md,MANIFEST.tsv}
  (not in git; workspace-only).