Fix rc12 review cleanup issues (#313)

* Fix rc12 review cleanup issues

* Update orchestrator docs for worker context rename

* Clarify calibration peptide feature sizing

* Extract calibration CUDA batch sizing

* Add release workflow and notebook audit

Author: iskandr

README.md

@@ -20,10 +20,10 @@ Have a bugfix or other contribution? We would love your help. See our [contribut
 ## 2.3.0 release candidate
 
 > [!IMPORTANT]
-> 2.3.0 is currently a release candidate (`2.3.0rc11`), not yet a final release.
+> 2.3.0 is currently a release candidate (`2.3.0rc12`), not yet a final release.
 > It keeps the same public API and pre-trained models as 2.2.x. Install it with
 > `pip install --pre mhcflurry`, or pin the version with
-> `pip install mhcflurry==2.3.0rc11`. A plain `pip install --upgrade mhcflurry`
+> `pip install mhcflurry==2.3.0rc12`. A plain `pip install --upgrade mhcflurry`
 > stays on the latest stable release (2.2.x) until 2.3.0 is final, since pip
 > skips pre-releases.
 

docs/2023_retraining_notebook_audit.md

@@ -0,0 +1,137 @@
+# 2023 Retraining Notebook Audit
+
+This audits the copied notebook bundle in `notebooks/2023-retraining/` against
+the maintained release scripts and 2.3.x evaluation commands. The notebooks are
+valuable source material, but they include generated data, checkpoint notebooks,
+old imports, and local paths, so they should be ported into scripts rather than
+committed as the workflow.
+
+## What The Notebooks Cover
+
+- Benchmark assembly: joins `data_evaluation` sample shards into wide
+  monoallelic and multiallelic benchmark tables with MHCflurry, NetMHCpan, and
+  MixMHCpred columns.
+- Accuracy scoring: per-sample and per-length AUC / PPV tables, percent-change
+  columns against NetMHCpan BA, NetMHCpan EL, MixMHCpred, and older MHCflurry.
+- Figure generation: paper-style scatter panels, score bars, processing motifs,
+  processing-vs-affinity correlation plots, and presentation coefficient plots.
+- Supplemental outputs: sample tables, benchmark data files, model-selection
+  accuracy workbook, antigen-processing motif workbook, and supplementary data
+  CSV/XLSX files.
+- Exploratory analyses: novel-allele analysis, proteasome mass-spec analysis,
+  and an immunogenicity experiment depending on external local files.
+
+## Differences From Maintained Workflows
+
+- Data curation is split differently. Current maintained scripts use
+  `downloads-generation/` to build raw downloads and `scripts/training/` to build
+  release models. The notebooks start from already-generated downloads and create
+  analysis artifacts under `notebooks/2023-retraining/artifacts/`.
+- The current `data_evaluation` download generates train-excluded benchmark
+  files and sample shards. The notebooks additionally join those shards into
+  wide analysis tables and include an abandoned no-exclude-train path.
+- The notebooks label recent multiallelic samples using PMIDs `31844290` and
+  `31154438`. Current presentation training excludes `31844290`, `31495665`, and
+  `31154438`; this needs one named holdout policy before new public weights are
+  trained and evaluated.
+- The notebooks use old notebook-era dependencies and names (`keras`,
+  `tensorflow`, `mhcnames` in places). Ports should use the current PyTorch code,
+  `mhcgnomes` / existing allele normalization helpers, and maintained CLI
+  commands.
+- Current `mhcflurry compare-models` covers affinity and presentation regression
+  metrics, and `mhcflurry plot-model-comparison` renders generic plots. It does
+  not yet recreate the paper-style benchmark tables, sample-group comparisons,
+  model-selection workbook, motif workbook, or supplemental figures.
+- The current full training script trains processing `no_flank` and
+  `short_flanks` variants, then uses `short_flanks` as the with-flank processing
+  component for presentation. Older download-generation trained a separate
+  `models.selected.with_flanks` variant. That semantic difference should be
+  resolved explicitly before publishing 2.3.0 weights.
+- Historical runplz files in `jobs/` patched scripts in place for one experiment
+  and called deleted comparison scripts. Remote execution is now represented by
+  maintained scripts instead of patching a remote copy.
+
+## Missing Figure And Table Ports
+
+- Predictor metadata / aesthetics: convert `0 aesthetics.ipynb` into a small
+  `predictor_info.csv` generator with stable labels, colors, and descriptions.
+- Wide benchmark tables: port the monoallelic and multiallelic shard-join logic
+  from `1 prepare benchmark dataset*.ipynb`.
+- Accuracy score tables: port the AUC / PPV / percent-change tables from
+  `2 monoallelic accuracy plots.ipynb` and `2 multiallelic accuracy.ipynb`.
+- Monoallelic figures: BA/EL/MixMHCpred scatter panels, training-count vs AUC,
+  and novel-allele comparison tables / plots.
+- Multiallelic figures: PPV/AUC scatter panels comparing NetMHCpan BA,
+  NetMHCpan EL, MixMHCpred, older MHCflurry, and presentation score variants,
+  including recent-vs-old sample grouping.
+- Model-selection evidence: port the unselected-model held-out decoy AUC workbook
+  and HLA locus score bars.
+- Processing evidence: port antigen-processing motif count/PWM workbook,
+  processing logo figure, and processing-vs-affinity correlation plots.
+- Supplemental outputs: port sample table, supplemental sample table with
+  accuracies, benchmark supplemental CSVs, and proteasome mass-spec additional
+  file generation.
+- Immunogenicity experiment: defer until external source files are located and
+  licensed; do not make this a release gate yet.
+
+## Porting Plan
+
+1. Add `scripts/analysis/` for reusable release-analysis scripts. Keep notebooks
+   out of the maintained path; use them only as references.
+2. Extract shared metric helpers: PPV@N, sign normalization, bootstrap intervals,
+   percent-change calculations, and sample/length grouping.
+3. Add a benchmark-assembly script that consumes a `data_evaluation` directory
+   and writes wide monoallelic/multiallelic analysis CSVs from the group files.
+4. Add an accuracy-table script that writes the notebook-style
+   `accuracy_scores.*.csv` tables from those wide benchmarks.
+5. Add figure scripts that consume the generated tables and produce stable PNG /
+   PDF outputs under a release-analysis directory.
+6. Add processing-motif generation from the presentation benchmark and trained
+   processing predictors. This should be scriptable without `logomaker` unless
+   the logo plot is explicitly requested; the workbook can be generated first.
+7. Decide the 2.3.0 holdout policy in one place: PMIDs, pMHC overlap removal,
+   and whether evaluation data is filtered out of affinity / processing /
+   presentation training. Move any useful logic from `jobs/filter_training...`
+   into maintained scripts after that decision.
+8. Resolve the processing variant naming: either train a real
+   `models.selected.with_flanks` variant again or document and test that
+   `short_flanks` is the canonical with-flank presentation input for 2.3.x.
+9. Make the release gate one command:
+   `scripts/release/retrain_evaluate_deploy.sh` trains, evaluates, plots, and
+   runs deployment validation. It supports local, runplz/Brev, and SSH-backed
+   remote machines.
+
+## Current Single-Command Entry Point
+
+Local:
+
+```bash
+scripts/release/retrain_evaluate_deploy.sh \
+    --run-dir /path/to/release-run \
+    --release 2.3.0 \
+    --backend local \
+    --deploy-mode dry-run
+```
+
+Brev through runplz:
+
+```bash
+scripts/release/retrain_evaluate_deploy.sh \
+    --run-dir /path/to/release-run \
+    --release 2.3.0 \
+    --backend runplz \
+    --deploy-mode dry-run
+```
+
+Generic SSH machine:
+
+```bash
+scripts/release/retrain_evaluate_deploy.sh \
+    --run-dir /path/to/local-copy \
+    --release 2.3.0 \
+    --backend ssh \
+    --remote user@host \
+    --remote-repo /path/to/mhcflurry \
+    --remote-run-dir /path/to/remote-release-run \
+    --deploy-mode dry-run
+```

docs/development.md

@@ -0,0 +1,32 @@
+# Development Maintenance
+
+## Private API and Fallback Cleanup
+
+MHCflurry should keep internal helpers movable. Tests and sibling modules should
+not pin underscore-prefixed functions unless the helper is truly private to the
+same module.
+
+1. Promote shared helpers before cross-module use.
+   If a helper is imported by another module or tested directly, give it a
+   public name in its owner module and document the contract in one sentence.
+
+2. Test owner modules, not compatibility facades.
+   Tests for moved helpers should import the helper module directly. Compatibility
+   shim tests should cover public behavior only: old imports still expose the
+   documented entry point and produce the same result.
+
+3. Make fallback paths observable.
+   Fallbacks for missing optional tools or hardware probes may return a safe
+   default. Fallbacks caused by import errors, bad local refactors, or invalid
+   user configuration should fail loudly instead of being caught by broad
+   `except Exception` blocks.
+
+4. Keep compatibility shims thin.
+   Old module paths may re-export public entry points and delegate unknown
+   attributes, but they should not replace `sys.modules` or assert identity with
+   the new implementation module.
+
+5. Track remaining private test hooks.
+   The remaining direct private imports in tests should be triaged into:
+   public helper, same-module implementation detail tested through behavior, or
+   compatibility shim slated for removal after the 2.3 release line stabilizes.

docs/index.md

@@ -9,4 +9,5 @@ python_tutorial
 commandline_tools
 api
 testing
+development
 ```

docs/orchestrator.md

@@ -25,8 +25,8 @@ orchestrator process               training worker process
 ─────────────────────              ───────────────────────
 
 parse args                         (forked / spawned)
-load training data                  inherit GLOBAL_DATA
-build GLOBAL_DATA           ─┐
+load training data                  inherit WORKER_CONTEXT
+build WORKER_CONTEXT        ─┐
                              │
 hoist env knobs              │
   TORCHINDUCTOR_COMPILE_THREADS
@@ -91,7 +91,7 @@ Two backends, one CLI surface:
   historical CPU-overflow behavior.
 - **Cluster** (`cluster_parallelism.py`): one job per work-item
   submitted via `bsub` / `sbatch` / `sh`. Workers serialize
-  GLOBAL_DATA to NFS and deserialize on the worker side.
+  WORKER_CONTEXT to NFS and deserialize on the worker side.
 
 Worker-side code (`train_model()`, etc.) is identical between
 backends. Only the orchestrator branches on `args.cluster_parallelism`.
@@ -269,7 +269,7 @@ before compiling losses to avoid the PyTorch 2.4 / Triton
 
 1. Add a `_initialize_<name>(args, all_work_items)` helper near the
    existing ones in `train_pan_allele_models_command.py`.
-2. Stash the result in `GLOBAL_DATA["<name>"]`.
+2. Stash the result in `WORKER_CONTEXT["<name>"]`.
 3. Document the lookup key. Workers retrieve via
    `constant_data["<name>"]` (forked workers inherit; spawned/cluster
    workers receive via pickle).