Skip to content

README.md

Latest commit

 

History

History
118 lines (94 loc) · 5.68 KB

README.md

File metadata and controls

118 lines (94 loc) · 5.68 KB
Raw

dynbench

A metric-calibrated benchmark for protein conformational dynamics. Dynbench is a measurement protocol, not a new generator or a single new metric. It reads every score against a target-removed floor and a metric-specific operational reference. Most temporal relaxation metrics use an Ornstein-Uhlenbeck operational reference; transition and mean-first-passage metrics use a held-out MD replicate. These references are empirical rulers rather than theoretical maxima. Dynbench also applies geometric and fluctuation guardrails defined separately from the temporal metrics, and enforces applicability by task so that unordered ensemble models are not assigned a temporal failure.

The central controlled result is that an i.i.d. resample of equilibrium frames attains an ensemble informative-pass fraction of 0.99 but a post-gate kinetic pass rate of 0.00. A held-out MD replicate reaches 0.95 under the same fixed-resolution test. Ensemble agreement therefore does not, by itself, establish learned temporal dynamics.

Install

pip install -r dynbench/requirements.txt

Score a submission

A submission is a per-protein NumPy archive containing C-alpha coordinates (T, L, 3), the reference replicates and an ordered flag. The flag is true for a time-ordered trajectory and false for an unordered ensemble. The adapter also records the physical frame interval when a temporal claim is made.

python -m dynbench.run --input_dir submissions/<model> --out <model>_report.json --n_modes 10
python -m dynbench.qc_model --report <model>_report.json --subs submissions/<model> \
    --name <model> --official frozen_board_v8/official_split.json --out qc/<model>_qc.json

To score on a machine without the full ATLAS H5 file, first build a compact reference bundle on a machine that has the source trajectories:

python -m dynbench.scripts.make_ref_bundle --h5 data.h5 \
    --split frozen_board_v8/official_split.json --out ref_bundle_eval.npz

Model-specific adapter examples are provided in dynbench/scripts/.

Measurement layers

Layer Question
L1 frame Is an individual frame distinguishable from a frame-level foil?
L2 ensemble Does the sample set reproduce the reference conformational ensemble?
L3 temporal For an ordered and supported contract, are temporal statistics reproduced?
L4 admissibility Does the rollout pass C-alpha geometric and fluctuation guardrails?

The headline kinetic pass rate is the fraction of proteins that pass the majority of informative temporal metrics and the Layer-4 guardrails. It is not the fraction of physical dynamics learned. The public JSON schema retains the historical field name realdyn for backward compatibility. Unordered submissions receive NA on temporal metrics rather than zero.

Frozen board v8

The board contains thirteen rows on one frozen 82-protein ATLAS split: four calibration anchors and comparators, four unordered ensemble generators, and five ordered generators.

  • Split: frozen_board_v8/official_split.json, SHA-256-derived split identifier c42f261eb93b57fb (the first 16 hexadecimal characters of the canonical split-content digest; the manuscript table displays c42f261e). The serialized file digest is recorded in the content manifest.
  • Board and confidence intervals: frozen_board_v8/leaderboard.json and frozen_board_v8/leaderboard_ci.json (1,000 protein-cluster bootstrap draws, seed 0).
  • AlphaFlow provenance: audits/alphaflow_v7_ci_provenance.json records the v7 Layer-2 confidence interval carried into Table 1. No v8 temporal or Layer-4 value is inferred for AlphaFlow.
  • Matched coverage: frozen_board_v8/matched_subset.json.
  • Sequence-overlap sensitivity: the post hoc MMseqs2 audit is active; the earlier k-mer screen is retained only as version history.
  • Content manifest: frozen_board_v8/manifest.json records the SHA-256 digest and byte count of every frozen-board file.
  • Guardrail sweep: frozen_board_v8/l4_guardrail_sweep.json replays the Layer-4 constants and exactly reproduces the frozen defaults.
  • Reportability audit: audits/native_contract_interpretability_v8.json and audits/contract_reportability_grid_v8.json record the reference-to-floor separation and temporal support checks.
  • Figures: figures/ contains five main and five Extended Data figures. Vector PDF/SVG and submission EPS/TIFF/PNG formats are included in the deposition archive together with source data and rendering scripts.

Only the 1-ns ConfRover and MDGen contracts satisfy the operational reportability criteria. BioKinema and ProTDyn retain insufficient reference-to-floor range at their 10-ns contracts, and MarS-FM retains only three reference frames and a median of one informative temporal metric at approximately 50 ns. Their raw values remain diagnostic records and are not placed in a temporal ranking. BioKinema is evaluated in verified intended-MSA mode; its model-specific overlap audit is included with the release.

Reproduce selected audits

python audits/realdyn_threshold_sweep.py
python audits/l4_guardrail_sweep.py
python audits/oracle_robustness.py
python audits/contract_reportability_grid.py
python audits/recompute_peptide_temporal_only.py
python audits/build_figure5_temporal_summary.py

The release archive contains the exact inputs and additional scripts needed for the reported operational-reference, external mdCATH, temporal-resolution and reference-assisted diagnostic audits.

Citation and archive

See CITATION.cff. The manuscript release is prepared for deposition under the reserved DOI 10.5281/zenodo.21376741. The final repository tag, commit SHA and archive manifest are recorded at release time.

License

MIT. See LICENSE.