add benchmarking numbers

Author: etowahadams

.gitignore

@@ -8,3 +8,6 @@ dist/
 .venv/
 venv/
 .DS_Store
+
+
+examples/rnp100_ab/*

README.md

@@ -12,10 +12,38 @@ Implements the two mechanisms used by Boltz-1x and Protenix-v2:
 * **Feynman-Kac steering** — sample a particle population and resample it
   toward low-energy trajectories.
 
-The engine is host-agnostic. It consumes a `ChemicalContext` of atom-indexed
-constraint tensors and exposes two pure functions (`guide`, `resample_indices`);
-it never sees the denoiser, the noise schedule, or the model. Adding a host
-means writing one adapter.
+## Benchmark on OF3p2
+
+OpenFold3-p2 (155k) on the 100 smallest Runs N' Poses post-2025 targets, 1 seed
+× 5 samples, guidance only (20 GD steps), scored with PXMeter. Rates are over
+the 103 ligand chains, each represented by its **top-1 sample ranked by
+`chain_pair_iptm`**.
+
+| | baseline | steered |
+| --- | --- | --- |
+| PoseBusters valid | 66.0% | **89.3%** |
+| PB valid & ligand RMSD < 2 Å & lDDT-PLI > 0.8 | 38.8% | **47.6%** |
+| ligand RMSD < 2 Å & lDDT-PLI > 0.8 | 49.5% | 50.5% |
+| wall clock, 4× GH200 | 9m56s | 17m11s (**1.73×**) |
+
+Steering fixes chemistry without moving docking accuracy: paired ligand by
+ligand it fixed 25 validity failures and broke 1 (McNemar p < 1e-4), while
+accuracy moved by a net +1 (p = 1.0). Nearly all of the gain is sterics —
+`minimum_distance_to_protein` failures fall 141 → 3 across all 515 poses;
+chirality is second (31 → 13).
+
+Caveat: 10 of 161 multi-atom ligand chains were skipped because RDKit rejected
+the molecule rebuilt from `atom_array` (valence errors on quaternary nitrogen
+and boron, which need a formal charge the rebuild does not assign), leaving 7 of
+the 100 targets unsteered and still counted in the steered arm.
+
+
+# TODOs
+
+- Test out the FK steering to see how much better / slower that is 
+- See how valency errors can be overcome (possibly some change to featureization is needed?)
+
+# Usage 
 
 ## Install
 
@@ -24,34 +52,41 @@ pip install -e .                 # engine only, no folding model needed
 pip install -e '.[openfold3]'    # with the OpenFold3 adapter
 ```
 
+**Already have OpenFold3 installed?** Install foldsteer into that same
+environment with `pip install -e . --no-deps` — it must share the interpreter
+with OF3 (see below), and `--no-deps` keeps pip from touching your pinned
+torch. Nothing about the OF3 install changes; foldsteer imports it lazily and
+patches at runtime. You need OF3 >= 0.4.5 (or any `main` carrying
+`SampleDiffusion._sample_rollout`; at tag 0.4.4 that loop is still inlined in
+`forward` and the adapter will raise `AttributeError`).
+
 ## Use with OpenFold3
 
-Under OF3's runner Lightning builds the model itself, so patch the class before
-inference and invoke the CLI **in the same process**:
+From the command line, `examples/run_of3_steered.py` does the patches OF3 inference to use the steering.
+It passes everything after `--` straight to `run_openfold
+predict`, so the steered and unsteered arms take identical OF3 arguments:
 
-```python
-from foldsteer import default_config
-from foldsteer.adapters.openfold3 import patch_sample_diffusion_class
-from openfold3.run_openfold import cli
-
-unpatch = patch_sample_diffusion_class(default_config(fk_steering=False))
-cli.main(["predict", "--query-json", ..., "--output-dir", ...],
-         standalone_mode=False)
-unpatch()
+```bash
+python examples/run_of3_steered.py --steer --num-gd-steps 20 \
+    --stats-json steering_stats.json -- \
+    --query_json examples/query_protein_ligand.json \
+    --inference_ckpt_path /path/to/of3-p2-155k.pt \
+    --num_diffusion_samples 5 \
+    --use_msa_server false --use_templates false \
+    --output_dir out_steered
 ```
 
-If you already hold the model object, patch that instance instead:
-
-```python
-from foldsteer.adapters.openfold3 import patch_sample_diffusion
-
-unpatch = patch_sample_diffusion(model, default_config(num_particles=3))
-```
+Swap `--steer` for `--no-steer` to get the baseline. **Always check the
+`[foldsteer] targets=... ligand_chains=... guided_steps=...` line it prints**: a
+run that found no steerable ligand produces output indistinguishable from an
+unsteered one.
 
-Guidance only is the default above: it is the cheaper half and carries most of
-the benefit. Feynman-Kac steering reuses OF3's rollout-sample axis as the
-particle axis, so `num_diffusion_samples` must be a multiple of `num_particles`
-and fewer structures come back than were sampled.
+Guidance only is the default: it is the cheaper half and carries the benefit
+measured below. **Feynman-Kac steering has not been tested end-to-end against a
+real model** — it is exercised only by unit tests against a mock sampler. It
+also reuses OF3's rollout-sample axis as the particle axis, so
+`num_diffusion_samples` must be a multiple of `num_particles` and fewer
+structures come back than were sampled.
 
 A complete runnable example — query JSON, both arms, and how to confirm
 steering actually fired — is in [`examples/`](examples/README.md).
@@ -68,18 +103,6 @@ engine = SteeringEngine(default_config(), ctx)
 coords = coords + engine.guide(coords, t=0.5)   # t: 1 = noisy, 0 = clean
 ```
 
-## Validation
-
-![validation](docs_validation.png)
-
-**(a)** Guidance relaxes a distorted ligand. **(b)** Ablation over 6 seeds
-against a mock sampler reproducing OpenFold3's rollout contract. **(c)** The
-package detects the failure modes Protenix-v2 reports slipping past standard
-PoseBusters validity; valid geometry scores exactly zero.
-
-Every analytic gradient is verified against autograd to ~1e-15. See
-[DESIGN.md](DESIGN.md) §7 for the full results and caveats.
-
 ## Potentials
 
 `BoundsMatrixPotential`, `VDWOverlapPotential`, `ChiralAtomPotential`,
@@ -106,5 +129,6 @@ OpenFold3 `AtomArray` reconstruction path has now been exercised against real
 OF3 inference input — which turned up three silent failures in it (integer
 `MoleculeType` ids, Kekulé order on aromatic bonds, stereo perceived from
 `ref_pos` rather than the not-yet-predicted `coord`), all fixed and pinned by
-`tests/test_of3_extraction.py`. Steering quality itself has not been benchmarked
-at scale from this repo.
+`tests/test_of3_extraction.py`. Guidance is now benchmarked against real OF3
+inference (see Benchmark above); Feynman-Kac steering is not, and the formal
+charge gap in the `AtomArray` rebuild is the clearest outstanding fix.