Skip to content

Commit 86782d7

Browse files
ProfessorDongProfessorDong
committed
Reliability cost of novelty in molecular design: code, data, and frozen outputs
Curated activity data for five targets (SCD-1, FADS, NK1R, DRD2, DRD3), the activity models and reward, the search operators, and the experiment and analysis scripts behind the reported results: - ensemble disagreement as a validated reliability score (risk-coverage, partial correlations against novelty and distance to training) - novelty-driven distribution shift across optimizers and penalties - fingerprint-surrogate triage for budgeted search, with a same-pool random-triage control and hierarchical target-level statistics - reported negative results: uncertainty acquisition, latent surrogate, cross-target compatibility, leakage-free scaffold recovery verify_results.py asserts 60 numeric claims against the frozen outputs.
0 parents  commit 86782d7

109 files changed

Lines changed: 85983 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
__pycache__/
2+
*.pyc
3+
*.pkl
4+
!outputs/cwm_v1/chembl_fragment_pool.pkl
5+
*.log
6+
.ipynb_checkpoints/

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Liang Dong
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Quantifying the reliability cost of novelty in molecular design
2+
3+
Code, curated data, and frozen experimental outputs for the study of how prediction
4+
reliability degrades as a molecular search is pushed toward novelty, across five targets
5+
relevant to therapy and molecular imaging: **SCD-1**, **FADS** (endoplasmic-reticulum
6+
membrane lipid desaturases) and **NK1R**, **DRD2**, **DRD3** (class A G-protein-coupled
7+
receptors).
8+
9+
Author: **Liang Dong**.
10+
11+
Every number reported in the manuscript and its Supplementary Information is produced by a
12+
frozen analysis script run against a frozen output file in `outputs/cwm_v1/`, and is checked
13+
by `verify_results.py`.
14+
15+
## Summary of what this code establishes
16+
17+
1. **A validated reliability score.** The standard deviation of the per-tree predictions of a
18+
random-forest activity model, `sigma_T`, predicts the model's *measured* error
19+
(pooled Spearman 0.40 over 21,173 out-of-fold predictions). Retaining the
20+
lowest-disagreement fifth of predictions reduces pooled RMSE from 0.71 to 0.41 pIC50.
21+
The score remains predictive after controlling for both support-novelty and distance to
22+
the training set (partial Spearman 0.38).
23+
2. **Novelty drives molecules off the training distribution.** Sweeping a novelty incentive
24+
for two optimizers at two uncertainty penalties (1,200 runs) drives generated molecules
25+
away from the training distribution (Spearman 0.97-0.98), raising `sigma_T` and lowering
26+
predicted potency. This holds with the uncertainty penalty set to zero. Support-novelty
27+
by itself is a weak predictor of measured error (pooled Spearman 0.04).
28+
3. **A fingerprint surrogate improves budgeted search.** An online ECFP random-forest
29+
surrogate that triages a Graph GA offspring pool under a fixed evaluation budget improves
30+
the top-10 reward over vanilla Graph GA in all 15 target-by-k cells (target-level mean
31+
+0.057, 95% CI [+0.030, +0.083], P = 0.004), beats a same-pool random-triage control, and
32+
outperforms a learned dual-encoder latent surrogate on every target.
33+
4. **Reported negative results.** The uncertainty term in the acquisition does not help
34+
end to end; a learned latent representation underperforms fingerprints; source-target
35+
compatibility does not predict warm-start benefit; and targeted scaffold recovery is not
36+
improved by the triage.
37+
38+
## Reproducing the results
39+
40+
```bash
41+
pip install -r requirements.txt
42+
43+
# 1. Build and gate the activity models (writes outputs/cwm_v1/oracle_{target}.pkl).
44+
# Required before any search experiment; the .pkl files are NOT in this repository
45+
# (see "Large files" below).
46+
python -m world_model.oracle_sanity_gate
47+
48+
# 2. Reliability analyses (CPU only; these refit the models internally)
49+
python -m world_model.analyze_calibration # sigma_T vs measured error
50+
python -m world_model.run_applicability_v1 # distance-to-training, partial correlations
51+
python -m world_model.run_reliability_v1 # measured error vs novelty; risk-coverage
52+
53+
# 3. Search experiments (need the .pkl activity models from step 1)
54+
python -m world_model.run_frontier_v2 # novelty sweep x 2 optimizers x 2 penalties
55+
python -m world_model.run_methods_v2 # primary method comparison (fingerprint surrogate)
56+
python -m world_model.run_ecfp_baseline_v1 # fingerprint vs dual-encoder latent surrogate
57+
python -m world_model.run_beta_ablation_v1 # uncertainty-acquisition ablation
58+
python -m world_model.run_recovery_v2 # leakage-free scaffold-cluster recovery
59+
python -m world_model.run_hierstats_v1 # hierarchical / target-level statistics
60+
61+
# 4. Check every reported number against the frozen outputs
62+
python verify_results.py
63+
```
64+
65+
`verify_results.py` asserts 60 numeric claims and exits non-zero on any mismatch.
66+
67+
## Repository layout
68+
69+
```
70+
data/ curated per-target activity data (SMILES, pIC50) + featurizers
71+
scd1_binding.csv 762 compounds (active threshold 7.0)
72+
fatty_acid_desaturase_bioactivity.csv 1,187 (active threshold 6.5)
73+
nk1r_combined.csv 3,056 (active threshold 7.0)
74+
drd2_bioactivity.csv 9,966 (active threshold 6.5)
75+
drd3_chembl.csv 6,202 (active threshold 7.0)
76+
world_model/ activity model, reward, search operators, experiments
77+
oracle.py activity model API, target config, ECFP featurization
78+
oracle_sanity_gate.py model selection + gating; writes the frozen .pkl models
79+
reward.py multi-objective reward with the uncertainty penalty
80+
actions.py BRICS fragment-edit action space + admissibility filter
81+
graph_ga.py Graph GA baseline (crossover / mutation)
82+
wm_guided_ga.py surrogate-triaged search
83+
dynamics.py dual-encoder latent surrogate (ablation only)
84+
run_reliability_v1.py measured error vs novelty and distance; risk-coverage
85+
run_applicability_v1.py distance-to-training analysis; partial correlations
86+
run_frontier_v2.py novelty sweep across optimizers and penalties
87+
run_methods_v2.py primary method comparison (fingerprint surrogate)
88+
run_ecfp_baseline_v1.py fingerprint vs latent surrogate
89+
run_beta_ablation_v1.py uncertainty-acquisition ablation
90+
run_recovery_v2.py leakage-free scaffold-cluster recovery
91+
run_hierstats_v1.py hierarchical / target-level statistics
92+
analyze_*.py frozen analysis scripts for the individual experiments
93+
outputs/cwm_v1/ frozen result files; the source of every reported number
94+
verify_results.py asserts every reported number against those files
95+
```
96+
97+
## Data
98+
99+
The per-target datasets are curated from ChEMBL and the primary literature under a uniform
100+
pipeline. Activities are pooled to pIC50; replicate ChEMBL measurements are aggregated to one
101+
value per canonical structure before thresholding, while the SCD-1 literature-curated binding
102+
set retains a small number of replicate records. FADS aggregates FADS1 and FADS2 desaturase
103+
activity drawn mostly from a curated literature panel, so its activity function is the most
104+
heterogeneous of the five.
105+
106+
Throughout, "few-seed" refers to the k in {5, 10, 20} starting active compounds supplied to
107+
the search, not to the activity model, which is trained on all measured compounds for a target.
108+
109+
## Large files
110+
111+
The frozen activity models `outputs/cwm_v1/oracle_{target}.pkl` are not included: the DRD2
112+
model exceeds the 100 MB per-file limit on GitHub. Rebuild them with
113+
`python -m world_model.oracle_sanity_gate`, which reproduces them deterministically from the
114+
data in this repository. The frozen JSON outputs that back the reported numbers are included,
115+
so `verify_results.py` runs without rebuilding the models.
116+
117+
## A note on internal naming
118+
119+
The package directory (`world_model/`), some module and class names (`wm_guided_ga`,
120+
`WMGuidedGA`, `LatentWorldModel`, `CWMReward`) and the keys of the frozen JSON outputs
121+
retain identifiers from an earlier version of this work. They are kept unchanged so that the
122+
frozen outputs and the reproduction path stay valid, and they carry no meaning beyond naming.
123+
The primary method reported in the manuscript is the fingerprint-surrogate triage in
124+
`run_methods_v2.py`.
125+
126+
## License
127+
128+
MIT. See `LICENSE`.

‎__init__.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Theranostics Drug Discovery Framework
2+
# Dual Graph + SMILES representation with Meta-RL (MAML)
3+
# Attention visualization and causal inference support
4+
5+
from .models import *
6+
from .data import *
7+
from .meta_learning import *
8+
from .reinforcement import *
9+
from .interpretability import *
10+
from .utils import *
11+
12+
__version__ = '1.0.0'
13+
__author__ = 'Theranostics Team'

‎data/__init__.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Data Module
2+
from .dataset import MoleculeDataset, SMILESDataset, GraphDataset, DualDataset
3+
from .featurizer import MoleculeFeaturizer, SMILESTokenizer, GraphFeaturizer
4+
from .loader import GeneratorData, create_data_loaders
5+
6+
__all__ = [
7+
'MoleculeDataset', 'SMILESDataset', 'GraphDataset', 'DualDataset',
8+
'MoleculeFeaturizer', 'SMILESTokenizer', 'GraphFeaturizer',
9+
'GeneratorData', 'create_data_loaders'
10+
]

0 commit comments

Comments
 (0)