Reproduce problem

[VicFisher]

Hi,
Thank you for sharing OMTRA, which is really a wonderful project.

Recently, I have mannually installed the project as suggested and downloaded the model weights. I tried to reproduce the results on CrossDocked in the OMTRA paper with the provided code:
python ./omtra_pipelines/docking_eval/docking_eval.py \ --ckpt_path omtra/trained_models/protpharm_cond_cr_2025-11-24_13-00-028503/checkpoints/last.ckpt \ --task fixed_protein_ligand_denovo_condensed \ --dataset crossdocked \ --split val \ --n_samples 86 \ --n_replicates 100 \ --max_batch_size 100 \ --timeout 2700

This command throws:
FileNotFoundError: [Errno 2] No such file or directory: '/xxx/OMTRA/omtra/trained_models/protpharm_cond_cr_2025-11-24_13-00-028503/.hydra/config.yaml'

I checked the checkpoint directory, but didn't find the requested file. I also checked the provided downloading link 'https://bits.csb.pitt.edu/files/OMTRA/omtra_v0_weights/', but also didn't find something related.

So, how to solve the reproduce problem? Do you have any suggestions?

Kind regards,
JunLin Yu.

[VicFisher]

Besides, the model checkpoint description in the code base and the manuscript makes me a little confused.

The task and checkpoint in omtra.utils.checkpoint gives the following correlation, which is easy to understand:
unconditional:
"denovo_ligand_condensed": "uncond_xxxxxx/xxx/last.ckpt"
"ligand_conformer_condensed": "uncond_xxxx/xxx/last.ckpt"

pharmacophore-conditioned:
"denovo_ligand_from_pharmacophore_condensed": "phcond_xxx/xxx/last.ckpt"
"ligand_conformer_from_pharmacophore_condensed": "phcond_xxx/xxx/last.ckpt"

protein-conditioned:
"rigid_docking_condensed": "mt_plcd_cr_xxx/xxxx/last.ckpt"
"fixed_protein_ligand_denovo_condensed": "mt_plcd_cr_xxx/xxx/last.ckpt",

Protein + pharmacophore -conditioned:
"rigid_docking_pharmacophore_condensed": "protpharm_cond_cr_xxx/xxx/last.ckpt"
"fixed_protein_pharmacophore_ligand_denovo_condensed": "protpharm_cond_cr_xxx/xxx/last.ckpt"

Regarding these, I have two specific questions:

• I am not clear whether the checkpoint for protein-conditioned sampling ('mt_plcd_cr_xxx/xxx/last.ckpt') is the multi-task version in the manuscript? Which is the checkpoint trained and evalated on CrossDocked benchmark (Table 1 in the manuscript)? Is it just 'mt_plcd_cr_xxx/xxxx/last.ckpt'? But it seems to be trained on plinder and crossdocked?
• Besides, is the checkpoint for docking 'st_docking_cr_ext_xxx' or 'mt_plcd_cr_xxx/xxxx/last.ckpt' in Table 2 of the manuscript?
• If possible, could you please list the corresponding checkpoint of each evaluation in the manuscript to ensure the correctness?

Thank you for your kind help.

[dkoes]

bump @LivToft

[VicFisher]

@LivToft @dkoes , I’m checking in to see if you’ve had a chance to look at this yet. I’m currently at a standstill on the next steps of OMTRA's evaluation reproduction and would greatly appreciate your guidance whenever you have a moment to spare. Thank you!

[dkoes]

Sorry for the very delayed response. The file is actually there on bits, but you can't see it because the .hydra directory isn't visible. To simplify everything, I have tarred up all the models so the hidden files/directories don't get omitted from a download:
https://bits.csb.pitt.edu/files/OMTRA/omtra_v0_weights.tgz

[dkoes]

If you just want the file: https://bits.csb.pitt.edu/files/OMTRA/omtra_v0_weights/phcond_seeded_r192_debug_2025-11-20_16-15-701316/.hydra/config.yaml

[VicFisher]

Thank you for your help.

I have spent some time on reading the code and finally found that .hydra/config.yaml is loaded while not really used in the code. Simply skipping the config file loading seems to work. However, there are still some other problems making me stuck:
(1) The dataset caches for the benchmark sets (crossdocked, posebusters, plinder) seemed missing.
(2) The corresponding of the provided model checkpoints with those used in the manuscript is a little ambiguous.

I am currently using the omtra command line to reproduce the results. I am not sure whether it's different from the provided way.

[dkoes]

(1) You should be able to reconstruct these caches from publicly available data using out processing code, but I will check to see if these are reasonable enough in size to provide as downloads.
(2) Please provide your assumed correspondences and we will confirm if they are correct.

For reproducing and reporting results, please refer to this model as the OMTRA-MLSB model. We are working on improving the model and expect the official 1.0 release to be significantly better.

[VicFisher]

Thank you very much for the guidance. I am currently focusing on benchmarking the model using CrossDocked and PoseBusters across different levels of pharmacophore information (protein-based, pharmacophore-based, and joint protein-pharmacophore-based).

Regarding the points mentioned:

(1) Data Processing and Caches
I have reviewed the processing code, but encountered a few hurdles regarding the directory structures:

• CrossDocked: The type file directory /net/galaxy/home/koes/paf46_shared/cd2020_v1.3/types appears to be missing, and the overall file structure within the root directory /net/galaxy/home/koes/paf46_shared/cd2020_v1.3 is currently unclear.
• PoseBusters: I was unable to find specific instructions or documentation for processing this dataset.

If it is feasible to provide these caches as downloads, it would greatly assist in ensuring the environment is set up correctly.

(2) Model Correspondences
I am currently using the task-checkpoint mappings defined in TASK_TO_CHECKPOINT within OMTRA/omtra/utils/checkpoints.py. Specifically:

# Pharmacophore-conditioned de novo ligand generation
"denovo_ligand_from_pharmacophore_condensed": "phcond_seeded_r192_debug_xxx/checkpoints/last.ckpt", 

# Pharmacophore-conditioned ligand conformer generation
"ligand_conformer_from_pharmacophore_condensed": "phcond_seeded_r192_debug_xxx/checkpoints/last.ckpt",  

# Rigid docking
"rigid_docking_condensed": "mt_plcd_cr_xxx/checkpoints/last.ckpt", 

# Rigid protein, de novo ligand generation
"fixed_protein_ligand_denovo_condensed": "mt_plcd_cr_xxx/checkpoints/last.ckpt",  

# Pharmacophore-conditioned rigid docking
"rigid_docking_pharmacophore_condensed": "protpharm_cond_cr_xxx/checkpoints/last.ckpt",  

# Rigid protein + pharmacophore, de novo ligand generation
"fixed_protein_pharmacophore_ligand_denovo_condensed": "protpharm_cond_cr_xxx/checkpoints/last.ckpt"

I would like to confirm if the checkpoint used to "train and evaluate OMTRA on the CrossDocked dataset" corresponds to the multi-task version (mt_plcd_cr_xxx). I found this slightly confusing for the docking task. Additionally, since the manuscript notes the data leakage risks of PoseBusters when evaluated against Plinder, I suspect the docking model checkpoint might be trained on Plinder. Could you clarify if these two tasks utilize the same checkpoint or if they are distinct?

We will make sure to refer to the model as OMTRA-MLSB in our reporting. We look forward to the official 1.0 release!

[VicFisher]

By the way, I have some other questions:
(1) I also would like to confirm that will the omtra command line way of reproduction be different? I have sampled the molecules for all the benchmarks, waiting for evaluation. If you can provide the requested dataset cache, do I need to re-sample?
(2) Besides, the score function used for docking is vina score mentioned in the manuscript. Is there any difference between the gnina version with the original vina (version 1.2.2).

[LivToft]

Hi JunLin,
To answer your questions in order:

(1) Data Processing and Caches

• CrossDocked: This is an absolute file path that points to a local copy of the dataset, which is why you cannot access it (we have residual absolute file paths in our code, sorry about that). You can download the relevant files here. Note, we use CrossDocked v1.3
• PoseBusters: We did not publish our PoseBusters processing pipeline. We are working on getting it ready and will publish it shortly

(2) Model Correspondences

• That is correct.mt_plcd_cr_xxx is a multi-task model that was trained on both protein-conditioned de novo ligand design (fixed_protein_ligand_denovo_condensed) and rigid docking (rigid_docking_condensed). The training dataset is both Plinder (pl) and CrossDocked (cd)
• Yes, the docking model evaluated with PoseBusters was trained on Plinder. PoseBusters is an evaluation benchmark that is explicitly designed to prevent leakage from PDBbind (v2020). There is no guarantee that it is leakage-free from Plinder (and is likely not).
• The tasks use the same checkpoint

[VicFisher]

Hi,
Thank you for your help and provided data.

So, I want to finally confirm whether the correspondence in the TASK_TO_CHECKPOINT is OK for the evaluation pipeline? The model checkpoint should be specified for the provided evaluation script.

Best regards.

[LivToft]

Yup you can use those checkpoints for the evaluation script

[VicFisher]

OK. Thank you for your patience and kind help.