Reproduce Results

[kalifadan]

Hey, I'm trying to get the results of the EC and Thermostability tasks, with the following config, but getting lower results (for example, 0.712 in the Thermostability and 0.866 in the EC). What can it be? Is the number of epochs too big?
Thank you!!

EC:

setting:
seed: 20000812
os_environ:
WANDB_API_KEY: ~
WANDB_RUN_ID: ~
CUDA_VISIBLE_DEVICES: 0,1,2,3 # ,4,5,6,7
MASTER_ADDR: localhost
MASTER_PORT: 12315
WORLD_SIZE: 1
NODE_RANK: 0
wandb_config:
project: EC
name: SaProt_650M_AF2

model:
model_py_path: saprot/saprot_annotation_model
kwargs:
config_path: weights/PLMs/SaProt_650M_AF2
load_pretrained: True
anno_type: EC

lr_scheduler_kwargs:
last_epoch: -1
init_lr: 2.0e-5
on_use: false

optimizer_kwargs:
betas: [0.9, 0.98]
weight_decay: 0.01

save_path: weights/EC/SaProt_650M_AF2.pt

dataset:
dataset_py_path: saprot/saprot_annotation_dataset
dataloader_kwargs:
batch_size: 4 # 8
num_workers: 4 # 8

train_lmdb: LMDB/EC/AF2/foldseek/train
valid_lmdb: LMDB/EC/AF2/foldseek/valid
test_lmdb: LMDB/EC/AF2/foldseek/test
kwargs:
tokenizer: weights/PLMs/SaProt_650M_AF2
plddt_threshold: 70

Trainer:
max_epochs: 100
log_every_n_steps: 1
strategy:
find_unused_parameters: True
logger: True
enable_checkpointing: false
val_check_interval: 0.1
accelerator: gpu
devices: 4 # 8
num_nodes: 1
accumulate_grad_batches: 4 # 1
precision: 16
num_sanity_val_steps: 0

Thermostability:
setting:
seed: 20000812
os_environ:
WANDB_API_KEY: ~
WANDB_RUN_ID: ~
CUDA_VISIBLE_DEVICES: 0,1,2,3 # ,4,5,6,7
MASTER_ADDR: localhost
MASTER_PORT: 12315
WORLD_SIZE: 1
NODE_RANK: 0
wandb_config:
project: Thermostability
name: SaProt_650M_AF2

model:
model_py_path: saprot/saprot_regression_model
kwargs:
config_path: weights/PLMs/SaProt_650M_AF2
load_pretrained: True

lr_scheduler_kwargs:
last_epoch: -1
init_lr: 2.0e-5
on_use: false

optimizer_kwargs:
betas: [0.9, 0.98]
weight_decay: 0.01

save_path: weights/Thermostability/SaProt_650M_AF2.pt

dataset:
dataset_py_path: saprot/saprot_regression_dataset
dataloader_kwargs:
batch_size: 4 # 8
num_workers: 4 # 8

train_lmdb: LMDB/Thermostability/foldseek/train
valid_lmdb: LMDB/Thermostability/foldseek/valid
test_lmdb: LMDB/Thermostability/foldseek/test
kwargs:
tokenizer: weights/PLMs/SaProt_650M_AF2
mix_max_norm: [40, 67]
plddt_threshold: 70

Trainer:
max_epochs: 200
log_every_n_steps: 1
strategy:
find_unused_parameters: True
logger: True
enable_checkpointing: false
val_check_interval: 0.5
accelerator: gpu
devices: 4
num_nodes: 1
accumulate_grad_batches: 8
precision: 16
num_sanity_val_steps: 0

[LTEnjoy]

Hi @kalifadan ,

What GPU did you run the experiments on? It should not be the problem of the number of training epochs as in our implementation the best model on validation set would be saved. There are many factors that would affect the final performance, such as gradient accumulation, batch size, type of GPUs etc. For fair comparison, you could probably fine-tune ESM-2 with the same setting and see how it performs.

[kalifadan]

Thank you for the comment!

My GPUs are:
256 AMD EPYC 7742 64-Core
2.0TB RAM
4 x NVIDIA A100 – 80GB

I follow the values in your paper for the batch size and gradient accumulation, mentioning it should be a batch size of 64. Therefore, In my case of 4 devices, I set the batch to be 4 and accumulate_grad_batches to be 4, I also tried 8 but it produced similar results.

If you can compare it to your values it will help :)

[LTEnjoy]

Hi,

I ran the experiments on 8 A100 GPUs, setting batch size to 8 and gradient accumulation to 1, which formed the batch size of 64. I guess the implementation of gradient accumulation in pytorch-lightning may influence the final result?

If you could fine-tune ESM-2 with the same setting you fine-tuned SaProt and find its result still lower than reported by our paper, I think it is something like systematic bias?

[kalifadan]

I'm trying now :)
Can you provide the max_epochs for each task? does it equal to the one provided in the config dir?

[LTEnjoy]

Yes. It is listed in each config file for each task:)

[kalifadan]

Nice!
Can you please provide the full SaProt model trained already on the downstream tasks?
In order to reproduce the results on the downstream tasks?
Thank you!!!

[LTEnjoy]

Sorry we have no longer saved those models because it has been too long since the paper was released (around 1.5 year) :(
Please let me know if there are any other things that I can help!