Exact Script for Reproducing the Results on rMD17 Dataset

[yilunliao]

Hello.

Thanks for open-souring the great work.

I have some questions regarding rMD17:

1. Can you provide the exact script for reproducing the rMD17 results in the paper? I check the documentation but do not find a script for rMD17.
2. Can you also point out how and where you download rMD17 dataset? I think we need to download some .xyz files on our own before running training.

Thank!

[ilyes319]

Hey,

1. There have been quite a lot of changes to the interface, so it is not the original script, but here is an input that will reproduce the result for ethanol:

python ./mace/scripts/run_train.py \
    --name="MACE_ethanol" \
    --train_file="/rds/user/ib467/hpc-work/Equivariant_GNN/mace_rmd17/rmd17_ethanol_train.xyz" \
    --valid_fraction=0.05 \
    --test_file="/rds/user/ib467/hpc-work/Equivariant_GNN/mace_rmd17/rmd17_ethanol_test.xyz" \
    --energy_weight=8.0 \
    --forces_weight=1000.0 \
    --config_type_weights='{"Default":1.0}' \
    --E0s='average' \
    --model="ScaleShiftMACE" \
    --interaction_first="RealAgnosticResidualInteractionBlock" \
    --interaction="RealAgnosticResidualInteractionBlock" \
    --num_interactions=2 \
    --error_table="TotalMAE" \
    --max_ell=3 \
    --hidden_irreps='256x0e + 256x1o + 256x2e' \
    --num_cutoff_basis=5 \
    --correlation=3 \
    --r_max=6.0 \
    --scaling='rms_forces_scaling' \
    --batch_size=5 \
    --max_num_epochs=3500 \
    --lr=0.01 \
    --patience=200 \
    --swa_forces_weight=1000.0 \
    --swa_energy_weight=8.0 \
    --swa_lr=0.001 \
    --weight_decay=5e-7 \
    --ema \
    --ema_decay=0.99 \
    --amsgrad \
    --default_dtype="float32"\
    --swa \
    --start_swa=2000 \
    --clip_grad=10 \
    --device=cuda \
    --seed=3 \
    --restart_latest \

For other molecules just change the energy_weight to the number of atoms in this molecule.

2. Here the link: https://figshare.com/articles/dataset/Revised_MD17_dataset_rMD17_/12672038. For each molecule, sample 1000 for training (using 50 for val) and 1000 for test. Note that the units are in kcal/mol and you need to convert them to eV for MACE.

[yilunliao]

I see. I will try this later.

Thank a lot for the help, @ilyes319 and @davkovacs !

[yilunliao]

Hi @ilyes319 and @davkovacs

I have several questions regarding reproducing the results of ethanol:

1. In the splits provided by revised MD17 dataset, they have index_train_0X.csv with X from 1 to 5. Can you please let me know which splits you used?
2. I cannot reproduce the result on ethanol using index_train_01.csv and index_test_01.csv. The paper reports energy MAE = 0.4 meV and force MAE = 2.4 meV/Å while I ran the above command but got (validation) energy MAE = 0.5 meV and force MAE = 2.3 meV/Å and (test) energy MAE = 0.8 meV and force MAE = 3.0 meV/Å. The differences are significant, so I surmise this can be due to the split being used.

Thanks!

[ilyes319]

Hi @yilunliao, can you share your log file with me, so I can have a look.

[yilunliao]

Sure. Here is the log.

I slightly modify the code and evaluate on the test set every 10 epochs to see how the errors decrease.

Thanks!

[Melody-1128]

Hi I'm wondering whether you finally figure out the problem? Because I'm reproducing the training and I found the MAE results are quite different from the paper.