DGL and Keras MEGNet performance discrepency

[Luo-Yiqun]

Hi all,

I am trying to reproduce the MEGNet results. I have done hyperparameter tuning on QM9 and MP. Following are my best results (the hyperparameter tuning is not thorough):

Task	setting	Test MAE	Test MAE in the paper
MP, formation energy	20k data, 1 block, 15 epochs	0.16 eV/atom	0.028 eV/atom
MP, gap	20k data, 3 blocks, 15 epochs	0.634 eV	0.33 eV
QM9-simple, gap	40k data, 2 blocks, 10 epochs	0.018 eV	0.066 eV
QM9-full, gap	40k data, 6 blocks, 10 epochs	0.017 eV	0.061 eV

For the MP task, if looking at the convergence test in the SI, with the dataset multiplied by 3, from 20k to 60k, MAE halved. It makes my MP gap comparable with the paper, but MP formation energy larger than the paper.

However, what really confused me is that for the QM9 task, I only used a fraction of the dataset and beat the paper. Has anyone observed this similar behavior as me? Since the original QM9 was implemented using Keras, is there any benchmark on comparing the original MEGNet Keras-based implementation with this DGL-based implementation? Was MEGNet optimized in the DGL version?