Training a reduced model by mixing DFT data with a subset of foundational training data

[dominicvarghese]

Hi everyone,

I have a question regarding model training strategy using the fine_tuning_select.py CLI tool.
Instead of fine-tuning the full foundational model, I was wondering whether it is practical to train a new model from scratch by combining:

My own system-specific DFT data, and a fraction of the training data originally used for the foundational model (selected via fine_tuning_select.py).

The main motivation is to reduce the final model size while still benefiting from the diversity and stability provided by part of the foundational dataset. Ideally, I would like the resulting model to behave similarly to a fine-tuned foundational model, but with fewer parameters and lower computational cost.

In addition, I’m concerned about consistency between datasets. My DFT data were generated using a different pseudopotential and DFT setup than the one used for the foundational model.

Specifically:

How should one account for differences in pseudopotential choice (and possibly XC functional, cutoff, etc.) between the sampled foundational data and the user-generated DFT data?

Also would this give a smaller model compared to what is done with Multihead replay finetuning (since we use a smaller dataset)?

Thanks
Dominic

[ilyes319]

@dominicvarghese Hello, this is possible, though I would not recommend it. For consistency, you should be using the multihead training, and make sure that you have E0s also for your model. https://mace-docs.readthedocs.io/en/latest/guide/heterogeneous_data.html

What i would recommend instead is that you fine-tune the small omat model: https://github.com/ACEsuit/mace-foundations/releases/tag/mace_omat_0. It is already small, and I think it will work quite well.