Saving model weights before nan

[marcofrancis]

Hi,
I'm training my model (pytorch backend) usong adam for 10K steps and then BFGS.
BFGS stops because it hits some nans. How can I recover the model state before it hit the nans?
Is there a way to save the last state before nans?

[lululxvi]

Use dde.callbacks.ModelCheckpoint https://deepxde.readthedocs.io/en/latest/modules/deepxde.html?highlight=modelcheck#deepxde.callbacks.ModelCheckpoint