Resuming Training

[snrazavi]

Hello,
I am trying to train a seq-att model for translation, but always after 3 or 4 epochs, the training process stops unexpectedly (Kill). Moreover, when I try to resume the training using --model_in option I recieve out of memory error and this is regardless of how much GPU memory I use using --dynet-mem. I have a GTX 980 GPU with 8 Giga bytes of graphics RAM. Also I should add that the memory problem is more critical and severe when I try to use other optimization methods such as adam.
Many thanks in advance.

[neubig]

This is two problems I think.

1. The training process is stopping unexpectedly. I'll need more details about the exact error message you get then to try to debug this.
2. When re-loading models you're running out of memory. Unfortunately this is a known bug in DyNet: reallocate memory when dynet_load_model cause double memory consumption. clab/dynet#110 and will probably not be fixed until this bug is fixed.