Restoring model and training with new data

[frankiezoo]

I'm attempting to use deepXDE for an active learning application where I seed the model with a training dataset, then iteratively add new data to the training dataset and train the model from where it left off. Here's my code:

seed_data = dde.data.DataSet(
    X_train,
    y_train.reshape((len(y_train),1)),
    X_test,
    y_test.reshape((len(y_test),1))
)

layer_size = [2] + [50] * 3 + [1]
activation = "sigmoid"
initializer = "Glorot uniform"
regularization = ["l2", 1e-5]
dropout_rate = 0.01
net = dde.nn.FNN(
    layer_size,
    activation,
    initializer,
    regularization=regularization,
    dropout_rate=dropout_rate,
)

BNN_model = dde.Model(seed_data, net)
BNN_uncertainty = dde.callbacks.DropoutUncertainty(period=1000)
BNN_model.compile("adam", lr=0.001, metrics=["l2 relative error"])
checkpointer = dde.callbacks.ModelCheckpoint(
    filepath = "model/model.ckpt", 
    verbose = 1, 
    save_better_only = True, 
    period = 1000
)

losshistory, train_state = BNN_model.train(iterations=10000, callbacks=[BNN_uncertainty, checkpointer])

for i_sample in range(n_start+1,n_samples):
    new_data = dde.data.DataSet(
        X_train,
        y_train.reshape((len(y_train),1)),
        X_test,
        y_test.reshape((len(y_test),1))
    )

    BNN_model = dde.Model(new_data, net)
    BNN_model.compile("adam", lr=0.001, metrics=["l2 relative error"])
    BNN_model.restore("model/model.ckpt-" + str(train_state.best_step) +".ckpt", verbose=1)
    losshistory, train_state = BNN_model.train(iterations=5000, callbacks = [BNN_uncertainty, checkpointer])
    

I get the error:

Compiling model...
'compile' took 1.311290 s

Initializing variables...
Restoring model from model/model.ckpt-5000.ckpt ...

INFO:tensorflow:Restoring parameters from model/model.ckpt-5000.ckpt
Traceback (most recent call last):

File "C:\Users\frank\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1378, in _do_call
return fn(*args)

File "C:\Users\frank\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1361, in _run_fn
return self._call_tf_sessionrun(options, feed_dict, fetch_list,

File "C:\Users\frank\anaconda3\lib\site-packages\tensorflow\python\client\session.py", line 1454, in _call_tf_sessionrun
return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,

NotFoundError: Key beta1_power_18 not found in checkpoint
[[{{node save_20/RestoreV2}}]]

Reading previous threads #228 #57, it seems that the model that was trained and the model that is being restored are different, but I haven't modified them. I call the restored model pretty soon after I store the model, so I do not know what could have changed other than the new data. I also tried dde.utils.apply to add new data but that didn't work for me. Do you have example code of iteratively adding new data to a restored model to continue training?

edit: my dde.utils.apply code:

for i_sample in range(n_start+1,n_samples):
    # BNN
    X_train = X_train_full[0:int(i_sample),:]
    y_train = y_train_full[0:int(i_sample)]

    new_data = dde.data.DataSet(
        X_train,
        y_train.reshape((len(y_train),1)),
        X_test,
        y_test.reshape((len(y_test),1))
    )
    res = dde.utils.apply(BNN_model, (new_data,))

with error:

Traceback (most recent call last):

File "C:\Users\frank\AppData\Local\Temp/ipykernel_7380/4235859398.py", line 1, in
res = dde.utils.apply(BNN_model, (new_data,))

File "C:\Users\frank\anaconda3\lib\site-packages\deepxde\utils\external.py", line 77, in apply
r = p.apply(func, args=args)

File "C:\Users\frank\anaconda3\lib\multiprocessing\pool.py", line 357, in apply
return self.apply_async(func, args, kwds).get()

File "C:\Users\frank\anaconda3\lib\multiprocessing\pool.py", line 771, in get
raise self._value

File "C:\Users\frank\anaconda3\lib\multiprocessing\pool.py", line 537, in _handle_tasks
put(task)

File "C:\Users\frank\anaconda3\lib\multiprocessing\connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))

File "C:\Users\frank\anaconda3\lib\multiprocessing\reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)

TypeError: cannot pickle '_thread.RLock' object

[lululxvi]

See gPINN enhanced by RAR examples at https://github.com/lu-group/gpinn

[lululxvi]

Then you can read the source code of dde.data.DataSet and directly update its attributes.

[frankiezoo]

Thanks! I tried your suggestion and ran a test to see if directly attributing the new data into the existing model would incorporate the new data into training and it did not!
Here's my code for the iterative loop:

## define new training set
X_train = X_train_full[0:int(i_train+1),:]
y_train = y_train_full[0:int(i_train+1)]
## directly update BNN_model training data attribute
BNN_model.data.train_x = X_train
BNN_model.data.train_y = y_train
## print statements to check if attribution took prior to training
print('direct attribute definition')
print(BNN_model.data.train_x)
## compile and train
BNN_model.compile("adam", lr=0.001, metrics=["l2 relative error"])
losshistory, train_state = BNN_model.train(iterations=5000, callbacks = [BNN_uncertainty, checkpointer])
## check that post training, for whatever reason, the attribution held
print('post training recall')
print(train_state.X_train)

Funny thing, the training data was properly updated before and after training but the model did not get more accurate with the incorporation of more samples as I would expect. The model was only as accurate as the initial model training. Is there a way I can check which points the model is using during training? Can you see anything glaringly wrong with my implementation or is there a more fundamental problem in the backend?
Thanks, F

[lululxvi]

The code looks OK.

[frankiezoo]

And just to double check, the most up to date, lowest training error test predictions are in train_state.best_y? Because that's where I'm extracting my test predictions and what I'm observing to not improve

[lululxvi]

Yes.

[frankiezoo]

An explanation for why I wasn't seeing the test predictions update with new training data:
While incrementally adding training data and then training the BNN model, the best solution was stuck on a very early solution using very little training data. A trivial solution of getting low error because there was little data to represent. Instead, to see the updated model trained to new data, I called
y_pred = BNN_model.predict(X_test)
in every loop to see that the trained model definitely represented the new training data.

[gabrielord]

Hii Frankie!
Sorry for the question, but would you mind sharing your code? I am trying to solve a similar problem, but been having some problems.