Adaptive loss weight

[chang-change]

I want to use the adaptive loss weight during the neural network training according to the method showed by the following picture.

My code is following:

model = dde.Model(data, net)
model.compile("adam", lr=learning_rate, metrics=["l2 relative error"])
losshistory, train_state =model.train(iterations=1000)

num_train = 0
while num_train < 5:
    Lj = losshistory.loss_train[-1]
    Lj_1 = losshistory.loss_train[-2] - 0.0000001
    L_rate = Lj /Lj_1
    miu = np.max(L_rate)
    L_rate_miu = L_rate - miu 
    numerator = np.exp(L_rate_miu)
    denominator = np.sum(numerator)
    loss_weight = numerator/denominator
    model.compile("adam", lr=learning_rate,loss_weights=loss_weight,metrics=["l2 relative error"])
    losshistory, train_state =model.train(iterations=10000)
    num_train += 1

However, the neural network need to recompile at each update loss weight. It spends a lot of time. Is there any way to do this with just one compile? Thanks.

[praksharma]

You need to modify the source code which is very easy thanks to @lululxvi 's coding style. Download this project. This is the function which you are going to start with.

Please check this if-else condition for you backend. Here I assume you are using PyTorch.

So, you need to add your code here for a PyTorch backend.

I am not super genius in coding. Somebody may tell you a better way to do this.

[chang-change]

@praksharma Sorry for the late reply! I use Tensorflow as my backend, whether I need to add my code here.
If adaptive loss weight is the following format，
$$weight = \frac{max(loss_i)}{loss_i} $$
whether the following code is right?

    if loss_weights is not None:
        max_loss = tf.ruduce_max(losses)
        loss_weights = max_loss/losses
        
        losses *= loss_weights
    return outputs_, losses

And I have some question, the loss is the current train_step loss, how can I get the loss before the current train_step? As the following picture, the loss weight is related to the loss term at iterationiandi-1.

[JinglaiZheng]

I'm sorry to bother you, but have you solved the problem? I'm struggling with implementing adaptive weights,thanks

[lululxvi]

If you don't want to re-compile, you can implement a callback to update the weights. The implementation is backend dependent.

[chang-change]

@lululxvi Thanks for your reply. How should I implement the callback, can you provide an example?

[lululxvi]

I don't have an example now. It is a little tricky to implement in TensorFlow. An easy solution is recompile every N steps like every 10000 steps. I guess this won't hurt the performance too much.