optimizer and crossimage_encoder update problem

[Dog-Yang]

Hello, thank you very much for your outstanding work. I have a few questions that I hope you could help clarify:

1. In lines 63-66 of train.py, when setting up the optimizer, you use Adam as follows:
optimizer = torch.optim.Adam(model.parameters(), lr=args.lr)
This actually selects all parameters of the model to update, including the last four transformer encoder layers of the DINOv2 backbone, SuperVLAD (Conv2d), and the crossimage_encoder components. Then you also use:
optim_encoder = torch.optim.Adam(model.module.encoder.parameters(), lr=args.lr_encoder)
to specifically select the parameters of the crossimage_encoder. Wouldn't that mean that the crossimage_encoder is updated twice?

2. In lines 203-208 of train.py, the code is written as:

if args.crossimage_encoder:
    for p in model.module.encoder.parameters():
        p.requires_grad = True
    optim_encoder.step()
    for p in model.module.encoder.parameters():
        p.requires_grad = False

After the optim_encoder step, the crossimage_encoder is set to requires_grad = False. Even if requires_grad is later set to True again, since the loss.backward() call (at line 202) happens before this, at that moment p.requires_grad is still False. Therefore, isn’t the crossimage_encoder updated only once during the entire process?

I appreciate your time and look forward to your explanation.

[Lu-Feng]

Yes, the crossimage_encoder is updated only once. As in lines 203-208, we update this module separately, then freeze it, and update the rest with optimizer.step()