How to train RankHardLoss on coco?

[sjy234sjy234]

Hi, there
I am trying to train RankHardLoss on coco dataset.
First, I train a lstmla with random sequence generated by 'get_rand_seq()'.
Then I lock the net and use 'RankHardLoss' provided to train on coco.
Following vse++'s 2-step training strategy, I failed to get better recall then hard negative triplet.
I also tried to unlock the lstmla after prior 2-step training is done, but only got slightly improvement, still less then triplet.

I guess the key point is how to use UpdatingWrapper, but updating every epoch only makes things worse.
Would you please elaborate on the details of training strategy on coco?