Loss implementation function

[Vicktoria13]

Hello,

Thanks for the open-source code. I was trying the use the training code for my own dataset, and thus adapting both dataloader and RobustLoss class. For each pair of image, I have an homography matrix H between A and B, from which i can deduce dense wrap (approximative). However, I have some trouble understanding your loss implementation. To be more precise, here is your forward loss :

def forward(self, corresps, batch):
        scales = list(corresps.keys())
        tot_loss = 0.0
        # scale_weights due to differences in scale for regression gradients and classification gradients
        scale_weights = {1:1, 2:1, 4:1, 8:1, 16:1}
        for scale in scales:
            scale_corresps = corresps[scale]
            scale_certainty, flow_pre_delta, delta_cls, offset_scale, scale_gm_cls, scale_gm_certainty, flow, scale_gm_flow = (
                scale_corresps["certainty"],
                scale_corresps.get("flow_pre_delta"),
                scale_corresps.get("delta_cls"),
                scale_corresps.get("offset_scale"),
                scale_corresps.get("gm_cls"),
                scale_corresps.get("gm_certainty"),
                scale_corresps["flow"],
                scale_corresps.get("gm_flow"),

            )
            if flow_pre_delta is not None:
                flow_pre_delta = rearrange(flow_pre_delta, "b d h w -> b h w d")
                b, h, w, d = flow_pre_delta.shape
            else:
                # _ = 1
                b, _, h, w = scale_certainty.shape
            gt_warp, gt_prob = get_gt_warp(                
            batch["im_A_depth"],
            batch["im_B_depth"],
            batch["T_1to2"],
            batch["K1"],
            batch["K2"],
            H=h,
            W=w,
        )
            x2 = gt_warp.float()
            prob = gt_prob

etc ...

I was thinking about replacing the following lines :

gt_warp, gt_prob = get_gt_warp(                
            batch["im_A_depth"],
            batch["im_B_depth"],
            batch["T_1to2"],
            batch["K1"],
            batch["K2"],
            H=h,
            W=w,
        )
            x2 = gt_warp.float()
            prob = gt_prob

by my own function whose prototype should be smthg like

def compute_dense_matching_from_H(self, H, imA,imgB) ==> x2,prob with x2 representing my matches scaled between -1/1 and prob a torch.ones tensor in my case by default.

However, I am not sure about how to implement it. here is my question:

1. My Homography matrix is the matrix to go from im_A to im_B present in the batch. However you introduced a scale with h and w in a for loop : what does it correspond to concretly ? Whose is being rescaled exactly ? and should I adapt my matrix according to them ?

Thanks for your reply !

[Parskatt]

h and w are used here to get the coordinates of the warp, basically we have full resolution depthmaps but predictions in some different resolutions, so we need to interpolate the GT warp in some way.

[Vicktoria13]

Thanks for the reply.

Do you confirm that basically, corresps contains matches between im_A and im_B in batch after each image has been resized to h and w ?

[Parskatt]

Thats correct.

[DruncBread]

Hi, @Viciktoria13
Is your dataset of the type where there are ground-truth matching relationships between image pairs (im_A and im_B) but lacking depth information (e.g., im_A is generated by affine transformation from im_B)?
How is the training effect on your custom dataset? Would you mind sharing your demo for training on your custom datasets?
Thanks!

[Vicktoria13]

Hi

Yes i only have a homography information between the 2 images, from which i can infer pseudo dense matches. So far results are not good but maybe because my dataset is too small or something is wrong in my implémentation.

[Vicktoria13]

Hi @Parskatt ,

I continued on the implementation of my own losse function, and I have a doubt about the range of the gt supervision : trying to replace

gt_warp, gt_prob = get_gt_warp(                
        batch["im_A_depth"],
        batch["im_B_depth"],
        batch["T_1to2"],
        batch["K1"],
        batch["K2"],
        H=h,
        W=w,)
        
 x2 = gt_warp.float()
 prob = gt_prob

with my own function :

x2, prob = self.warp_kpts_homography_custom(batch, h, w, batch["im_A"].shape[2], batch["im_A"].shape[3]) # [b,h*w,2], [b,h*w]

When reading the documentation, it seems like the function get_gt_warp takes as input kpt0 in the range -1 / 1. Is it the case for its output, ie is gt_warp is the range [-1 1] too ?
EDIT : I just saw this line
w_kpts0 = torch.stack( (2 * w_kpts0[..., 0] / w - 1, 2 * w_kpts0[..., 1] / h - 1), dim=-1 ) # from [0.5,h-0.5] -> [-1+1/h, 1-1/h] which confirm that the kps output should be in [-1+1/h, 1-1/h] range. But the h parameter is depending on the scale right ? so it means the range is dynamic what is the meaning behind it?

Also In my case, I put prob as as tensor full of one, simulating a full confidence in each of my match, does it sound consistent ?

Also, I try to finetune RomA on a much smaller dataset, do you have any recommendations on part of the model I should freeze to avoid catastrophic forgetting or overfit ?

Thanks !!!