Hidden State Warping - GT vs Prediction

[mohammed-amr]

Hello,

I'm looking at your description for how to train the fusion model in the supplemental:

Finally, we load the best checkpoint and finetune only the cell for another 25K iterations with a learning rate of 5e−5 while warping the hidden states with the predicted depth maps.

The current training script at fusionnet/run-training.py doesn't have a flag for this. I can see that the GT depth is used for warping the current state at line 249.

What should I use as a depth estimator for this step? Should I borrow from this line at fusionnet/run-testing.py? Or (more likely) this differentiable estimator at line 157 in utils.py?

Thanks.

[ardaduz]

Sorry for not having that part in the repository. Both should work and give similar results since training is done with square shaped images and gradient flow disabled. As far as I can remember, non-differentiable function was having issues with the GPU memory space on GTX 1080Ti for some reason during training (which is already quite maxed out with the current batch and sub-sequence sizes). Therefore,

1. You can borrow the lines 87, 88, [179, 191], 201, 202 from fusionnet/run-testing.py.
2. Replace the non-differentiable function with the differentiable function just because of potential memory issues. Don't forget to .detach() the prediction tensor while assigning it to the previous_depth variable to disable the gradient flow.
3. Set the image size function parameters correctly to Config.training_image_width and Config.training_image_height
4. Finetune only the LSTMFusion module

I can not test this right now, so please let me know how it goes or if you need more info.

[mohammed-amr]

Thanks!

I've gotten it working given the changes you've suggested and the available code snippets. I've put my version of the file here.

It's training now. I'll let you decide if you want to close this issue, or keep it to signpost for others.

Thanks again for your quick reply and help.