Why optimize vo or imu model every data sequence not every epoch?

[ChillTerry]

Hi,

Great work on the project! After reviewing your entire codebase, I have two questions:

Optimization Frequency: Why do you optimize the VO or IMU model after each data sequence rather than at the end of every epoch? I noticed that during each epoch you only perform backpropagation and defer calling optimizer.step() until after the entire dataset has been processed. Have you tried updating the model weights at the end of each epoch?

Alternating Optimization: Why do you optimize the VO and IMU models alternately instead of updating them simultaneously? Have you experimented with optimizing both models together?

Thanks for your time and insights!

[RomanStadlhuber]

Since noone has anserwed this I am going to give my two cents:

The first paper that comes to mind (though there are probably countless ones with the same insight) is Adaptive Robust Kernels
for Non-Linear Least Squares Problems where the authors decide to split the main optimization in two separate steps, first optimizing for the parameter of the adaptive loss (kernel) function and then for the state estimate itself.

This is beneficial to the estimation (learning) process as you would otherwise have one estimator (network) try to "explain away" the errors of the other and vice versa, but we don't want the modules to need to care about the others but rather have each one perform to the best of their abilities.

That way, if the loss is affected badly by incorrect weights of one network, only that one is forced to improve in that epoch and vice versa.