Understanding the State Estimator and State Predictor

[shreshtashetty]

Hi,

I’ve been reviewing the state estimation and state prediction modules in the code, and I want to clarify my understanding of how they work. From what I gather:

• The state estimation module updates the history and samples a new state based on the current inputs (i.e., using mu_t and sigma_t), effectively creating a semantic reconstruction of the scene.
• The state prediction module takes this estimated current state and predicts future states, along with semantic reconstructions of its predictions.

Please correct me if I’m mistaken.

I have the following questions:

• The inference_prediction function evaluates the results of the imagine_step function, which estimates the history and state, and stores them in the prior_output variable. imagine_step is also used elsewhere (in the observe_step function) to get priors, which are then used to calculate the posterior. If the function (inference_prediction) is evaluating future state predictions, why is it evaluating a "prior"? Intuitively, shouldn't state estimation give priors and state prediction give posteriors?

• Assuming imagine_step handles state prediction, why is it also used during the observation step? Shouldn’t prediction be a separate step, performed after the formation of the latent state space during the observation step?

• Building on the previous two points, can you clarify what the observation step and the prediction step entail? How are they ordered and integrated into the overall model architecture during the forward pass and inference?