Slow rendering or excessive calls to refresh the sensors?

[ankile]

Why is it that (if we don't care about the backwards compatibility of the physics), we can't change out these lines in the MujocoEnv step function:

        for i in range(int(self.control_timestep / self.model_timestep)):
            if self.lite_physics:
                self.sim.step1()
            else:
                self.sim.forward()
            self._pre_action(action, policy_step)
            if self.lite_physics:
                self.sim.step2()
            else:
                self.sim.step()
            self._update_observables()
            policy_step = False

with this?

        n_substeps = int(self.control_timestep / self.model_timestep)
        self._pre_action(action, policy_step)
        mujoco.mj_step(self.sim.model._model, self.sim.data._data, nstep=n_substeps)
        self._update_observables()

I.e., it shouldn't be necessary to call the camera rendering every single physics step (i.e., 500 / 20 = 25 times for every image we actually use in the policy)?

When I'm running training with these envs, I'm getting around 20 fps with 3 cameras at 84 by 84 pixels, which seems very slow, especially compared with the 3500 fps mentioned in the docs.

Any hints on why the rendering is so slow will be super appreciated -- thank you!

Best, Lars

EDIT: I now realize the there's a lot of internal bookkeeping inside the observables that makes sure that the rendering is in fact only happening at the correct 20 Hz rate. However, when I do implement the change above, the resulting fps is somewhere between 50% to 100% higher, so there definitely is some performance improvements to be had. However, despite changing the timestep increment in the observables to 1/20, the above change breaks the rollouts.

[ankile]

Hey, I just want to add that my understanding is a bit refined since first posting this, see the edit in the original comment. It seems that the 20 fps might be the inherent limitation of the Mujoco renderer? If so, then I really don't understand where the 3500 fps number could come from?

[kevin-thankyou-lin]

Hey @ankile, thanks for your question - the 3500 fps is from running purely rendering. Something like this:

n_steps = 10000
start = time.time()
for _ in range(n_steps):
    env.sim.render()
end = time.time()

The smaller envs have higher FPS; I'm assuming you're seeing 20Hz on the robocasa scenes? What do you mean by "changing the timestep increment in the observables to 1/20"? and "change breaks the rollouts" means replay fails?