Logging more metrics during evaluation

[csirois14]

Hello! I am using rl-zoo to train a manipulator to perform contact tasks. I want to compare the agents based on the total contact forces produced during the evaluation episodes. How would you log this metric?

The force values are computed by the environment and included in the info dictionnary at each step. My first idea was to modify ExperimentManager to use MyCustomEvalCallback instead of EvalCallback but then I believe I would also need to implement a custom evaluation_policy to replace the one from SB3 (stable_baselines3.common.evaluation.evaluate_policy).

Any other ideas on how to implement this?

PS: Thanks for the awesome library