Unable to reproduce the results in the paper

[ZuhongLIU]

Hello, I try to train the student policy using distill_mask_modes = {"omnih2o": DISTILL_MASK_MODES_ALL["omnih2o"]}. The teacher policy is trained for 120000 iterations and student policy is trained for 100000 iterations. Here are my results by following the training and eval scripts.

All: mpjpe_g: 148.483 mpjpe_l: 89.959 mpjpe_pa: 70.885 accel_dist: 3.113 vel_dist: 8.174 upper_body_joints_dist: 0.208 lower_body_joints_dist: 0.217 root_height_error: 0.068 root_vel_tracking_error: 0.300 root_r_error: 0.094 root_p_error: 0.100 root_y_error: 0.251 Succ: mpjpe_g: 139.726 mpjpe_l: 87.247 mpjpe_pa: 69.486 accel_dist: 2.685 vel_dist: 7.297 upper_body_joints_dist: 0.200 lower_body_joints_dist: 0.214 root_height_error: 0.065 root_vel_tracking_error: 0.257 root_r_error: 0.090 root_p_error: 0.096

It seems that the values of the metric are quite far away from those in the paper. Could your please give some suggestion to this problem? Thanks in adance.

[huihuaNvidia2023]

Can you please clarify the metrics more? Is this the teacher for the student? What's the success rate?

In terms of success rate
There is a slightly difference between this implementation and the paper. In the paper, the success criteria is {no illegal contact}. However, the implementation we have here is {no illegal concat, gravity vector<threshold, reference motion dist < threshold}, which is more strict. If you remove the criteria {reference motion dist < threshold}, you will see better success rate.

In terms of tracking metrics
Our suggestion is to get the best teacher policy. Internally, we did this by running eval with multiple checkpoints, and choose the best one. As you may noticed, the training is quite noisy.

As a reference, the best teacher metrics we got is
MPJPE-G: 100.3, MPJPE-L: 68.6, Success rate: 96.5%