Request for file keypoints_3d_mesh.pkl. [Spoken2Sign]

[rzhao-zhsq]

Hi @2000ZRL, this is the same issue mentioned in #85, but it has been closed.

I have checked the provided file 3D Dictionary in README and output of the fit_single_frame.py, including results and out_mesh :

SLRT/Spoken2Sign/fit_single_frame.py

Lines 527 to 530 in 38a4f7b

	results.append({'body_pose_rot': model_output.body_pose.detach().cpu().numpy(),
	'left_hand_pose_rot': model_output.left_hand_pose.detach().cpu().numpy(),
	'right_hand_pose_rot': model_output.right_hand_pose.detach().cpu().numpy()
	})

SLRT/Spoken2Sign/fit_single_frame.py

Lines 537 to 541 in 38a4f7b

	out_mesh = trimesh.Trimesh(vertices, body_model.faces, process=False)
	rot = trimesh.transformations.rotation_matrix(
	np.radians(180), [1, 0, 0])
	out_mesh.apply_transform(rot)
	# out_mesh.export(mesh_fn)

but find not one contains the key keypoints_3d in make_connector_dataset.py and in sign_connector_train.py:

SLRT/Spoken2Sign/make_connector_dataset.py

Line 44 in 38a4f7b

kp = kps[item1['video_file']]['keypoints_3d']

SLRT/Spoken2Sign/sign_connector_train.py

Lines 29 to 30 in 38a4f7b

	kps1 = torch.tensor(self.kps[clip1['video_file']]['keypoints_3d'][end][self.joint_idx]).float()
	kps2 = torch.tensor(self.kps[clip2['video_file']]['keypoints_3d'][start][self.joint_idx]).float()

I also checked the index-mapping in sign_connector_train.py but didn't really find anything useful.

SLRT/Spoken2Sign/sign_connector_train.py

Lines 60 to 62 in 38a4f7b

	joint_idx = np.array([3, 4, 6, 7, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
	37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
	53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66], dtype=np.int32)

I think the file keypoints_3d_mesh.pklis post-processed otherwise. Can you give a tip?

[2000ZRL]

The 3D keypoints will be obtained by forwarding the SMPL layer with the dictionary (which consists of mapping from video clip to SMPL-X params) as inputs.

[samloew8]

The 3D keypoints will be obtained by forwarding the SMPL layer with the dictionary (which consists of mapping from video clip to SMPL-X params) as inputs.

@2000ZRL many thanks for your feedback, could you please let us know which file in the repository helps us to generate the keypoints_3d_mesh.pkl ? Is it the file smplx2kps.py ?