Need some suggestion on reusing LandmarkResultSet and other components in my hackathon project

[henryouly]

TL;DR I created a hackathon project based on this project and https://github.com/homuler/MediaPipeUnityPlugin. The idea is to run the mediapipe graph directly in Unity, and use AnimationRigging as shown in this project to control the animation.

Here is how I call into the LandmarkResultSet. The (x,y) coordinate from NormalizedLandmarkList looks expected (x coordinate changes for left/right, y coordinate for up/down)

private void OnHandLandmarksOutput(List<NormalizedLandmarkList> handLandmarks)
    {
        if (handLandmarks != null) {
            // Logger.LogInfo(_TAG, handLandmarks[0]);
            resultSet.UpdateLandmark(LandmarkInterface.LandmarkType.LeftHand, handLandmarks[0]);  // only use lefthand for testing for now
        }
    }

I also simplified LandmarkOrientation, rather than taking the orientation from LandmarkResultSet which was set from the socket, I config it using a Vector3 field in Inspector, for my testing purpose.

        private void updateOrientation(Vector3 orientation)   // orientation is from the Inspector
        {
            this.transform.localEulerAngles = orientation;
        }

I copied the placement of the UnityChan GameObject from the sample scene - except one change - my main camera has Vector3.zero rotation, and my unitychan's rotation is (0, 180, 0). The sample scene has unitychan at Vector3.zero rotation, and main camera reverted. Everything in the unitychan hierarchy is cloned, including HandLandmarkAndRigs sub gameobject attached to the main character. I'm not sure if this is relevant but would like to point out.

Playing in the editor is as followed gif: The x,y coordinates look swapped unexpectedly, e.g. when I was actually moving my arm to the left (webcam), the UnityChan is moving it downwards, like every coordinate in the hand landmark is rotated counter clockwise along the z-axis. However I am not able to figure out exactly what I missed. Is there any logic that assumed the coordination system on a phone so I should have it corrected when debugging using a webcam in Unity Editor? Or any other ideas how I can further debug the coordinate system? Thank you.

[henryouly]

I'm playing with the LandmarkOrientation object.

It seems I have to set rotation to (0, 0, 90) for the right arm to be shown at the right position, but it appears the left palm will then be rotated incorrectly. Tweaking any number in 90 degree will twist the rig badly. I can't reason about how the rotation is applied to the model's world coordinates.

[TesseraktZero]

I think this was something with my assumptions on the phone orientation when I was implementing the stuff.
You may try to rotate the whole LandmarkOrientation object to fit it. Seems a rotation of (90,180,0) on the 'LandmarkOrientation' is a good number for some situations.
If you feel that the landmarks are flipped, you may also go to 'LandmarkOrientation\LeftHand' and play around with the toggles to flip the landmarks (Flip XY/Negate X/Negate Y).

[henryouly]

@TesseraktZero Thank you for your help. I ended up making it working, although I'm sure there is a better way.

For posterity, what I ended up doing is to apply a -90 Z Rotation to HandLandmarkSet (which is the parent game object of LandmarkOrientation), to invert the effect of LandmarkOrientation rotated at (0,0,90). I also need to recalibrate HandLandmarkSet to place the IK targets (e.g. the jointpoints) right to the front. The initial position (e.g. before hitting Play) will still be incorrect, but after hitting Play, the IK target should be updated to the right world placement.

I think to truly fix it I may need to recreate individual jointpoints in both LeftHand and RightHand to face the right orientation.