fix: stop drawing pose landmarks the model never saw - #27
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From a screenshot of the app running against a signer framed from the chest up: the torso box ran off the bottom of the picture onto the desk, the arms pointed at the frame edges instead of following the arms, and the neck was a wide triangle across the chest.
Root cause
MediaPipe extrapolates pose landmarks it cannot see rather than omitting them. Film someone from the chest up and it still returns hip coordinates — invented, somewhere below the picture. It marks them with a low
visibility, which the overlay was ignoring entirely.Now every pose edge requires both endpoints above
MIN_POSE_VISIBILITY. The shoulder line stays, the hip box disappears when the hips are out of shot, and arms are drawn only where they were actually found.The neck was my shortcut, not the model's fault
I had drawn it as nose→left-shoulder plus nose→right-shoulder, because MediaPipe has no neck landmark. That paints a triangle over the chest — over exactly the signing space the user needs to see. It is now one segment from the shoulder midpoint up to roughly the chin, stopping short of the nose so it does not run into the face mesh.
Chips now mean what they say
They reported
pose.length > 0— "the pose model returned an array" — so torso and arms went green even when nothing about them was visible. Each chip is now driven by the visibility of the landmarks that part is actually made of, so red means not seen.Nine tests, including that absent visibility is trusted rather than treated as invisible: hands and face do not report it, and getting that backwards would erase them from the overlay.