Skip to content

feat: show face, neck, torso and arms tracking on camera - #21

Merged
Endika merged 1 commit into
mainfrom
feat/body-overlay
Aug 5, 2026
Merged

feat: show face, neck, torso and arms tracking on camera#21
Endika merged 1 commit into
mainfrom
feat/body-overlay

Conversation

@Endika

@Endika Endika commented Aug 5, 2026

Copy link
Copy Markdown
Owner

The overlay drew hands only, so "it is not working" and "it cannot see your face" looked identical. Now every tracked part is drawn in its own colour, with a chip per part that goes green when it is being tracked and red when it is not.

Part Colour Source
Manos violet HandLandmarker
Cara cyan FaceLandmarker
Cuello pink Pose, nose → shoulders
Torso amber Pose, shoulder line and hip box
Brazos green Pose, shoulder → elbow → wrist

Separate colours because the parts fail independently: hands track fine at arm's length while the face is out of frame, or the torso is cropped while everything else is good. One uniform skeleton would hide which.

The face is drawn as points rather than mesh edges — 468 edges are unreadable at video size, and a scatter already shows it is tracked. Legs are not tracked at all: nothing below the hips carries meaning in LSE.

Why pose is here at all

Measured, not assumed. On SWL-LSE's held-out test split, normalising hand position against the torso instead of the image frame is worth +5.7 points of top-1 (0.632 → 0.689). "Hand at chin height" is a fixed number in body coordinates and a moving one in image coordinates — it changes when the signer steps closer.

Pose and face load with .catch(() => null) and the app keeps working without them: hands are the only model recognition genuinely needs, so the other two are enhancements, not dependencies. source.available reports which came up.

Engine grows from ~32 MB to ~41 MB; the storage panel says so and can still free it.

@Endika
Endika merged commit a700811 into main Aug 5, 2026
7 checks passed
@Endika
Endika deleted the feat/body-overlay branch August 5, 2026 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant