Skip to content

feat: read fingerspelled LSE letters from the camera - #9

Merged
Endika merged 2 commits into
mainfrom
feat/camera-and-alphabet
Aug 5, 2026
Merged

feat: read fingerspelled LSE letters from the camera#9
Endika merged 2 commits into
mainfrom
feat/camera-and-alphabet

Conversation

@Endika

@Endika Endika commented Aug 5, 2026

Copy link
Copy Markdown
Owner

The camera now works and the app spells. First engine end to end: MediaPipe → segmenter → handshape table → stabiliser → transcript.

What landed

  • MediaPipeLandmarkSource — camera → 21 hand landmarks, WASM served same-origin. Releases the tracks on stop so the camera indicator actually goes off.
  • HandshapeAlphabetClassifier — scores the live handshape against a table of LSE fingerspelling shapes. No dataset, no download, useful on first launch.
  • RecognizeSignsUseCase — routes frame engines (a held letter) and window engines (a travelled sign) differently, each with its own stabiliser.
  • Container — constructor injection, no framework.

Two things worth a look

The fixture was lying. It modelled a curled finger as a straight finger pointing backwards, so the joint angle stayed at 180° and every handshape measured as fully straight — curl was 0.00 for every input. Probing before writing assertions is the only reason this was caught; tests written against it would have passed while measuring nothing.

Averaging alone picks confident wrong letters. d and l differ only in the thumb, so a completely wrong thumb cost one fifth of the score and l won on its three matching curled fingers. Any feature outside tolerance now vetoes the letter outright, so an ambiguous shape returns nothing. A missing letter is obvious to the signer; a wrong one is not.

Deliberately not recognised

g h j m n ñ p q r t x z — traced through the air, or dependent on palm orientation, which the invariant features discard on purpose. Listed in the UI so the gap is visible rather than mysterious. They are the trained model's job.

Camera untested under WSL2 (no device access); the pipeline's logic is covered by 43 tests against synthetic hands.

@Endika
Endika merged commit 0d07c31 into main Aug 5, 2026
6 checks passed
@Endika
Endika deleted the feat/camera-and-alphabet branch August 5, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant