fix: recognise signing that never pauses - #25
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported from testing against a YouTube video of someone signing: the app detected nothing at all.
The bug
SignSegmenterclosed a window only when the hand held still for six consecutive frames. That is how isolated dictionary signs are delimited, and it is how SWL-LSE was recorded — but a fluent signer never holds still.Measured on 300 frames of continuous motion:
Zero. The vocabulary engine was never invoked once, so the app could not have recognised anything no matter how good the model was. It looked broken rather than inaccurate, which is the worse failure of the two.
The fix is a hard cap: if the window reaches
maxFrameswhile motion continues, emit it anyway. Stillness is still preferred when it happens, so isolated signs are not chopped mid-sign — there is a test for that.Found a second thing while probing
My first probe reported 0 windows after the fix too. The synthetic motion I wrote came out at 0.077 palm-widths per frame against a 0.08 threshold — it was never activating the segmenter, so I was measuring nothing. Worth recording because it applies to real use as well: gentle or small-amplitude signing sits under the activation threshold and will not trigger the vocabulary engine. That is tuning, and tuning needs real footage rather than synthetic hands, so it is not changed here.
What this does not fix
Being honest about the limits, because this change will not make that video work:
MIN_CONFIDENCEcorrectly stays silent rather than guessing.So: the app will now try on continuous signing, where before it structurally could not. It will not read a fluent conversation.