fix: measure sign windows in time, so recognition survives any frame rate - #53
Merged
Conversation
Endika
enabled auto-merge (rebase)
August 6, 2026 19:06
Endika
force-pushed
the
fix/segmenter-time-thresholds
branch
from
August 7, 2026 07:34
6972c47 to
3d1431b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Esku never wrote a vocabulary word in a browser while scoring 0.696 offline. Every
SignSegmenterthreshold was a frame count, swept against SWL-LSE — which is 20.00 fps inall 300 of its reference videos. The live pipeline runs three MediaPipe models per frame and
reaches whatever the device allows, so those counts silently meant a different duration on
every phone. Below the corpus rate a sign never reached
minSignFrames, every window wasdiscarded as too short, and the vocabulary engine was never asked at all.
Nothing offline could see it: the benchmarks replay landmarks that carry the corpus's own
frame rate baked in.
Evidence
Measured in a real browser — same build, same weights, only the input speed changed:
The arithmetic: the corpus median sign is 30 frames at 20 fps, about 1.5 s, so a 24-frame floor
is 1.2 s there — shorter than the sign, which is why it worked offline. At 10 fps the same
floor means 2.4 s of sustained motion, longer than the sign, so a window could only ever close
at the cap.
What changed
motionThreshold: 0.03per framemotionRate: 0.6palm widths per seconddecelerationHold: 1decelerationHoldMs: 50minSignFrames: 24minSignMs: 1150minFrames: 4minMs: 150andminFrames: 4maxFrames: 48maxMs: 2350minFramessurvives because they are two different concerns:minMsrejects what is too briefto be a sign,
minFramesrejects what has too few samples to describe one — sixteen featureslots resampled from two frames is a vector the model has never seen.
N frames span N-1 intervals, so the conversions are
(n-1)/20seconds, notn/20. Kept exacton purpose, so the offline benchmarks reproduce their pre-conversion numbers and any drift is a
porting mistake rather than a silent retune.
Verification
one passed and the other three failed — the bug as a test. 19/19 now.
tools/train/simulate_app.pyreproduces 0.741 / 0.696 / 0 dropped exactly, which is whatproves the Python port faithful rather than merely green.
sweep_continuous.py: the shipped row comes back identical(
0.45 / 50 ms / 1150 ms-> 0.696 isolated, 38.4% continuous, 75.5% precision) and is stillthe best trade. 850 ms buys 3.5 points of recall for 4.2 of precision and 8.4 of isolated —
the same trade already rejected once. No retune needed; the conversion preserves the optimum.
sweep.pyshowsmotionRateis flat from 0.4 to 1.6 palm widths/s on this corpus: theactivation floor does no work on dictionary recordings where the signer clearly moves. It is
left at the exact conversion rather than pretending the corpus validated a different value.
A test fixture bug surfaced on the way:
stillFramesin the use-case test stampedtimestampMs: 0on every held frame. Invisible while the segmenter counted frames; with timeit collapses the span to zero. Timestamps are continuous now.
Also in here: tools/browser
The harness that produced the evidence above, added as a tool rather than a vitest test — CI
has no Chrome, no camera and no corpus. It replaces the camera by playing a corpus clip into
getUserMediaviacaptureStream()(no fake-device flags, no y4m, no file serving) and readsthe diagnostics panel's DOM the way a person would.
It asserts only what holds on any device — the clip decoded, the weights loaded, a hand was
seen, the app mounted, and, only when the measured frame rate clears the shipped floor, that
the engine was asked. Frame rate, window counts, raw scores and the feature profile are
measured and printed, not asserted.
Recognition is deliberately not asserted: it needs a frame rate software-WebGL headless does
not reach (~1.3 fps here against a 3.5 fps floor), and a permanently red check invites someone
to lower a shipped threshold to make it green. The floor is read out of
SignSegmenter.tsbythe harness rather than copied, and throws if the constants are renamed — duplicating them
would let the tool disagree with the app it measures, which is the bug class it exists to catch.
Verified it can fail, not just pass: a non-video clip and a server whose SPA fallback answers
200 without a bundle both exit 1 with specific diagnoses.
Not verified
Real-device confirmation. This box maxes at ~1.3 fps headless, below the floor the fix implies,
so recognition end-to-end still needs a phone. The panel now answers it directly: "Fotogramas"
after ~15 s gives the device's frame rate, and "Ventanas cerradas (N descartadas por cortas)"
says whether it clears 3.5 fps.
Note the Pages deploy may fail on merge while GitHub Actions and Pages are in a major outage;
a timed-out deploy burns the merge SHA and recovery is another commit.