Skip to content

fix: measure sign windows in time, so recognition survives any frame rate - #53

Merged
Endika merged 2 commits into
mainfrom
fix/segmenter-time-thresholds
Aug 7, 2026
Merged

fix: measure sign windows in time, so recognition survives any frame rate#53
Endika merged 2 commits into
mainfrom
fix/segmenter-time-thresholds

Conversation

@Endika

@Endika Endika commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Esku never wrote a vocabulary word in a browser while scoring 0.696 offline. Every
SignSegmenter threshold was a frame count, swept against SWL-LSE — which is 20.00 fps in
all 300 of its reference videos. The live pipeline runs three MediaPipe models per frame and
reaches whatever the device allows, so those counts silently meant a different duration on
every phone. Below the corpus rate a sign never reached minSignFrames, every window was
discarded as too short, and the vocabulary engine was never asked at all.

Nothing offline could see it: the benchmarks replay landmarks that carry the corpus's own
frame rate baked in.

Evidence

Measured in a real browser — same build, same weights, only the input speed changed:

frames seen windows closed engine asked transcript
clip at 1x (~1.4 fps pipeline) 23 0, 6 discarded short 0 ""
same clip at 0.1x 45 1, 0 discarded 1 "Dolor" 68.0%

The arithmetic: the corpus median sign is 30 frames at 20 fps, about 1.5 s, so a 24-frame floor
is 1.2 s there — shorter than the sign, which is why it worked offline. At 10 fps the same
floor means 2.4 s of sustained motion, longer than the sign, so a window could only ever close
at the cap.

What changed

before (frames) after (time)
motionThreshold: 0.03 per frame motionRate: 0.6 palm widths per second
decelerationHold: 1 decelerationHoldMs: 50
minSignFrames: 24 minSignMs: 1150
minFrames: 4 minMs: 150 and minFrames: 4
maxFrames: 48 maxMs: 2350

minFrames survives because they are two different concerns: minMs rejects what is too brief
to be a sign, minFrames rejects what has too few samples to describe one — sixteen feature
slots resampled from two frames is a vector the model has never seen.

N frames span N-1 intervals, so the conversions are (n-1)/20 seconds, not n/20. Kept exact
on purpose, so the offline benchmarks reproduce their pre-conversion numbers and any drift is a
porting mistake rather than a silent retune.

Verification

  • Four new tests drive the same physical sign at 8, 20 and 60 fps. Before the change the 20 fps
    one passed and the other three failed — the bug as a test. 19/19 now.
  • Full suite 139/139, typecheck and biome clean.
  • tools/train/simulate_app.py reproduces 0.741 / 0.696 / 0 dropped exactly, which is what
    proves the Python port faithful rather than merely green.
  • Re-swept sweep_continuous.py: the shipped row comes back identical
    (0.45 / 50 ms / 1150 ms -> 0.696 isolated, 38.4% continuous, 75.5% precision) and is still
    the best trade. 850 ms buys 3.5 points of recall for 4.2 of precision and 8.4 of isolated —
    the same trade already rejected once. No retune needed; the conversion preserves the optimum.
  • sweep.py shows motionRate is flat from 0.4 to 1.6 palm widths/s on this corpus: the
    activation floor does no work on dictionary recordings where the signer clearly moves. It is
    left at the exact conversion rather than pretending the corpus validated a different value.

A test fixture bug surfaced on the way: stillFrames in the use-case test stamped
timestampMs: 0 on every held frame. Invisible while the segmenter counted frames; with time
it collapses the span to zero. Timestamps are continuous now.

Also in here: tools/browser

The harness that produced the evidence above, added as a tool rather than a vitest test — CI
has no Chrome, no camera and no corpus. It replaces the camera by playing a corpus clip into
getUserMedia via captureStream() (no fake-device flags, no y4m, no file serving) and reads
the diagnostics panel's DOM the way a person would.

It asserts only what holds on any device — the clip decoded, the weights loaded, a hand was
seen, the app mounted, and, only when the measured frame rate clears the shipped floor, that
the engine was asked. Frame rate, window counts, raw scores and the feature profile are
measured and printed, not asserted.

Recognition is deliberately not asserted: it needs a frame rate software-WebGL headless does
not reach (~1.3 fps here against a 3.5 fps floor), and a permanently red check invites someone
to lower a shipped threshold to make it green. The floor is read out of SignSegmenter.ts by
the harness rather than copied, and throws if the constants are renamed — duplicating them
would let the tool disagree with the app it measures, which is the bug class it exists to catch.

Verified it can fail, not just pass: a non-video clip and a server whose SPA fallback answers
200 without a bundle both exit 1 with specific diagnoses.

Not verified

Real-device confirmation. This box maxes at ~1.3 fps headless, below the floor the fix implies,
so recognition end-to-end still needs a phone. The panel now answers it directly: "Fotogramas"
after ~15 s gives the device's frame rate, and "Ventanas cerradas (N descartadas por cortas)"
says whether it clears 3.5 fps.

Note the Pages deploy may fail on merge while GitHub Actions and Pages are in a major outage;
a timed-out deploy burns the merge SHA and recovery is another commit.

@Endika
Endika enabled auto-merge (rebase) August 6, 2026 19:06
@Endika
Endika force-pushed the fix/segmenter-time-thresholds branch from 6972c47 to 3d1431b Compare August 7, 2026 07:34
@Endika
Endika merged commit f6f76b8 into main Aug 7, 2026
7 checks passed
@Endika
Endika deleted the fix/segmenter-time-thresholds branch August 7, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant