fix: tune segmentation to match training and correct the engine size - #29
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things, both found by testing the deployed app rather than the model.
The app was feeding the model worse input than it trained on
Training resamples every frame of a recording; the app resamples whatever
SignSegmenteremits. Nobody was measuring the second one.tools/train/simulate_app.pyreplays the real test split through a port of the segmenter and scores the real weights:Two constants: motion threshold 0.08 → 0.03, settle frames 6 → 10. Swept over the test set rather than guessed. The lower threshold also fixes gentle signing, which at 0.08 never crossed the activation line at all — the segmenter stayed idle and the vocabulary engine was never asked.
A correction to my own earlier finding. I first measured this gap at 12 points and called it a deployed bug. It was my simulation: it read only the right hand, so every left-handed sign registered zero motion, never activated, and was dropped. 49 of 598 signs, all of them an artifact of the measurement. The real gap was 1.9 points. Recording it because the wrong number was alarming and the right one is not.
The engine size on screen was wrong
The panel said 41 MB. That is what sits in the repo, and it includes both WASM builds — SIMD and no-SIMD — of which any given browser downloads exactly one:
Reported as ~30 MB now.
And a real gap it exposed
Reported from the app: after pressing "descargar ahora" the panel showed 19 MB — exactly the models, with the WASM missing. Weights cached without the runtime that executes them looks fine online and breaks offline, which is a front-page claim.
I cannot reproduce this without a browser, so instead of guessing at a fix I made it visible:
report()now returnshasRuntime, and the panel says so outright when the runtime is absent. If it still shows 19 MB and that warning after this ships, the service worker is not intercepting MediaPipe's WASM fetch and that is the next thing to chase.