feat: recognise 238 LSE signs with a trained model - #16
Merged
Conversation
Endika
force-pushed
the
feat/lse-vocabulary-model
branch
from
August 5, 2026 20:44
8a17ef6 to
527fab0
Compare
Endika
enabled auto-merge (rebase)
August 5, 2026 20:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The vocabulary engine is real now. Trained on SWL-LSE (CC-BY-4.0), 63% top-1 and 83% top-3 on the dataset's own held-out test split — 598 samples, 238 classes, against a 0.4% random baseline. Useful, not authoritative; the UI says so.
No inference runtime, on purpose
onnxruntime-web needs 13 MB of WASM to run a 2.4 MB model, and on GitHub Pages it cannot even use threads because Pages sends no COOP/COEP headers. Shipping that would have tripled the download for one small network — and this network is a fixed stack: LayerNorm → 2-layer bidirectional GRU → mean-pool → 2 linear layers.
So
gru.tscomputes it directly and the weights ship as one flat float32 blob. That is only defensible if it is provably correct, so the whole stack is checked against logits PyTorch produced for a fixed input, to 3 decimal places. A hand-written GRU that is subtly wrong — reset gate on the wrong term, gates read in the wrong order, a transposed weight — still runs and still returns plausible numbers. Nothing but this test catches that.ONNX is still exported by the trainer as an interchange artifact; it just is not what the browser downloads.
Choices measured rather than assumed
Two honest limitations
Signature layout gained the wrist position, because LSE gives location meaning. Prototypes taught before this are a different length and can never match, so
PrototypeSignClassifiernow drops them on load — the sign disappears and can be re-taught, rather than sitting in the list looking fine and never firing.Position is weakly weighted in taught signs. It is 3 floats out of 66 per hand, so plain distance matching barely registers it: two taught signs differing only in height will be confused. The trained model learns its own weighting and copes. There is a test asserting the real figure rather than a hoped-for one.
Dataset vocabulary is clinical (
ACUFENO,COLONOSCOPIA,QUIMIOTERAPIA) — that is what is openly licensed for LSE. The alphabet and taught signs cover everything else.