Skip to content

feat: recognise 238 LSE signs with a trained model - #16

Merged
Endika merged 1 commit into
mainfrom
feat/lse-vocabulary-model
Aug 5, 2026
Merged

feat: recognise 238 LSE signs with a trained model#16
Endika merged 1 commit into
mainfrom
feat/lse-vocabulary-model

Conversation

@Endika

@Endika Endika commented Aug 5, 2026

Copy link
Copy Markdown
Owner

The vocabulary engine is real now. Trained on SWL-LSE (CC-BY-4.0), 63% top-1 and 83% top-3 on the dataset's own held-out test split — 598 samples, 238 classes, against a 0.4% random baseline. Useful, not authoritative; the UI says so.

No inference runtime, on purpose

onnxruntime-web needs 13 MB of WASM to run a 2.4 MB model, and on GitHub Pages it cannot even use threads because Pages sends no COOP/COEP headers. Shipping that would have tripled the download for one small network — and this network is a fixed stack: LayerNorm → 2-layer bidirectional GRU → mean-pool → 2 linear layers.

So gru.ts computes it directly and the weights ship as one flat float32 blob. That is only defensible if it is provably correct, so the whole stack is checked against logits PyTorch produced for a fixed input, to 3 decimal places. A hand-written GRU that is subtly wrong — reset gate on the wrong term, gates read in the wrong order, a transposed weight — still runs and still returns plausible numbers. Nothing but this test catches that.

ONNX is still exported by the trainer as an interchange artifact; it just is not what the browser downloads.

Choices measured rather than assumed

  • GRU over MLP. A flat MLP over the same features managed 54% / 69%. The trajectory modelling is worth the recurrence, so I did not simplify it away.
  • hidden=128, not 256. At 256 the model was ~2M parameters against 6,336 training examples and 7.6 MB of weights. Halving it cost nothing measurable (63.2 vs 65.4 top-1) and improved top-3, at a quarter of the size.
  • Feature parity verified across languages. Generated the same synthetic hand in TypeScript and Python and compared the vectors: same length, same values, sums differing by 6e-5 (float32 rounding).

Two honest limitations

Signature layout gained the wrist position, because LSE gives location meaning. Prototypes taught before this are a different length and can never match, so PrototypeSignClassifier now drops them on load — the sign disappears and can be re-taught, rather than sitting in the list looking fine and never firing.

Position is weakly weighted in taught signs. It is 3 floats out of 66 per hand, so plain distance matching barely registers it: two taught signs differing only in height will be confused. The trained model learns its own weighting and copes. There is a test asserting the real figure rather than a hoped-for one.

Dataset vocabulary is clinical (ACUFENO, COLONOSCOPIA, QUIMIOTERAPIA) — that is what is openly licensed for LSE. The alphabet and taught signs cover everything else.

@Endika
Endika force-pushed the feat/lse-vocabulary-model branch from 8a17ef6 to 527fab0 Compare August 5, 2026 20:44
@Endika
Endika enabled auto-merge (rebase) August 5, 2026 20:44
@Endika
Endika merged commit ab4e20a into main Aug 5, 2026
6 checks passed
@Endika
Endika deleted the feat/lse-vocabulary-model branch August 5, 2026 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant