Interactive web UI for aligning 3D motion-capture keypoints to MuJoCo body models. The exported YAML feeds the stac-mjx inverse-kinematics pipeline.
The app supports five species out of the box (rat, fly, mouse, worm, stick insect) and runs in two modes:
- Standalone (default) — static SPA, all computation in the browser via the MuJoCo WebAssembly module. Mapping, Procrustes alignment, live Jacobian-IK preview, and closed-form offset refit all run client-side. Hosted on GitHub Pages, no install required.
- With local backend — start the FastAPI backend and the SPA picks it up
automatically (per-call probe). Adds the features that need lab-private
Python: ACM multi-trial auto-load via
monsees-retargetand custom model-path loading. IK, offset refit, and alignment then route through the realstac-mjxcore instead of the in-browser JS ports.
Live IK preview — map keypoints and watch the Jacobian-IK fit update in real time, entirely in the browser.
https://github.com/talmolab/stac-keypoints-web/raw/main/docs/inverse-kinematics.webm
Multi-species — five MuJoCo body models out of the box, switchable from a single dropdown.
https://github.com/talmolab/stac-keypoints-web/raw/main/docs/animals-selection.webm
Visit the deployed site or build locally:
cd frontend
npm install
npm run devOpen http://localhost:5173. Pick a species from the model dropdown, load
your keypoints (.h5 / .mat), map keypoints to bodies, export YAML.
See docs/user-guide.md for the full workflow.
./start.shSpawns the backend and frontend in a tmux session. Requires Python 3.12+
with the project installed (pip install -e ".[dev]") and Node 22+. See
docs/deploy.md for env-var configuration.
- Click-to-assign: pick a keypoint, then a body part
- Drag 3D gizmos to fine-tune marker offsets
- Per-segment skeleton editor (e.g. shorten spine to match model proportions)
- Procrustes auto-alignment of keypoint cloud to model frame
- Color-by-error mode: per-keypoint Euclidean distance to the model fit
- Region error summary (forelimbs, hindlimbs, head, back) with auto L/R grouping
- Confidence-tinted keypoints; gap heatmap on the timeline
- Quality Report export: per-keypoint gap %, confidence histogram, per-keypoint error — JSON output suitable for downstream QC scripts
- Timeline scrubber, frame labelling (SLEAP-style)
- Undo/redo over mappings + offsets
- Save / Save As… via the File System Access API on Chrome/Edge (re-saves to the same file without re-prompting); blob download fallback on Firefox/Safari
- Persistent state via
localStorage
| Key | Action |
|---|---|
Space |
Play / pause |
← → |
Prev / next frame (Shift: ±10) |
Home / End |
Jump to first / last frame |
1 / 2 |
Mapping / Offset mode |
Cmd/Ctrl + S |
Save / re-save YAML |
Cmd/Ctrl + ⇧ + S |
Save As… |
Cmd/Ctrl + Z / ⇧Z |
Undo / redo |
L |
Label current frame |
? / H |
Toggle help overlay |
WASD QE RF |
Camera pan / orbit / vertical |
- User guide — loading data, mapping, exporting
- Developer guide — architecture, SPA dual mode, parity tests, contributing
- Deploy guide — GitHub Pages deploy, asset bundling, mesh preprocessor
- Known issues — limitations and future directions
MIT