fix(spa): current-frame Procrustes, working live IK, model-scale fit, UI polish#61
Merged
HugoFara merged 5 commits intoJun 3, 2026
Merged
Conversation
The per-row x sat at the right edge with only 4px padding, so an overlay scrollbar floated on top of it and it couldn't be clicked. Reserve a 10px gutter on the scroll area, enlarge the button hit target, and use a proper multiplication-sign glyph.
The global rigid+scale alignment fit its transform on the per-keypoint mean pose averaged over the whole clip. When the animal walks or turns, that mean collapses toward the centroid (bbox diag 0.74 vs a real per-frame ~2.17 on the bundled rat demo), so Procrustes picks a hugely inflated scale (~23x instead of ~9x) and a meaningless rotation. Applied to real full-size frames, the keypoints rendered several times too big and at the wrong angle. The s>10 bbox fallback didn't help — it fits the same collapsed mean. Fit instead on a single representative pose: the frame the user is viewing (frameIndex), picking the nearest gap-free frame and falling back to the legacy time-mean only when no frame has a complete mapped-keypoint set. Since IK re-solves per frame (trunk Procrustes seed), this global step is just the coarse display alignment. Fixed in both the browser (localApi) and backend (alignment.py) paths.
Load STAC H5 has no in-browser path, so it's disabled + dimmed whenever the backend is unreachable. Load ACM is gated on the loaded species: it stays enabled for a bundled demo model (rat) and disables for non-demo models in standalone, where it would otherwise load mismatched rat keypoints. Threads a session-only modelHasDemoData flag through setXmlData, derived from the bundled species table via xmlHasDemoData/hasBundledDemo.
jacobianIk passed a plain Float64Array as the mj_jacBody out-parameter. The @mujoco/mujoco binding copies a plain JS array *in* but never writes it *back* (mjtNum* out-params must be a heap-allocated DoubleBuffer read via GetView()), so the Jacobian read as all-zeros. That zeroed the gradient, so the joints-only refinement never moved a single DOF — the solver placed the root via the JS trunk-Procrustes seed and then ran 25 no-op iterations, returning the seed pose. Visibly: the model reoriented as a whole but never bent, and IK error sat at the seed-only value (~30mm) instead of converging (~15mm). Allocate the Jacobian on the WASM heap, read it through GetView(), free with delete(). This is the only such misuse — mOptOffsets reads heap views already.
modelScale was applied only as a render-group scale on the model mesh, about
the model centre. The IK, keypoints, and error lines ignored it, so dragging
the slider grew the model but left it fitted (and error-lined) at the old size
— it visibly detached from the keypoint cloud and the error lines went stale.
Make modelScale scale the model about the ORIGIN (outermost group) so it's
invertible by the solver, then:
- localApi.runQuickStac divides the IK targets by modelScale (fit the native
model to keypoints/modelScale; the ×modelScale render lands the bodies back
on the fixed cloud) and reports error in rendered/world space;
- ikRunner passes modelScale through;
- ErrorLines scales the body+offset endpoint by modelScale so lines connect
to the rendered body and the error is the true rendered distance (and
recomputes on slider change);
- MuJoCoModel's click-to-set-offset divides the world hit point back to the
native frame;
- setModelScale clears the warm-start cache so the next auto-IK cold-starts
and re-seeds the root (joints-only warm refine can't recover a big scale
jump).
The slider is now a genuine fit control: changing it re-fits and the error
reflects how well a model of that size matches the data. No-op at the default
modelScale=1. Standalone-only — the backend q_opt also ignores scale_factor,
so backend-mode model scaling remains a separate (untouched) parity gap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bundle of standalone-mode fixes found while recording the demo video (#59).
Commits
fix(ui)— keep the mapping-row delete×clear of the overlay scrollbar so it stays clickable.Before:
After:
fix(align)— fit Procrustes auto-alignment on the current frame instead of the time-averaged mean pose. The mean collapses when the animal turns (per-frame rotation std ~77°), producing a wildly oversized/mis-rotated fit. Applied to both the in-browserlocalApiand the FastAPI backend.feat(spa)— gate backend-only toolbar buttons (Load STAC H5, Load ACM) in standalone mode so they're visibly disabled rather than silent dead ends.fix(ik)— the important one: readmj_jacBody's output via a heap-allocatedDoubleBufferinstead of a plainFloat64Array. Embind never writes back into a JS-owned typed array, so the Jacobian was always zero → zero gradient → joints never articulated (the solver returned the seed pose, err ~30mm). Live IK now actually bends the skeleton.Before:
After:
fix(spa)— make the model-scale slider meaningfully re-fit in standalone: scale the model about the origin (invertible by IK), divide IK targets by modelScale, scale the error-line endpoints, fix the click-to-offset handler, and clear the warm-start cache on scale change. No-op at the default modelScale = 1.Verification
Each algorithm change was checked against a faithful Python port run on the bundled rat data (mujoco 3.8.1): single-frame fit gives rmse ~2.46 mm vs the mean-pose 3.18 mm; the IK port articulates to ~35° max joint where the buggy path returned the seed; model-scale fit is best at 1x and degrades at 2x/0.5x as expected. Frontend typecheck passes.
Known parity gaps (not blocking)