registration.mesh_other() returns reflection even when disabled

[jpmaterial]

When you try to register a mesh with a reflection of itself, you sometimes get a transformation that contains a reflection as result, even if you disabled that.

def test_icp_mesh_no_reflection(self):
    from pathlib import Path
    a = g.trimesh.load_mesh((Path(__file__) / "../data/bool.stl").open("rb"), file_type="stl")
    b = a.copy()
    b.apply_transform(g.np.array([
        [ 0.14453548, -0.98680484,  0.0729774 ,  0.        ],
        [ 0.98752505,  0.13919728, -0.07360979,  0.        ],
        [ 0.06248024,  0.08270624,  0.99461344,  0.        ],
        [ 0.        ,  0.        ,  0.        ,  1.        ]]))
    b.apply_transform(g.np.diag([-1,-1,1,1]))
    b.vertices[:, 1] *= -1
    g.np.random.seed(0)
    T, cost = g.trimesh.registration.mesh_other(
        a,
        b,
        samples=500,
        reflection=False,
        scale=False,
    )
    assert g.np.linalg.det(T) > 0

Tried on the 7.6.5 branch, but there is nothing relevant in the changelogs, so the issue should still be active.

Main libs:

    numpy                      2.2.4                  py312h3150e54_0        conda-forge
    python                     3.12.9                 h3f84c4b_1_cpython     conda-forge
    python_abi                 3.12                   8_cp312                conda-forge
    rtree                      1.4.1                  pyh11ca60a_0           conda-forge
    scipy                      1.16.3                 pypi_0                 pypi
    shapely                    2.1.2                  py312h37f46ab_2        conda-forge

If helpful I can send this as a PR

bool.zip

[jpmaterial]

Mirroring b's vertices and thus turning its triangles inside out seems to be a necessary precondition for this bug.

[mikedh]

Yeah a PR with a fix and test would be great! I suppose this is incorrect?

# no reflection allowed, so determinant must be 1.0
R = np.dot(np.dot(u, np.diag([1, 1, np.linalg.det(np.dot(u, vh))])), vh)

That's a lot of dot products but is it incorrectly handling non-XY-plane reflections?

[jpmaterial]

Sorry if I was misleading, I only have a PR reproducing the problem.

And my guess would be that that check does not work if one mesh is inside-out.