[Bug]: Module-scope TF32 writes leak across the test suite; can the library flag be opt-out?

[rakhimovv]

Thanks for releasing v2.

Question

TF32 looks like a deliberate default for cuRobo, so I am not arguing with the precision choice.
My question is about the scope: would you take a PR that keeps TF32 on by default but sets it
somewhere a caller can opt out of, instead of at module import?

Setup

• cuRobo at 8e734f3 (current main), installed per docs/getting-started/installation.rst
• Python 3.11.15, torch 2.13.0+cu130, CUDA 13.1, NVIDIA H100 80GB, driver 580.126.16
• TF32 only exists on SM80+, so none of this reproduces on older cards

Why I think the default is deliberate

Four places turn it on, which is what convinced me this is intentional rather than an oversight:
seed_ik_solver.py#L37-L40,
benchmark/ik_benchmark.py:46-48, benchmark/motion_plan_benchmark.py:46-49, and
test_graph_planner_prm.py#L14-L15.

The concrete problem: it leaks between test modules

test_graph_planner_prm.py sets allow_tf32 = True at module scope, so importing it changes global
torch state for every test that shares the process. Under the default -n 4 --dist loadscope that
is whatever else lands on the same worker.

Two tests are sensitive to this and fail in a full-suite run while passing alone, which reads as
flakiness. Three runs, all -n0, on the versions above:

Selection	allow_tf32 at end	Result
the two tests alone	False	3 passed
the two tests, TF32 forced on	True	2 failed, 1 passed
the two tests + test_graph_planner_prm.py, nothing forced	True	2 failed, 1 passed

FAILED curobo/tests/_src/types/test_pose.py::TestPose::test_pose_apply_kernel[cuda:0]
FAILED curobo/tests/_src/perception/test_pose_detector.py::TestSampleRotations::test_rotations_are_valid

The third row is the one that matters: no forcing, and seed_ik_solver is never imported in that
session (I asserted it against sys.modules). The PRM test module alone is enough. The 1 passed
in rows two and three is the [cpu] parametrization, which TF32 does not affect.

Both assertions compare against tight fp32 bounds — test_pose.py:209 uses torch.allclose
defaults (rtol 1e-5) and test_pose_detector.py:211 asserts R @ Rᵀ ≈ I at atol=1e-5 — and
TF32's 10-bit mantissa cannot meet either. Observed error is ~2e-4 in both cases. Arguably those two
tests should pin the precision they assume rather than inherit process state; I am happy to send
that instead if you would rather fix it from that side.

The library-side half

Separately from the tests, the same pattern in seed_ik_solver.py means importing a public entry
point changes float32 matmul precision for the whole process:

$ python -c "
import torch
print('before:', torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.benchmark)
import curobo.inverse_kinematics
print('after: ', torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.benchmark)"
before: False False
after:  True True

curobo.motion_planner does the same. A user importing cuRobo alongside their own training code
gets reduced matmul precision globally with no documented opt-out. That import also sets
cudnn.benchmark = True, which reverses what conftest.py:45 sets for determinism — not the cause
of the two failures above, since neither test runs a cuDNN convolution, but the same class of
surprise.

Measured vs. inferred

Measured: the three runs above, the flag flip on import, the four locations.
Inferred: that TF32 was meant as a solver speedup rather than a global default. The lines date to
the v2 initial release (457b658), so I have no signal on intent and could easily be wrong.

If it wasn't meant to be global, exposing it as a runtime flag next to cuda_graphs /
torch_compile, or setting and restoring it inside SeedIKSolver, would both work. Glad to send a
PR once you say which shape you'd accept.

---

I searched open and closed issues before filing and found nothing covering this, and reproduced on latest main (8e734f3).