[Feature] Agent-side feedback: trajectory sampling, physics params on create_object, and three more silent-failure guards

[whats2000]

Feature request

Feedback from the consumer side: one long agentic session (build a warehouse
pick-and-place demo end to end on 6.0 PhysX, then repro work on 5.1 and Newton).
Ranked by how much each would actually have changed the session, with the moment
that motivates it.

---

1. step_simulation should be able to return a trajectory, not just the end state.

observe_prims / observe_joints return the state after N steps. Measuring a
rate therefore means differencing two single points across separate round trips.

This produced the worst misdiagnosis of the session. Measuring conveyor speed
that way gave ~1.16 m/s against a configured 0.3, and I chased a phantom 3.9x
discrepancy across roughly six play/stop cycles — the samples were silently
mixing in the box's drop onto the belt and its fall off the end. When I finally
subscribed to physics steps by hand and printed per frame, the answer was
obvious on sight: a flat 1.2000 m/s across ~200 frames, exactly 4x, which is
the belt prim's X scale.

Proposal: sample_every: int = 0 on step_simulation, so observe_prims /
observe_joints return a series instead of one point. Rates, convergence,
oscillation and settling all become one call instead of a hand-rolled
subscriber. This is also the single best lever against execute_script gravity
— it removes the round-trip incentive rather than arguing against it.

2. create_object needs mass / friction / restitution.

physics_enabled=True is not enough for an object that interacts. A box
carried by a conveyor needs a bound physics material and a mass, and neither is
expressible, so the entire station build dropped to raw USD:

UsdPhysics.MassAPI.Apply(p).CreateMassAttr().Set(0.5)
mat = UsdShade.Material.Define(stage, path)          # + UsdPhysics.MaterialAPI
UsdShade.MaterialBindingAPI(p).Bind(mat, materialPurpose="physics")

This one gap accounts for most of the raw USD in the build. (PhysxMaterialAPI
applied straight to the rigid body is silently inert — the boxes slip in place
on a moving belt — which is exactly the sort of trap a tool should absorb.)

3. An overlap / "what is this touching" query.

The pallet intersected fr3_link0, so the arm was physically jammed: every
joint command looked correct, fr3_joint1 did not move for 450 stepped frames,
and nothing errored. I found it by computing world AABBs by hand for each link.
get_physics_state has a contacts field but it was empty in practice.

"What is this prim intersecting?" would have been one call instead of a
diagnostic detour, and it is a generic manipulation need, not a one-off.

4. Robot frame introspection.

RMPflow drives gripper_center; I was measuring convergence against
fr3_hand_tcp. The 0.14 m gap never closed, so the state machine sat in one
state indefinitely with no error. I only found it by loading the motion-policy
config by hand:

cfg = interface_config_loader.load_supported_motion_policy_config("FR3", "RMPflow")
cfg["end_effector_frame_name"]   # -> 'gripper_center', not 'fr3_hand_tcp'

get_robot_info could report available kinematics frames and the motion-policy
end-effector frame. Anyone doing manipulation hits this.

5. stop_simulation should warn when the restored pose differs from the authored USD pose.

PhysX caches each rigid body's reset pose by prim path at the session's first
Play, and that cache survives deleting and recreating the prim. Rebuilt 0.05 m
boxes silently came back as 0.06 m at superseded coordinates — outside the
current pallet, so they spawned off it and fell to the floor. It reads exactly
like the arm knocking boxes over.

stop_simulation is the moment the divergence appears and could compare cheaply.
This is the same instinct as velocity_warning and position_source, which are
the two highest-value things in this server — they are what stopped me
publishing wrong measurements. Applying that pattern to a third silent failure
seems more valuable than any new capability here.

6. A one-line-per-tool index in the always-in-context instructions.

Under deferred tool schemas, "is there a tool for X?" costs a lookup while
"write Python" costs nothing. That asymmetry is invisible server-side and
decisive at the agent's decision point. A compact index in _INSTRUCTIONS
makes the surface discoverable at zero marginal cost.

7. Smaller things

• capture_image returns an error telling you to call again, on every camera's
  first capture. Polling internally would halve the calls.
• execute_script could set sys.dont_write_bytecode — stale .pyc from Kit's
  interpreter silently ran old code twice after I edited a module.
• create_object / create_robot with replace_if_exists would help iteration
  loops; I hand-rolled teardown-and-rebuild.
• reload_script also executes standalone .py files, but nothing in the name
  says so. Looking for "run my build script", nobody reaches for "reload" — I
  used execute_script with a hand-rolled sys.modules dance instead.

Motivation

The tools that helped most were not the most-used ones. get_physics_state was
called about twice all session, and one of those returned velocity_warning
telling me the zeros were placeholders rather than a measurement — without it I
would have concluded the conveyor was dead and "fixed" a non-bug.
get_joint_config was called once and cracked the jammed-arm diagnosis in a
single response.

So the pattern worth extending is not breadth, it is tools that carry knowledge
the caller does not have (create_camera(target=) computing look-at,
create_physics_scene encoding the initialise-before-articulation ordering) or
that warn about a silent failure. The gaps that cost the most were the ones that
forced raw USD for ordinary work — items 1-4 above.

Related: #37, #38, #39 for defects found in the same session.

Your contribution

Happy to implement any of these; items 1, 2 and 7 look self-contained. Item 5 is
the one I would pick first — it is the smallest change with the largest effect on
whether a wrong answer reaches the caller.