[Bug][Newton] MultiMeshRayCaster tracked targets: registered body_pattern (clone-dest glob) never matches prototype body labels

[wert93]

Isaac Lab version: v3.0.0-beta2.patch1 · Isaac Sim: 6.0.1 · backend: newton_mjwarp (PhysX same scene works)

Repro

A scene with per-env cloned kinematic RigidObjects ({ENV_REGEX_NS}/Obstacle_i) used as MultiMeshRayCasterCfg targets with RaycastTargetCfg(is_shared=False, track_mesh_transforms=True). On env creation with the Newton backend:

ValueError: Site 'ft_2' with body_pattern '/World/envs/env_*/Obstacle_0' matched no prototype bodies across 1 prototype(s).

Root cause (verified by instrumenting the error to print available labels)

Prototype builders label bodies with their full clone-SOURCE prim paths (e.g. /World/envs/env_0/Obstacle_0, /World/envs/env_0/Robot/FL_hip), and NewtonManager.cl_register_site matches body_pattern as a regex against those labels. But _NewtonRayCasterMixin._resolve_target_owner_exprs (clone-plan branch) registers the clone-DEST glob dest_glob + suffix = /World/envs/env_*/Obstacle_0 — and env_* as a regex means env + repeated underscores, which can never match env_0. (The cl_register_site docstring examples like Robot/link0 don't match the actual label format either.)

There is a second latent bug behind it: _tracked_site_labels_by_target is keyed by the registration-time owner expressions, while _create_tracked_target_view receives a third, differently-formatted expression list from _build_mesh_records — so after fixing the pattern, the dict lookup KeyErrors.

Fix (tested)

1. Register the escaped full source prim path (re.escape(owner_prim_path)) — exact literal match against the prototype label.
2. Consume tracked-site labels positionally (the base base_multi_mesh_ray_caster loop creates views exactly once per tracked target, in _raycast_targets_cfg order) instead of keying by unstable string forms.
3. (Optional QoL) extend the no-match ValueError in _cl_inject_sites to print available body labels — this is what made the root cause obvious.

With the patch below, our local-planner task (28 tracked per-env kinematic targets + robot articulation, 8×4 m grid ray caster) trains on newton_mjwarp with no errors; PhysX behavior unchanged.

diff --git a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
index 238253b..b5edbf1 100644
--- a/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
+++ b/source/isaaclab_newton/isaaclab_newton/physics/newton_manager.py
@@ -853,10 +853,12 @@ class NewtonManager(PhysicsManager):
                 proto_sites.setdefault(proto_id, {})[label] = site_indices
 
             if not any_matched:
+                available = sorted({lbl for proto in proto_builders.values() for lbl in proto.body_label})
                 raise ValueError(
                     f"Site '{label}' with body_pattern '{body_pattern}' matched no prototype bodies "
                     f"across {len(proto_builders)} prototype(s). "
-                    f"Check that the pattern matches a body label in the prototype builder."
+                    f"Check that the pattern matches a body label in the prototype builder. "
+                    f"Available body labels (first 40 of {len(available)}): {available[:40]}"
                 )
 
         cls._cl_pending_sites.clear()
diff --git a/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py b/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
index b514d52..033f8ee 100644
--- a/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
+++ b/source/isaaclab_newton/isaaclab_newton/sensors/ray_caster/ray_caster.py
@@ -100,12 +100,18 @@ class _NewtonRayCasterMixin:
         """Register sensor and dynamic target sites before cloning occurs."""
         super().__init__(cfg)  # pyright: ignore[reportCallIssue]
         self._sensor_site_labels = self._register_sites_for_expr(self.cfg.prim_path)
-        self._tracked_site_labels_by_target: dict[tuple[str, ...], list[str]] = {}
+        # Labels are stored PER TRACKED TARGET IN CFG ORDER and consumed positionally by
+        # _create_tracked_target_view: the expression strings computed at registration time
+        # (prototype-local body patterns) never match the differently-formatted expressions
+        # the base class passes at view-creation time (clone-dest globs / regex-expanded
+        # paths), so string keys cannot work here.
+        self._tracked_site_labels_seq: list[list[str]] = []
+        self._tracked_view_cursor: int = 0
         for target_cfg in getattr(self, "_raycast_targets_cfg", []):
             if target_cfg.track_mesh_transforms:
-                owner_exprs = self._resolve_target_owner_exprs(target_cfg.prim_expr)
-                labels = self._register_target_sites_for_exprs(owner_exprs)
-                self._tracked_site_labels_by_target[tuple(owner_exprs)] = labels
+                owner_patterns = self._resolve_target_owner_exprs(target_cfg.prim_expr)
+                labels = self._register_target_sites_for_exprs(owner_patterns)
+                self._tracked_site_labels_seq.append(labels)
 
     def _register_sites_for_expr(self, prim_expr: str) -> list[str]:
         """Register Newton sites for a prim expression and return site labels."""
@@ -145,7 +151,13 @@ class _NewtonRayCasterMixin:
                         " to dynamic targets."
                     )
                 owner_prim_path = str(body.GetPath())
-                owner_exprs.append(dest_glob + owner_prim_path[len(source_path) :])
+                # Prototype builders label bodies with their full clone-SOURCE prim paths
+                # (e.g. "/World/envs/env_0/Obstacle_0", "/World/envs/env_0/Robot/FL_hip"),
+                # and cl_register_site matches body_pattern as a REGEX against those labels.
+                # The previous clone-DEST glob ("/World/envs/env_*/...") never matches:
+                # "env_*" as a regex means "env" + repeated underscores, not "env_0".
+                # Register the escaped source path — an exact literal match.
+                owner_exprs.append(re.escape(owner_prim_path))
             return list(dict.fromkeys(owner_exprs))
 
         # Legacy fallback: no clone plan, or the target is not owned by any plan row.
@@ -240,8 +252,11 @@ class _NewtonRayCasterMixin:
 
     def _create_tracked_target_view(self: Any, target_prim_path: str | list[str]):
         """Resolve dynamic multi-mesh target sites to raw Newton site indices."""
-        target_exprs = target_prim_path if isinstance(target_prim_path, list) else [target_prim_path]
-        labels = self._tracked_site_labels_by_target[tuple(target_exprs)]
+        # Views are created once per tracked target, in cfg order, by the
+        # base_multi_mesh_ray_caster loop — consume the registered labels positionally
+        # (the modulo guards a hypothetical re-initialization pass).
+        labels = self._tracked_site_labels_seq[self._tracked_view_cursor % len(self._tracked_site_labels_seq)]
+        self._tracked_view_cursor += 1
         site_indices = self._resolve_site_indices(labels, str(target_prim_path), self._num_envs)
         return wp.array(site_indices, dtype=wp.int32, device=self._device)