[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) by hujc7 · Pull Request #6324 · isaac-sim/IsaacLab

hujc7 · 2026-07-02T08:52:26Z

DO NOT MERGE — integrated reference branch for the dexterous task-cleanup series; the reviewable stack is in the Review Map below. Per-part details live in the part PRs.

Update (2026-07-16) — torch-first rework: the mainline task code was reworked torch-first per the stable task convention; the warp implementation moved to isaaclab_tasks_experimental (#6582, preserved from this branch's history).

Update (2026-07-17) — restack + validation: all parts rebuilt on the torch tree and CI-green; the Newton pin split out to #6584. Torch re-validation: Direct state SR 0.85–1.00 @2000 it; camera rsl_rl takeoff (reward peak 1222, SR peak 1.0) @1500 it — both with frame-verified videos.

Update (2026-07-18) — lump review round propagated: Parts 5/6/8 histories rebuilt with the review folded in (they had no human review yet); Parts 3/4 received append commits; Part 7 merged the rebuilt base. Union-of-parts verified byte-identical to this branch. The Newton pin remnants left this branch and live only in #6584.

Review Map

Exact changes: a stacked PR's page shows the cumulative diff of its dependency chain; the link pins the commit range that is the PR's own contribution.
Links pin specific SHAs and can go stale after a branch update — this table on [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) #6324 is refreshed first.

PR	Depends on	Exact changes
#6410 [Docs] Environment overview regen	—	—
#6411 Part 1/8: Newton cloner/cubric/visualizer fixes	—	—
#6412 Part 2/8: OVPhysX articulation + manager runtime	—	—
#6413 Part 3/8: Reorient Direct, torch	—	—
#6414 Part 4/8: MARL-to-single-agent fix + handover/camera Direct	#6413	changes
#6418 Part 5/8: Reorient manager counterparts	#6413	changes
#6421 Part 6/8: Handover + camera manager counterparts	#6413, #6414, #6418	changes
#6415 Part 7/8: Benchmark success-rate utilities + docs	#6413, #6414, #6418, #6421	changes
#6582 Part 8/8: Warp variants → experimental (draft; merges last)	#6413	changes
📌 #6324 [DO-NOT-MERGE] Lumped validation reference (this PR)	ALL	—

Not split out (superseded on develop or out of series scope): the environment overview regen (#6410 covers develop) and a develop-obsoleted install-order test.

Summary

Manager-based counterparts, boolean success metrics (Metrics/success_rate), and PhysX/Newton/OVPhysX presets for the assigned dexterous tasks (Allegro + Shadow reorientation, OpenAI FF/LSTM, camera, handover).
Fixes handover construction on Newton (renamed distal joints) and hand resets below joint limits.
The campaign table below is the warp-era record; rows refresh as torch reruns land.
Per-part scope, review updates, and validation: see each part PR (Review Map above).

Direct vs manager training reference (qualifying campaign, seed 42)

Cell format: reward|eplen|success_rate|{trained}iter (final Train/mean_reward, Train/mean_episode_length, Metrics/success_rate).

Iteration budgets: 500 (Allegro), 3000 (all other rows). Every cell's source run consumed its full budget — verified cell-by-cell against the runs' event files (all three metrics of each cell match one run's final-iteration scalars). Cells read reward | episode length | success rate | iterations actually trained.

Task	Backend	Direct	Manager
Reorient Allegro	PhysX	340 \| 259 \| 0.63 \| 500iter	212 \| 285 \| 0.23 \| 500iter
Reorient Allegro	Newton	267 \| 254 \| 0.48 \| 500iter	203 \| 295 \| 0.17 \| 500iter
Reorient Shadow	PhysX	5725 \| 553 \| 1.00 \| 3000iter	6066 \| 566 \| 0.99 \| 3000iter
Reorient Shadow	Newton	3783 \| 523 \| 0.85 \| 3000iter	3866 \| 534 \| 0.86 \| 3000iter
Shadow OpenAI FF	PhysX	5422 \| 480 \| 1.00 \| 3000iter	3688 \| 599 \| 0.94 \| 3000iter
Shadow OpenAI FF	Newton	2762 \| 356 \| 0.67 \| 3000iter	2779 \| 507 \| 0.75 \| 3000iter
Shadow OpenAI LSTM	PhysX	5640 \| 605 \| 1.00 \| 3000iter	3811 \| 809 \| 0.93 \| 3000iter
Shadow OpenAI LSTM	Newton	3441 \| 479 \| 0.75 \| 3000iter	2278 \| 503 \| 0.76 \| 3000iter
Shadow Camera	PhysX	224 \| 564 \| 0.00² \| 3000iter	243 \| 595 \| 0.00² \| 3000iter
Shadow Camera	Newton	1307 \| 503 \| 0.83 \| 3000iter³	164 \| 457 \| 0.00² \| 3000iter
Shadow Handover	PhysX	1634 \| 441 \| 1.00 \| 3000iter	724 \| 367 \| 0.99 \| 3000iter
Shadow Handover	Newton	1471 \| 412 \| 1.00 \| 3000iter¹	1415 \| 400 \| 0.93 \| 3000iter¹

¹ Post-fix handover-Newton rows (Part 4's Newton fixes), full-budget qualifying runs on the warp-first environment code.
² Same success-rate definition as the other reorientation rows. Camera learning takes off late (reward ~400 plateau until a sharp transition; PhysX evidence crosses reward 1000 near iter ~1000, Newton near iter ~2300 — see ³); these rows' policies remained pre-takeoff at the recorded budget, so success at the 0.1 rad tolerance reads 0.00 and reward is the tracked pre-takeoff signal. The criterion itself is unchanged and proven reachable.
³ Full-budget qualifying run on the current branch head (seed 42, rgb_depth preset, Newton renderer, 1225 envs, rl_games workflow with its asymmetric actor-critic recipe): crossed reward 1000 at iter ~2300, tail-50 reward 1307, success rate 0.83 with ≥0.3 sustained for 43 consecutive iterations. Note the workflow difference: the other camera cells (and the rest of the table) are rsl_rl runs; the camera task has so far reached takeoff only under the rl_games recipe (also PhysX evidence: reward 1018, success 0.91 by iter ~1025), while the rsl_rl camera recipe remains pre-takeoff at full budget — a training-recipe sensitivity to be reconciled, not a physics/backend gap. Superseded on the torch tree: the torch-rework rsl_rl camera run reaches takeoff (reward > 1000, sr peak 1.0) by ~1500 iterations — the rsl_rl/rl_games recipe sensitivity no longer reproduces after the rework.
Reward scales differ per family; the Direct-vs-manager pair per row is the comparison that matters.

Torch re-validation (post-rework). Verified runs on the torch tree; the table below remains the warp-era campaign record.

Task	Workflow	Torch result	Warp-era, same recipe
Reorient Shadow Direct	Newton rsl_rl	3600–3900 \| sr 0.85–1.00 \| 2000it	3783 \| 0.85 \| 3000it
Shadow Camera Direct	Newton rsl_rl	peak 1222 \| sr peak 1.0 \| 1500it (takeoff)	pre-takeoff @3000it²

Phase-2 requirement fulfillment

🟢 Met 🟡 Partial / queued 🔴 Gap ⚪ Not applicable

Overall Phase-2 status	Count	Share of 392 applicable requirements
🟢 Met	291	74%
🟡 Partial / queued	101	26%
🔴 Gap	0	0%
Estimated progress	291 met + 0.5 × 101 partial	87%

118 N/A cells are excluded. Success-related acceptance criteria (P2-C3–C7 and dependent P2-D1–D2) are deferred pending definition; no provisional threshold is treated as a requirement. Reward and episode length remain diagnostic only. Previously validated evidence is retained wherever the code path is unchanged by this diff (the OVPhysX path always used the production assets); only artifacts tied to the removed asset are queued for re-run. Sim-to-sim and sim-to-real are later waves and excluded. Current PR head: 5d7ddbbee82.

Allegro cube reorientation — 🟢 49 · 🟡 15 · 88%

Registered variant	🟢	🟡	⚪	Progress
`Isaac-Reorient-Cube-Allegro-Direct`	18	7	5	86%
`Isaac-Reorient-Cube-Allegro`	18	7	5	86%
`Isaac-Reorient-Cube-Allegro-Play`	13	1	16	96%

Shadow state cube reorientation — 🟢 36 · 🟡 14 · 86%

Registered variant	🟢	🟡	🔴	⚪	Progress
`Isaac-Reorient-Cube-Shadow-Direct`	18	7	0	5	86%
`Isaac-Reorient-Cube-Shadow`	18	7	0	5	86%

Shadow OpenAI FF cube reorientation — 🟢 36 · 🟡 14 · 86%

Registered variant	🟢	🟡	🔴	⚪	Progress
`Isaac-Reorient-Cube-Shadow-OpenAI-FF-Direct`	18	7	0	5	86%
`Isaac-Reorient-Cube-Shadow-OpenAI-FF`	18	7	0	5	86%

Shadow OpenAI LSTM cube reorientation — 🟢 36 · 🟡 14 · 86%

Registered variant	🟢	🟡	🔴	⚪	Progress
`Isaac-Reorient-Cube-Shadow-OpenAI-LSTM-Direct`	18	7	0	5	86%
`Isaac-Reorient-Cube-Shadow-OpenAI-LSTM`	18	7	0	5	86%

Shadow Camera cube reorientation — 🟢 100 · 🟡 28 · 89%

Registered variant	🟢	🟡	⚪	Progress
`Isaac-Reorient-Cube-Shadow-Camera-Direct`	19	9	2	84%
`Isaac-Reorient-Cube-Shadow-Camera`	19	9	2	84%
`Isaac-Reorient-Cube-Shadow-Camera-Direct-Play`	15	3	12	92%
`Isaac-Reorient-Cube-Shadow-Camera-Play`	15	3	12	92%
`Isaac-Reorient-Cube-Shadow-Camera-Benchmark-Direct`	16	2	12	94%
`Isaac-Reorient-Cube-Shadow-Camera-Benchmark`	16	2	12	94%

Area	ID · Requirement	Direct train	Manager train	Direct Play	Manager Play	Direct benchmark	Manager benchmark
Structure	`P2-S1` · Assigned registration accounted for	🟢 Registered and assigned	🟢 Registered and assigned	🟢 Registered and assigned	🟢 Registered and assigned	🟢 Registered and assigned	🟢 Registered and assigned
	`P2-S2` · Manager-based training environment	🟢 Manager train exists for family	🟢 Manager train exists for family	🟢 Manager train exists for family	🟢 Manager train exists for family	🟢 Manager train exists for family	🟢 Manager train exists for family
	`P2-S3` · Direct/Manager contracts comparable	🟢 Focused parity checks pass	🟢 Focused parity checks pass	🟢 Focused parity checks pass	🟢 Focused parity checks pass	🟢 Focused parity checks pass	🟢 Focused parity checks pass
	`P2-S4` · Rendering identified by Camera name	🟢 Camera in public ID	🟢 Camera in public ID	🟢 Camera in public ID	🟢 Camera in public ID	🟢 Camera in public ID	🟢 Camera in public ID
	`P2-S5` · Play/Eval implementation consolidated	⚪ Not a Play/Eval variant	⚪ Not a Play/Eval variant	🟡 Public Play ID retained; generic consolidation pending	🟡 Public Play ID retained; generic consolidation pending	⚪ Not a Play/Eval variant	⚪ Not a Play/Eval variant
Assets & backends	`P2-B1` · New asset evaluated; fallback justified	⚪ Dropped: new-asset adoption removed from the consolidated standard	⚪ Dropped: new-asset adoption removed from the consolidated standard	⚪ Dropped: new-asset adoption removed from the consolidated standard	⚪ Dropped: new-asset adoption removed from the consolidated standard	⚪ Dropped: new-asset adoption removed from the consolidated standard	⚪ Dropped: new-asset adoption removed from the consolidated standard
	`P2-B2` · PhysX / Newton / OV presets resolve	🟢 All three physics presets resolve	🟢 All three physics presets resolve	🟢 All three physics presets resolve	🟢 All three physics presets resolve	🟢 All three physics presets resolve	🟢 All three physics presets resolve
	`P2-B3` · PhysX constructs and trains	🟢 Exact-head PhysX train smoke and full-budget training pass	🟢 Exact-head PhysX train smoke and full-budget training pass	🟢 PhysX replay and evaluation video exported from restored-asset checkpoint	🟢 PhysX replay and evaluation video exported from restored-asset checkpoint	🟢 PhysX benchmark bundle recorded on restored assets	🟢 PhysX benchmark bundle recorded on restored assets
	`P2-B3` · Newton constructs and trains	🟢 Exact-head Newton train smoke and full-budget training pass	🟢 Exact-head Newton train smoke and full-budget training pass	🟢 Newton replay and evaluation video exported from restored-asset checkpoint	🟢 Newton replay and evaluation video exported from restored-asset checkpoint	🟢 Newton benchmark bundle recorded on restored assets	🟢 Newton benchmark bundle recorded on restored assets
	`P2-B3` · OVPhysX constructs and trains	🟢 Train smoke and corrected task-frame runtime probes pass	🟢 Train smoke and corrected task-frame runtime probes pass	🟢 Exact OVPhysX replay and video pass (path unchanged by asset restoration)	🟢 Exact OVPhysX replay and video pass (path unchanged by asset restoration)	🟢 OVPhysX benchmark bundle passes (path unchanged by asset restoration)	🟢 OVPhysX benchmark bundle passes (path unchanged by asset restoration)
	`P2-B4` · Supported renderer matrix works	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap	🟡 7/8 pairs exit clean; Newton+OVRTX teardown gap
	`P2-B5` · Camera framing and ground are usable	🟡 8/8 frame routes captured; human framing gate pending	🟡 8/8 frame routes captured; human framing gate pending	🟡 8/8 frame routes captured; human framing gate pending	🟡 8/8 frame routes captured; human framing gate pending	🟡 8/8 frame routes captured; human framing gate pending	🟡 8/8 frame routes captured; human framing gate pending
Success	`P2-C1` · RSL-RL configuration provided	🟢 RSL-RL config resolves	🟢 RSL-RL config resolves	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-C2` · Success rate logged efficiently	🟢 Success metric emitted through training extras	🟢 Success metric emitted through training extras	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-C3` · Success criterion defined and configured	🟡 Acceptance criterion pending definition	🟡 Acceptance criterion pending definition	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-C4` · Success met within iteration/time budget	🟡 Acceptance run deferred until criterion is defined	🟡 Acceptance run deferred until criterion is defined	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-C5` · Multi-seed sanity check	🟡 Multi-seed acceptance deferred with criterion	🟡 Multi-seed acceptance deferred with criterion	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-C6` · Policy replay meets success criterion	🟡 Policy acceptance replay deferred with criterion	🟡 Policy acceptance replay deferred with criterion	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row
	`P2-C7` · Two-reviewer behavior validation	🟡 Behavior acceptance review deferred with criterion	🟡 Behavior acceptance review deferred with criterion	⚪ Validated at task-family policy level	⚪ Validated at task-family policy level	⚪ Validated at task-family policy level	⚪ Validated at task-family policy level
	`P2-C8` · Reward and episode length tracked (non-gating)	🟢 Exact M2 events contain reward and episode length	🟢 Exact M2 events contain reward and episode length	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
Quality	`P2-Q1` · Task-code hygiene	🟢 Retained diff hygiene review passes	🟢 Retained diff hygiene review passes	🟢 Retained diff hygiene review passes	🟢 Retained diff hygiene review passes	🟢 Retained diff hygiene review passes	🟢 Retained diff hygiene review passes
	`P2-Q2` · Independent package dependencies	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers	🟢 Wheel metadata covers direct imports, including Franka-soft deformable helpers
	`P2-Q3` · Variants and selections documented	🟢 Environment overview lists IDs and selections	🟢 Environment overview lists IDs and selections	🟢 Environment overview lists IDs and selections	🟢 Environment overview lists IDs and selections	🟢 Environment overview lists IDs and selections	🟢 Environment overview lists IDs and selections
	`P2-Q4` · Open-issue audit	🟢 Audit complete; only actually resolved issues close	🟢 Audit complete; only actually resolved issues close	🟢 Audit complete; only actually resolved issues close	🟢 Audit complete; only actually resolved issues close	🟢 Audit complete; only actually resolved issues close	🟢 Audit complete; only actually resolved issues close
	`P2-Q5` · Automated contract checks	🟢 Registration/parity/preset/asset/package checks pass	🟢 Registration/parity/preset/asset/package checks pass	🟢 Registration/parity/preset/asset/package checks pass	🟢 Registration/parity/preset/asset/package checks pass	🟢 Registration/parity/preset/asset/package checks pass	🟢 Registration/parity/preset/asset/package checks pass
Delivery	`P2-D1` · Current-asset qualifying checkpoint	🟡 Checkpoint qualification deferred with success criteria	🟡 Checkpoint qualification deferred with success criteria	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant	⚪ Not a training variant
	`P2-D2` · Replay/video and publication artifacts	🟡 Replay/publication acceptance deferred with success criteria	🟡 Replay/publication acceptance deferred with success criteria	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row	⚪ Not a qualifying-checkpoint row
	`P2-D3` · Runtime performance recorded	🟢 Steps/s, duration, and peak memory re-recorded on restored assets	🟢 Steps/s, duration, and peak memory re-recorded on restored assets	🟢 Steps/s, duration, and peak memory re-recorded on restored assets	🟢 Steps/s, duration, and peak memory re-recorded on restored assets	🟢 Steps/s, duration, and peak memory re-recorded on restored assets	🟢 Steps/s, duration, and peak memory re-recorded on restored assets
	`P2-D4` · Dashboard metrics recorded	🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets	🟢 Event, throughput, and peak-memory bundle re-recorded on restored assets	⚪ Playback-only	⚪ Playback-only	🟢 Two-step benchmark event and runtime bundle recorded	🟢 Two-step benchmark event and runtime bundle recorded
	`P2-D5` · Recurring benchmark schema compatible	🟢 Program-wide M2 result schema validated	🟢 Program-wide M2 result schema validated	🟢 Program-wide M2 result schema validated	🟢 Program-wide M2 result schema validated	🟢 Program-wide M2 result schema validated	🟢 Program-wide M2 result schema validated

Shadow handover — 🟢 34 · 🟡 16 · 84%

Registered variant	🟢	🟡	🔴	⚪	Progress
`Isaac-Shadow-Handover-Direct`	17	8	0	5	84%
`Isaac-Shadow-Handover`	17	8	0	5	84%

Restore the Shadow Hand orientations and current passive distal joints used by Newton handover training. Add RGB-depth and normals camera presets and normalize signed normals for Newton-rendered observations.

Avoid binding an incompatible USD or Carbonite build before Kit initializes by moving backend imports to their use sites. Extend the environment import guard to cover Gym entry points.

Adopt the standard Shadow Hand asset for Newton tasks and remove the asset-specific root, tendon, and passive-joint workarounds. Add the missing Newton and OVPhysX presets, migrate scoped schema uses, and keep source installs valid by declaring and ordering the PhysX dependency.

…-env-convergence-codex

Leave stage-context synchronization pending after a temporary Isaac Sim import failure so a later successful import can share the thread-local stage. Add a kitless regression test for the fail-then-retry sequence.

Normalize actuator joint selections to the common slice-or-tensor contract so manager event randomization can consume them. Keep concrete joint IDs for OVPhysX torque application.

Expose converted observations and forward episode-length state so recurrent and feed-forward RSL-RL runners can train converted multi-agent environments.

Register recurrent Shadow Hand and single-agent Handover runners so every owned dexterous training variant has an RSL-RL path.

Map both actor and critic to the converted policy observation explicitly so the new runner does not rely on deprecated fallback discovery.

Drop deferred-import, global asset dependency, and unused camera modality changes that validation showed were not required. Keep the task-local asset, backend, and RSL-RL fixes.

Provide a Manager workflow matching the existing multi-agent adapter's observation, action, reward, reset, and termination contracts. Register explicit benchmark criteria and keep Newton object properties solver-neutral.

Add the mandatory Manager counterpart for the state-based Shadow Hand reorientation task while preserving the Direct task's action, observation, reward, reset, goal, and termination contracts. Register matching benchmark criteria and backend presets.

Use the standard lazy-export initializer so importing the Handover Manager configuration does not preload USD modules before SimulationApp starts. Add a subprocess regression check for the pre-Kit import contract.

Avoid reading the RL reset buffer before ManagerBasedRLEnv creates it during initial environment reset. Preserve the one-step autoreset suppression only when that buffer exists.

Preserve Direct's same-step terminal-action observation with a reset snapshot instead of a custom runtime action class. This keeps Manager configuration imports free of USD modules before SimulationApp starts and covers the pre-Kit contract.

Select the reset-action snapshot while episode length is zero so initial, terminal, and explicit user resets all match Direct's same-step action observation until the next transition.

Keep both hands' terminal raw actions in Manager reset observations so the single-agent observation contract continues to match Direct during initial, manual, and same-step autoresets.

Drop an unnecessary Allegro OVPhysX alias and update the Handover Newton documentation now that both workflows have converged with the shared settings.

Provide Manager counterparts for the OpenAI feed-forward, recurrent, and camera reorientation variants. Preserve Direct observation, noise, action, reward, reset, and termination contracts while exposing every physics and rendering preset.

Use the same environment and agent contracts so the two workflows are directly comparable. Existing Manager checkpoints must be retrained against the new observation and reward contract.

Validate all supported physics and renderer preset pairs, exclude playback and benchmark registrations from training collection, and complete the public camera feature term documentation.

…-env-convergence-codex

Add explicit Menagerie asset configurations and select them for PhysX and Newton while retaining compatible legacy assets for OVPhysX. Preserve the task action contracts across joint namespaces and tighten benchmark validation.

…-env-convergence-codex

…nager runtime (#6412) ## Summary - Fixes OVPhysX actuator joint indices to follow the common actuator indexing contract. - Fixes OVPhysX initialization alongside Kit by reusing Kit's registered PhysX schema provider. - Fixes the OVPhysX manager to support both the declared public runtime API and the current runtime API. - Regression tests included. Validated by full dexterous training runs on the OVPhysX backend; split out of the lumped validation branch #6324 (Part 2 of 11). ## Dependencies - None. ## Series review map Full integrated diff + training/validation evidence: the lumped validation PR #6324 (DO-NOT-MERGE). | Part | PR | |---|---| | Docs: regenerate the environment overview table | #6410 | | Part 1/11: Newton runtime fixes (cloner rows, cubric fallback, viz teardown) | #6411 | | **Part 2/11: OVPhysX runtime fixes (this PR)** | #6412 | | Part 3/11: success-rate metrics for the Direct reorientation tasks | #6413 | | Part 4/11: RSL-RL training for the handover Direct task | #6414 | | Part 5/11: success-rate support in the benchmark utilities | #6415 | | Part 6/11: renderer presets for the Direct camera task | #6416 | | Part 7/11: OVPhysX presets for the dexterous tasks | #6417 | | Part 8/11: Allegro manager counterpart | #6418 | | Part 9/11: Shadow + OpenAI manager counterparts | #6419 | | Part 10/11: Shadow camera manager counterpart | #6420 | | Part 11/11: Shadow handover manager counterpart | #6421 | --- ### Exact changes in this PR - OVPhysX backend changes + tests: 1f7a433

Replace the module-level SHADOW/OPENAI/ALLEGRO/HANDOVER _SIM_CFG variables with four small task-cfg base classes that both the Direct and the manager configurations inherit (multiple inheritance with the framework env-cfg bases). The simulation parameters for each family are now authored exactly once and consumed through inheritance, removing the per-class sim assignments and the hand-copied simulation block in the OpenAI manager's __post_init__ — the one spot that had silently duplicated (and could drift from) the shared values. Verified: all fifteen env cfgs construct with byte-identical simulation parameters, and the value-parity, torch-math, and core-utils suites pass (30/30).

…us-env-convergence-codex

The OVPhysX runtime fixes merged to develop via their part PR with its own fragment; the lump's duplicate fragment would double the changelog entry.

Replace the local _quat_mul_xyzw tuple helper with isaaclab.utils.math.quat_mul, which already uses the (x, y, z, w) component order. The composition runs in float64 and was verified bit-identical to the previous helper (and to the wp.quatd math it originally replaced).

The per-backend scene preset entries repeated identical num_envs, env_spacing, and replicate_physics values, differing only in clone_in_fabric (Fabric cloning is unsupported on Newton). Move the invariant values onto each family's shared scene class as defaults so the presets declare only their backend delta. Verified byte-identical across all twenty preset variants.

Move the five helper classes that are meaningful only inside one preset class into that class as nested classes, per the repository's nested-class convention: the handover, Allegro, Shadow Direct, and Shadow OpenAI manager scene cfgs and the tiled-camera variant base. _ShadowHandManagerSceneCfg stays module-level (base of two subclasses, one in another file) as does _ShadowHandCameraManagerSceneCfg (consumed by both the train and the Play presets). Nested configclasses must carry their own @configclass decorator — without it, an outer-decorated preset reprocesses the preset instances and the nested class's unannotated defaults resolve to MISSING. All twenty scene preset variants and the camera preset verified value-identical.

- Fix the Allegro PLAY configuration leaving the default preset at 8192 environments: set_num_envs now covers the default alternative (the configclass deep-copies it independently of physx). - Add deprecation directives and one-shot runtime warnings to the three legacy reorient reward terms, matching the deprecated terminations. - Extend the value-parity test to the reset-noise scalars and the successes-based timeout threshold (0 on the Direct side corresponds to the plain time_out term on the manager side). - Document that resets now sample uniformly across the full joint range (previously biased toward the lower half) alongside the clamp fix. - Drop the stale OpenUSD installation bullet from the isaaclab fragment; that change is not part of this diff. - Build zero reset velocities with torch.zeros_like instead of clone-then-zero in both reset events.

evaluate_reorient_success now computes the orientation error through isaaclab.utils.math.quat_error_magnitude instead of a local formula: the shared utility is torch.jit-scriptable (verified by compiling a scripted caller from a source file) and decision-identical at the task tolerances — zero success-threshold flips over 20k random pairs, with differences up to 9e-4 rad only near pi where the shared atan2-based form is the more accurate one. The local direct_reorient_rotation_distance is deleted. Rename direct_reorient_reward to reorient_reward: shared symbols carry no paradigm prefix. The DirectReorientReward and DirectReorientTimeout manager terms keep the prefix deliberately — there "Direct" names the Direct-compatible contract they implement.

The manager-based terms wrap the shared evaluators rather than the Direct-workflow implementation, so the Direct prefix read as a paradigm crossing at the use sites. Rename DirectReorientReward to ReorientReward and DirectReorientTimeout to ReorientTimeout, matching HandoverReward, and state the Direct-parity contract in the class docstrings instead. Both classes are new on this branch, so no deprecation is required.

The Part 1 PR evolved through its own review (cloner and visualizer fixes) and had the Newton pin stripped out to the dedicated pin PR; mirror both into the lump so the series union and the lump stay byte-identical. The Newton pin and MuJoCo overrides now live only in the pin PR.

The legacy manager-based reorientation configuration ReorientObjectEnvCfg and the terms only it consumed (success_bonus, track_pos_l2, track_orientation_inv_l2, max_consecutive_success, object_away_from_goal) are replaced by the Direct-parity manager configurations and terms (ReorientReward, ReorientTimeout). Task-level symbols are not part of the core public API, so they are removed outright instead of shipping a deprecation cycle; the gym task IDs already point at the new configurations.

Its only consumer was the removed legacy ReorientObjectEnvCfg; neither the Direct env nor the parity manager configurations use an object-to-robot distance criterion.

The reorient manager cfgs repeated the same command, reward, and termination declarations per robot with one or two differing scalars. Move the defaults into two term factories and the invariant termination section in reorient_task_base; each variant now declares a small plain section class stating only its deltas (Allegro tolerance 0.2, OpenAI tolerance 0.4 with fall penalty and noisy-EMA action source). The camera manager cfg inherits the sections instead of re-declaring them.

Add reorient_joint_action and reorient_reset_event factories next to the existing command/reward factories: the action term differed only by the joint-name constant and the reset-event values were identical across all four copies (state, Allegro, and both OpenAI preset variants). Align the Allegro scene preset on the alias idiom used by the Shadow scene and loop set_num_envs over the backend alternatives.

Fix Newton dexterous task configurations

2b1b621

Restore the Shadow Hand orientations and current passive distal joints used by Newton handover training. Add RGB-depth and normals camera presets and normalize signed normals for Newton-rendered observations.

github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 2, 2026

Defer pxr/carb imports on environment construction

1d7dbd4

Avoid binding an incompatible USD or Carbonite build before Kit initializes by moving backend imports to their use sites. Extend the environment import guard to cover Gym entry points.

github-actions Bot added the asset New asset feature or request label Jul 2, 2026

hujc7 added 3 commits July 2, 2026 10:08

Merge remote-tracking branch 'origin/develop' into jichuanh/dexterous…

1d5dbec

…-env-convergence-codex

Format dexterous preset tests

99025bd

hujc7 force-pushed the jichuanh/dexterous-env-convergence-codex branch from 23c2ef8 to 99025bd Compare July 2, 2026 10:08

hujc7 added 22 commits July 2, 2026 10:13

Retry deferred Isaac Sim stage sharing

3f8b1b0

Leave stage-context synchronization pending after a temporary Isaac Sim import failure so a later successful import can share the thread-local stage. Add a kitless regression test for the fail-then-retry sequence.

Fix OVPhysX actuator joint indexing

eb651e9

Normalize actuator joint selections to the common slice-or-tensor contract so manager event randomization can consume them. Keep concrete joint IDs for OVPhysX torque application.

Fix MARL adapter observations for RSL-RL

0380e6b

Expose converted observations and forward episode-length state so recurrent and feed-forward RSL-RL runners can train converted multi-agent environments.

Add RSL-RL configs for dexterous tasks

a721d72

Register recurrent Shadow Hand and single-agent Handover runners so every owned dexterous training variant has an RSL-RL path.

Declare Handover RSL-RL observation groups

fae1971

Map both actor and critic to the converted policy observation explicitly so the new runner does not rely on deprecated fallback discovery.

Remove unrelated convergence changes

db92802

Drop deferred-import, global asset dependency, and unused camera modality changes that validation showed were not required. Keep the task-local asset, backend, and RSL-RL fixes.

Add manager-based handover environment

b9cdf2b

Provide a Manager workflow matching the existing multi-agent adapter's observation, action, reward, reset, and termination contracts. Register explicit benchmark criteria and keep Newton object properties solver-neutral.

Add state-based Shadow Manager task

ce1f1bf

Add the mandatory Manager counterpart for the state-based Shadow Hand reorientation task while preserving the Direct task's action, observation, reward, reset, goal, and termination contracts. Register matching benchmark criteria and backend presets.

Defer Handover MDP imports

b3070ae

Use the standard lazy-export initializer so importing the Handover Manager configuration does not preload USD modules before SimulationApp starts. Add a subprocess regression check for the pre-Kit import contract.

Handle initial Shadow Manager reset

17d0628

Avoid reading the RL reset buffer before ManagerBasedRLEnv creates it during initial environment reset. Preserve the one-step autoreset suppression only when that buffer exists.

Defer Shadow Manager action imports

55b6ad5

Preserve Direct's same-step terminal-action observation with a reset snapshot instead of a custom runtime action class. This keeps Manager configuration imports free of USD modules before SimulationApp starts and covers the pre-Kit contract.

Preserve actions across manual resets

7a0024b

Select the reset-action snapshot while episode length is zero so initial, terminal, and explicit user resets all match Direct's same-step action observation until the next transition.

Preserve Handover actions across resets

0e57f84

Keep both hands' terminal raw actions in Manager reset observations so the single-agent observation contract continues to match Direct during initial, manual, and same-step autoresets.

Remove redundant dexterous settings

7da09aa

Drop an unnecessary Allegro OVPhysX alias and update the Handover Newton documentation now that both workflows have converged with the shared settings.

Add Shadow OpenAI and camera Manager tasks

3ed8cf1

Provide Manager counterparts for the OpenAI feed-forward, recurrent, and camera reorientation variants. Preserve Direct observation, noise, action, reward, reset, and termination contracts while exposing every physics and rendering preset.

Order camera Manager test imports

2d6d284

Align Allegro Manager with Direct task

c4c9e96

Use the same environment and agent contracts so the two workflows are directly comparable. Existing Manager checkpoints must be retrained against the new observation and reward contract.

Cover camera variant validation gaps

36bb19f

Validate all supported physics and renderer preset pairs, exclude playback and benchmark registrations from training collection, and complete the public camera feature term documentation.

Merge remote-tracking branch 'origin/develop' into jichuanh/dexterous…

51b2833

…-env-convergence-codex

Add Menagerie dexterous hand presets

4bceaed

Add explicit Menagerie asset configurations and select them for PhysX and Newton while retaining compatible legacy assets for OVPhysX. Preserve the task action contracts across joint namespaces and tighten benchmark validation.

Merge remote-tracking branch 'origin/develop' into jichuanh/dexterous…

6f124ec

…-env-convergence-codex

Fix benchmark test import order

faeb4c0

Fix codespell finding in the value-parity test docstring

5d7ddbb

hujc7 mentioned this pull request Jul 17, 2026

[Task Clean-up] Dexterous Part 8/8: Move the warp Direct task variants to isaaclab_tasks_experimental #6582

Draft

hujc7 changed the title ~~[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (split into Parts 1-11)~~ [DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference) Jul 17, 2026

hujc7 requested review from AntoineRichard, StafaH, daniela-hase, marcodiiga, matthewtrepte, ooctipus and pascal-roth as code owners July 17, 2026 23:32

hujc7 requested a review from a team July 17, 2026 23:32

hujc7 added 9 commits July 17, 2026 16:35

Merge remote-tracking branch 'upstream/develop' into jichuanh/dextero…

2d2f495

…us-env-convergence-codex

Drop the OVPhysX changelog fragment superseded by the merged Part 2

289efcb

The OVPhysX runtime fixes merged to develop via their part PR with its own fragment; the lump's duplicate fragment would double the changelog entry.

hujc7 requested review from aserifi, david-cao-mueller, huidongc and rubengrandia as code owners July 18, 2026 11:10

hujc7 added 4 commits July 18, 2026 04:50

Remove the orphaned object_away_from_robot termination term

1732493

Its only consumer was the removed legacy ReorientObjectEnvCfg; neither the Direct env nor the parity manager configurations use an object-to-robot distance criterion.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference)#6324

[DO-NOT-MERGE][Task Clean-up] Dexterous: lumped validation branch (series reference)#6324
hujc7 wants to merge 128 commits into
isaac-sim:developfrom
hujc7:jichuanh/dexterous-env-convergence-codex

hujc7 commented Jul 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

hujc7 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review Map

Summary

Direct vs manager training reference (qualifying campaign, seed 42)

Phase-2 requirement fulfillment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hujc7 commented Jul 2, 2026 •

edited

Loading