Skip to content

Adds DisplayPort cable insertion envs#6398

Open
shauryadNv wants to merge 62 commits into
isaac-sim:developfrom
shauryadNv:shauryad/dp_cable_ship
Open

Adds DisplayPort cable insertion envs#6398
shauryadNv wants to merge 62 commits into
isaac-sim:developfrom
shauryadNv:shauryad/dp_cable_ship

Conversation

@shauryadNv

@shauryadNv shauryadNv commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Description

Added envs to train and run inference for the new task of DisplayPort cable insertion task with the Flexiv Rizon 4s robot. Supports multiple env variants based on policy design choices such as operating in joint space vs task space and the observation space used.
Added documentation for a tutorial on training a DisplayPort cable insertion policy with Flexiv Rizon 4s.

Type of change

  • New feature (non-breaking change which adds functionality)
  • Documentation update

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@shauryadNv shauryadNv self-assigned this Jul 7, 2026
@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label Jul 7, 2026
@greptile-apps

greptile-apps Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds DisplayPort cable insertion environments and tooling. The main changes are:

  • New DisplayPort plug/socket assets and diagnostic scripts.
  • New basic, joint-space, task-space, and ROS inference environment configs.
  • New reset, grasp, observation, reward, termination, action-delay, and success-logging helpers.
  • Gym registrations for the new DisplayPort task variants.

Confidence Score: 4/5

The task-space training path needs a config fix before merging.

  • The new environments are mostly additive and the main import and reset wiring is consistent.
  • The task-space reset curriculum is disabled by its own default values.
  • That can make task-space training run without the seeded insertion starts described by the config.

source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/cable_insertion/config/displayport_rizon_4s/task_space_env_cfg.py

Important Files Changed

Filename Overview
source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/cable_insertion/config/displayport_rizon_4s/task_space_env_cfg.py Defines the task-space DisplayPort insertion environment; the at-goal curriculum is configured but disabled.
source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/cable_insertion/config/displayport_rizon_4s/joint_pos_env_cfg.py Defines the joint-position DisplayPort insertion environment and wires robot, reset, action, and termination settings.
source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/mdp/events.py Adds generic object grasp reset logic and plug-at-goal curriculum reset logic.
source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/mdp/terminations.py Adds plug drop and plug orientation termination terms for the insertion environments.
source/isaaclab_tasks/isaaclab_tasks/contrib/deploy/cable_insertion/insertion_env.py Adds success metric logging for DisplayPort insertion rollouts.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[DisplayPort Gym task id] --> B[DisplayportInsertionEnv]
  B --> C[Environment config]
  C --> D[Reset events]
  D --> E[reset_scene_to_default]
  E --> F[randomize_socket_pose]
  F --> G[reset_plug_at_goal_curriculum]
  G --> H[set_robot_to_object_grasp_pose]
  C --> I[Policy observations and actions]
  C --> J[Rewards and terminations]
  B --> K[Success metric logging]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[DisplayPort Gym task id] --> B[DisplayportInsertionEnv]
  B --> C[Environment config]
  C --> D[Reset events]
  D --> E[reset_scene_to_default]
  E --> F[randomize_socket_pose]
  F --> G[reset_plug_at_goal_curriculum]
  G --> H[set_robot_to_object_grasp_pose]
  C --> I[Policy observations and actions]
  C --> J[Rewards and terminations]
  B --> K[Success metric logging]
Loading

Reviews (1): Last reviewed commit: "Relocate DisplayPort cable_insertion und..." | Re-trigger Greptile

Comment on lines +258 to +259
"at_goal_prob": 0.0,
"at_goal_prob_final": 0.0,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 At-Goal Curriculum Disabled

The task-space config documents and wires an at-goal curriculum, but both probabilities are 0.0. reset_plug_at_goal_curriculum only enters the at-goal branch when the current probability is positive, so every reset uses the normal approach-pose randomization and the task-space policy never receives the seeded insertion starts this config describes.

Suggested change
"at_goal_prob": 0.0,
"at_goal_prob_final": 0.0,
"at_goal_prob": 0.8,
"at_goal_prob_final": 0.0,

@shauryadNv
shauryadNv requested a review from ashwinvkNV July 7, 2026 20:59
curiep and others added 22 commits July 7, 2026 14:16
The Rizon 4s cable-insertion task was failing every episode at reset:
the IK + open-loop close sequence put the gripper next to the plug
instead of around it, the fingers closed on empty space, and the
plug fell to the floor (or twisted away when a finger clipped its
edge).

Root cause was a geometry mismatch between the un-flattened plug USD
and the IK target. The un-flattened plug has its visible/collision
``Mesh`` offset ~3 cm from the rigid-body origin
(``bbox_center_local = (0.0312, 0.0009, -0.0107)`` in body frame).
``set_robot_to_object_grasp_pose`` snapped the plug body to the
gripper-achieved pose, but the achieved pose was a function of the
*body* origin, not the visible centre, so the fingertips landed
~3 cm from the actual plug.

Switch to the flattened plug + socket USDs (``flattened/`` directory)
where ``bbox_center == (0, 0, 0)`` and the visible mesh coincides
with the rigid-body origin. With that, the grasp parameters reduce
to the gripper tool length only:

* ``grasp_offset = (0, 0, -0.13)`` -- only backs ``link7`` off by
  the Grav tool length so the fingertips land at the plug body
  origin (which is now also the visible centre).
* ``grasp_rot_offset = (-0.707, 0.707, 0, 0)`` (unchanged) keeps
  ``link7`` pointing straight down with the gripper finger axis
  perpendicular to the plug's long body Y axis.
* Plug + socket use identity ``init_state.rot``; the flattened USDs
  already align the long axis with body Y.
* Plug spawns 5 cm above the socket so the policy has descent room.
* ``hand_close_width = 0.0`` matches the gear-assembly small-gear
  close target. ``-0.155`` (the joint hard limit) was tried and
  ejected the plug because the drive over-commanded the fingers.

Also extend ``set_robot_to_object_grasp_pose``:

* Snap the held object to the achieved hand pose after IK so any
  IK residual error or geometry offset is washed out before the
  gripper closes.
* One-shot ``[GRASP-DBG]`` log dumps the initial pose, the IK
  target, the achieved hand pose and the snapped object pose --
  invaluable for diagnosing future grasp regressions.
The Flexiv Rizon 4s in the NVIDIA Hubble Lab is wall-mounted with the
arm rotated about the negative X-axis. Update the ROS inference env
config so the simulator's reset state matches the physical robot:

* Set the home joint pose to the lab's joint convention.
* Set the robot base orientation to the wall-mount quaternion so the
  arm hangs the same way in sim and in the real cell.

This keeps the policy roll-out and the live deployment in the same
robot frame, which is required for the policy obs (computed relative
to ``base_link``) to be valid at inference time.
Three small scripts under ``scripts/tools/`` to diagnose and repair
USD assets whose visible/collision geometry is offset from the
rigid-body origin -- the failure mode that broke the Rizon 4s
cable-insertion grasp.

* ``inspect_cable_usd.py``: standalone Pixar-USD walker. Lists xform
  ops, ``RigidBodyAPI`` / ``MassAPI`` / ``CollisionAPI`` markers and
  per-mesh ``bbox_center_local`` / ``bbox_size``. Use this from any
  shell that already has ``pxr`` on PYTHONPATH (e.g. a Kit Python).

* ``inspect_cable_usd_kit.py``: same walker run inside an IsaacLab
  ``AppLauncher`` context so ``pxr`` is available without extra setup.
  This is what we actually used during debugging from the IsaacLab
  conda env.

* ``flatten_cable_usd.py``: bake the chain of intermediate xforms
  down into the leaf mesh's points, then reset all xform ops on the
  rigid-body and its children to identity. The result is a USD whose
  visible mesh is centered at the rigid-body origin -- exactly the
  invariant the IK-based grasp event assumes. Supports an optional
  ``--rotate-z-deg`` knob for assets whose body-local pose needs an
  additional rotation baked in.

Both inspectors print the bounding-box centre in body-local frame so
``grasp_offset`` can be set deterministically from the asset rather
than tuned by trial and error.
The ``-Play-v0`` variants for Rizon 4s and UR10e gear assembly
omitted ``rsl_rl_cfg_entry_point`` in their gym registration, so
``play.py`` could not resolve the agent config and would error out
with ``KeyError`` before ever loading a checkpoint.

Add the entry-point so ``./isaaclab.sh -p .../play.py --task
Isaac-Deploy-GearAssembly-...-Play-v0`` works end-to-end. Also pick
up a pre-commit whitespace cleanup in ``play_hardcoded.py``.
Required for training: env cfg references these files but they were
excluded by the *.usd gitignore rule.
shauryadNv added 20 commits July 7, 2026 14:16
…eam rebase.

Upstream moved deploy tasks from manager_based/manipulation/deploy to
contrib/deploy; update module paths and scripts so DP env registration and
imports match the new layout while keeping existing Isaac-Deploy task IDs.
Remove debug scripts, asset tooling, passive drop-test env, legacy GB300
configs, unused assets, and disabled sysid action plumbing. Keep only the
displayport_rizon_4s joint/task/ROS env variants and the two SDF USD assets
used by the training envs.
@shauryadNv
shauryadNv force-pushed the shauryad/dp_cable_ship branch from 7ccabce to 2765083 Compare July 7, 2026 21:16
@shauryadNv
shauryadNv requested a review from a team July 17, 2026 23:38
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants