Skip to content

Simple Digital Twin in Omniverse and Unreal Engine 5

Jump to bottom
HAL edited this page May 16, 2024 · 25 revisions

Overview

This guide walks you through setting up a simple digital twin of the Elephant Robotics MyArm300 or Cobot280 using NVIDIA Omniverse and the Unreal Engine on Windows 11. In this configuration, you will need:

  • A Win11 computer with NVIDIA Omniverse and Isaac Sim installed. This will be the host computer.
  • A Win11 computer with Unreal Engine 5.3, NVIDIA Omniverse, Isaac Sim, and the Unreal Omniverse 5.3 Connector installed. This will be the client computer
  • An Elephant Robotics arm

All machines must be on the same wifi network.

Task Summary: Omniverse Setup Nucleus Setup Isaac Sim Scene Setup Robot Files Robot Descriptor Python Scripts Arm Code Unreal Engine Connector

Omniverse Setup

The main engine driving this digital twin application is NVIDIA Omniverse.

  • Step 1: Download Omniverse
    • If you don’t already have one, you will need to create an NVIDIA Developer account
  • Step 2: Create Local Nucleus
    • a) local nucleus
      • You can leave the local data path as the default
      • Create a username and password. This can be anything you want, but you will need the username and password later, so make sure to record it.
      • For this guide, username is admin and password is admin
      • We now have a nucleus folder called “localhost”. This is where we will store the USD scene file to open in Isaac Sim and Unreal Engine.

Add localhost Nucleus

* b) remote nucleus
	* To connect to this nucleus from another machine, you will need to add another user. To do this, open `localhost:3080` in your browser on the host computer. 
* First, click the enable sharing button to allow remote connections
	* Make sure to click to allow creating new accounts
	* You will probably have to allow network stuff 
* Next, navigate to the “Exchange” tab of the Omniverse Launcher and install “Nucleus Navigator”. 
* In the Nucleus Navigator, right click on the “localhost” directory, and click on “Administration”. You will not see this option if you are not using the Nucleus Navigator app installed from Exchange. 
* In the Administration menu, click on “Users”, then “Add User”
* Create a new user. Here, we will name the new user “client”
* Click on the client in the Users menu, click “Grant Admin Access”, then click “Invite User”. 
	* This will copy a link to your clipboard. Paste it into a browser. On this page, set a password for the new user.

We now have a remotely accessible Omniverse Nucleus with two users, one for the host and one for the client. We will connect the client later. For now, we will move on to the Issac Sim setup on the host machine.

Isaac Sim

Isaac Sim is a robotics simulator within the Omniverse ecosystem. Before we can set up our stage in Issac Sim, we need to download some files.

To make the remote access simpler, we are going to save all the necessary files locally on each machine. Create a folder on the C drive named “Omniverse_Files”.

Download the Elephant Robotics ROS repository to the C:\Omniverse_Files folder. Download vr_input.py and vr_ur.py from this repo and add to the Omniverse_Files folder. Download the robot_data folder from this repo and add to the Omniverse_Files folder

We can now set the stage in Isaac Sim. Launch Isaac Sim from the Omniverse Launcher.

Isaac Sim Viewport control shortcuts:

  • Pan: Middle Mouse Button (MMB)
  • Orbit: Right Mouse Button (RMB)
  • Zoom: Scroll Wheel
  • Focus on selected object: F

Go to Isaac Utils > URDF Importer. Under import, set the “Input File” to C:\Omniverse_Files\mycobot_ros\mycobot_description\urdf\<name_of_arm>\<name_of_arm>.urdf. Output Directory should be “(same as source)” If it is not checked, check “Create Physics Scene” in the Input Options. We need to update the velocity values in the URDF file. Open the URDF you are going to import in a text editor. Use ctrl + f to Find and Replace velocity = “0”with velocity = “1” Import the URDF into Isaac Sim Add a ground plane in from Create > Physics > Ground Plane If there is not already a light on the stage, add one from Create > Light > Dome Light. Arm Articulation Root Setup Expand the “firefighter” robot arm object, and click on “joint1” Find the “Articulation Root” under “Physics” in the “Properties” panel, and remove it. Click on “firefighter”, and click “Add” under the “Properties” panel, and Add > Physics > Articulation Root. Create Robot Description File Open Isaac Utils > Lula Robot Description Editor. This is where we will set up and save the robot_description.yaml file. Open Isaac Utils > Workflows > Articulation Inspector. This will help us identify the location of joints. Press the play button on the far left to start the simulation, and under “Select Articulation” on the two windows, select “/World/firefighter”. In the Lula Robot Description Editor, in the Command Panel, set all the joints to “Active Joint” Create Collision Spheres for each joint In the Selection Panel, under Select Link, start by selecting “joint1” In the Stage panel, expand “joint1”. Notice that the name of the joint for this object is “joint2_to_joint1”. In the Articulation Inspector, you can use the sliders to move “joint2_to_joint1” so that you can identify where it is. Set the joint back to 0 in the Articulation Inspector. Back in the Lula Robot Description Editor, go to the Link Sphere Editor. Expand the “Add Sphere”, “Scale Spheres in Link”, and “Clear Spheres in Link” dropdowns. Add a sphere to “joint1”, using the Viewport gizmos and “Scale Spheres in Link” button to position. The “Scaling Factor” of the “Scale Spheres in Link” can be set to “0.9” to incrementally shrink, or “1.1” to incrementally grow. The sphere should be as close to the rotation point of the link, and as close to the geometry size as possible (as shown in the image). Repeat this process for all of the joints in the “Select Link” field. If you are using the Articulation Inspector to identify joints, make sure you reset them all to 0 before you place a sphere. If your robot is not in the default position, but the joint positions all appear as “1”, click “REFRESH” under “Update controllers” with a joint selected in the Stage panel. Be careful to not move the joints too much, or you may break the physics simulation. You should end up with something like this: In the Lula Robot Description Editor, set the Output file under the “Export Robot Description File” to “C:\Omniverse_Files\robot_data\robot_description.yaml” Double check in the file explorer that the file saved. Stop the simulation Add Python Scripts Click on “firefighter” in the Stage view, and click “Add > Python Scripting” in the Properties panel. Add “vr_input.py” and “vr_ur.py” Open vr_input.py in a text editor to update files paths Set the PATH_BASE variable on line 40 to “C:\Omniverse_Files” Set the end effector joint name on line 85 to the name of the final joint on the robot. This will be the joint object in the scene view that doesn’t have a joint child object when expanded. Here, this is “joint6_flange” On line 101, update the yaml file path of robot_description_path to the name we saved from the Lula Robot Description Editor, robot_description.yaml On line 102, update the URDF file of urdf_path to C:\Omniverse_Files\mycobot_ros\mycobot_description\urdf\<name_of_arm>\<name_of_arm>.urdf. This should be the same URDF file you imported. Setting the IK target. The IK target will be the object that the arm follows. This can either be the end effector, or a disconnect object. For this guide, we will use a target cube. Create a cube with Create > Mesh > Cube. Drag the Cube up to :firefighter” to make it a child object. Scale and place the cube so that it is roughly in the size and position of the end effector. Rename this Cube “Target”. If you are using the end effector, open vr_input.py in a text editor and change the word “Target” on line 142 to the name of the end effector object. Save USD Scene in the localhost Nucleus file. Reload the USD file from the Content browser. Press Play on the simulation, select Target, and translate it down the Z axis slightly to confirm that the IK is working. If the console has an “IK failed” warning, try moving the Target closer to the robot. If moving the Target when the simulation is running does not move the arm, and there is no “IK failed” warning, double check all paths and file names.

Unreal Setup

Next, we are going to create and join a Live Session to view our Isaac Sim scene in the Unreal Engine from a client computer. On a second computer, make sure you have Unreal Engine 5.3 and the Omniverse Unreal 5.3 Connector installed.

As with the host computer, create a folder on the C drive named “Omniverse_Files”.

Download the Elephant Robotics ROS repository to the C:\Omniverse_Files folder When the URDF is imported on the host, it creates a folder for the USD file and the materials in place from where the URDF was imported. You will need to copy this folder from the host to the client so that it has the same file path. You will also need to open the URDF file in a text editor and change the velocity values from 0 to 1. Download vr_input.py and vr_ur.py from this repo and add to the Omniverse_Files folder. Download the robot_data folder from this repo and add to the Omniverse_Files folder

Now, we can begin to connect to the host. In the Omniverse Nucleus Navigator, click on “Connect to a Server” Enter the IP address of your host computer. Log in with the credentials of the client user you created earlier. Launch Unreal Engine 5.3 and create a new Blank project. If you have the Omniverse Unreal Connector installed, you should see an Omniverse icon in the top left corner. Click on the Omniverse Icon, then Add Server. Enter the IP address of the host server. In the Content Browser or Content Drawer, you should see a folder called “Omniverse”. In this folder, navigate to where you saved the USD on the host. On the host computer, start a Live Session in Isaac Sim from the icon on the top right. On the client, join the Live Session by clicking the Omniverse icon > Live Session. Now, if you start the simulation on the Host, the arms shown on both machines should move in sync.

Arm Setup

On the arm Download link_<name_of_arm>.py from this repo pip install zmq If this does not work, you may need to sudo apt-get update && sudo apt-get upgrade

In Host Isaac Sim Add vr_ur.py to the “firefighter” object. Update the IP Address on line 36 to the IP address of the arm you wish to connect to. Save and reload the scene.

Now, you should be able to run link_<name_of_arm>.py on the arm, and start the simulation on the host Isaac Sim to control the physical arm.

Debugging Tips

Arm:

sudo netstat -ltnp kill -9 [PID]

To find IP Address: ifconfig

python link.py python mirror_bot.py

ls cd Desktop/ ls cd py_scripts python link.py python mirror_bot.py

need to use ros2 terminal or source it ls cd colcon_ws/ ros2 launch myarm_300 mycobot_follow.launch.py ros2 launch myarm_300 slider_control.launch.py

Clone this wiki locally