Custom Dataset Generation for Training

[shreshtashetty]

Hi,

I have gone through the README.md for the training and evaluation as well as ros_deployment using the provided datasets and pretrained model checkpoints. Now, I am trying to generate my own custom dataset per my use case for training X-Mobility from scratch. I performed EDA for the provided datasets, x_mobility_isaac_sim_nav2_100k and x_mobility_isaac_sim_random_160k so that I have a foundation to base my new dataset on. Based on that, I have the following doubts and clarifications about the dataset fields:

---

1. Each .pqt file has the following columns:

'acqtime', 'camera_intrinsic_matrix', 'camera_extrinsic_matrix',
'perspective_semantic_image', 'perspective_semantic_image_shape',
'perspective_semantic_image_labels', 'camera_image', 'driving_command',
'route', 'ego_speed', 'path', 'route_poses', 'goal_id',
'semantic_labels'.

Is my deduction correct that 'acqtime', 'camera_intrinsic_matrix', 'camera_extrinsic_matrix' and 'goal_id' are NOT being used by either of the 2 models? If they are being used in the code, can you please clarify how data for 'acqtime', 'camera_extrinsic_matrix' and 'goal_id' are obtained?

---

2. 'route' vs 'route_poses'
   I see that the fields 'route' and 'route_poses' are both used in the ActionPolicy code. In the provided .pqt files, each entry of the 'route' field is a .jpg image encoded as a bytestring, which when decoded gives a 64x64 binary image representing the route being followed. The 'route_poses' field, on the other hand, is an array of len = 40, which is later converted into an array of coordinate pairs (((x1, y1), (x2, y2)), ((x2, y2), (x3, y3))...etc.) in the code. While going through action_policy.py, I noticed that 'route_vectors' which has been obtained from 'route_poses' is being used in the forward pass (line 174), while 'route' is being used in the inference function in (line 210). With 'route' and 'route_poses' being 2 different types of data (bytestring vs array), how is one used for the forward pass and the other used for inference?

---

3. How is the 'path' field generated?
   I understand that 'path' is the final output of the X-Mobility model during inference. But I need to generate data entries for the 'path' field for training. I am not sure how to go about this. I assumed there would be some similarities between the 'route_poses' field and the 'path' field because the former is being used to create the final path in x_mobility_navigator.py, line 276. In the .pqt file entries, however, 'path' is an array of len = 10 whereas 'route_poses' is of len = 40 and the values don't seem to be having any correlation with each other. Could you provide any guidance on how to generate the 'path' field for training purposes, and why 'route_poses' and 'path' have different lengths and formats?

---

4. Can transfer learning be performed in this framework? Specifically, can I use pre-trained checkpoints and continue training the model with my custom dataset?

---

Apologies in advance for the lengthy post.

[nv-liuw]

Hi @shreshtashetty , great questions.

1.You're correct—those fields are not used. We originally included them for flexibility when generating the datasets, but we ended up using only the ones shown in the table below:

NOTE: Speed should be R^1 in this table, as we only use linear speed for now.

2. For route encoding, we experimented with two strategies:

• route/route_map: Rasterizes the route into an image.
• route_poses: Contains the route as a sequence of pose coordinates.

With various experiments, route_poses showed much better results, so we deprecated route_map. Both the forward pass and inference now use route_poses, and we should unify the variable names for consistency.

3. Path comes from the teacher policy, but it is not actively used since the model directly outputs command velocities. If you don't have access to that imitation target, you can disable the path head in the policy network.

Regarding the difference between route and path:

• route provides global guidance using only environment priors.
• path is more local and accounts for collision avoidance based on environment dynamics.

4. We haven't specifically explored transfer learning with imitation learning, but we do use the pre-trained X-Mobility model for RL fine-tuning, as described in COMPASS. This approach has worked well across various embodiments.

[rvnandwani]

@nv-liuw I have a quick question, so when we are training for world model understanding and giving random actions, what is the significance of action command in that case i.e. why is action command in input for training the world model when those are nothing but random actions.

[nv-liuw]

@rvnandwani The world model is designed to learn the dynamics of the environment—that is, how the world state evolves in response to different actions. Even when actions are random, they serve two key purposes:

• Ease of Data Collection: Random actions can be efficiently generated using Isaac Sim’s Synthetic Data Generator, enabling large-scale data collection without the need for a teacher policy.
• Comprehensive Action Coverage: Random actions ensure broad exploration of the action space, helping the model learn generalizable dynamics across a wide range of possible scenarios.

[superjuices]

Hi, @nv-liuw
when generating custom data, I found that speed has only one dimension (the field is ego_speed), but the paper mentioned that it is two-dimensional, which is linear speed and angular speed. In addition, action command represents the linear speed and angular speed of three axes. So what is the difference between the speed in ego speed and action command?

[nv-liuw]

@superjuices Thanks for pointing it out, the arXiv paper needs update. We started experiments with 2-dimensional speeds including linear forward and angular z, but turns out encoding linear forward speed only produced better results according to the ablation studies.

Regarding the difference between ego_speed and action_command, action_command is the commands sent to controller, and the ego_speed is the actual measured robot states. We define the action_command as 6-dimension velocity is to keep the interface flexible to different embodiment types.

[shreshtashetty]

@nv-liuw Thanks for the detailed answers!

After trying to generate the dataset based on your responses, I have a few follow-up questions for clarification:

1. Synchronization of ROS 2 Topics:
   To obtain one entry in the .pqt file, it seems that ISAAC ROS topics like /chassis_odom, /cmd_vel, /plan, /local_plan, and /front_stereo_camera/left/image_raw need to be synchronized.

   • How is this synchronization achieved?
   • Since /cmd_vel (used for driving commands) doesn’t have timestamps, how is it synchronized with the other topics?

2. Structure of the .pqt file:

   • Am I correct in understanding that for each camera image, there must be corresponding values for the other fields as well?
   • For topics with lower publishing frequencies than the camera image (e.g., /cmd_vel, /plan) new values of these fields won't always be available every time a new camera image is obtained. In such a case, are the previous values of these fields used as current values (duplicated entries)?
     (I’ve noticed duplicated entries for driving_command and route_poses in the datasets on X-Mobility’s page.)

3. SDG and Semantic Data Generation:

   • Is the SDG used to generate the semantic image data?
   • In our case, we have an RTX 4090 system, and while the SDG provides semantic data dumped into a local folder, it significantly slows down the simulation, causing a lag between sending the driving command and it being executed in simulation. It also significantly hampers the running of Nav2.
   • Does this issue occur on your end as well? If so, do you have any suggestions to mitigate the slowdown?

[nv-liuw]

@shreshtashetty Great questions!

We add timestamps to all ROS topics, including cmd_vel, by creating a custom message type that wraps the original Twist.msg with an additional timestamp field. A republisher node then subscribes to /cmd_vel and publishes the new timestamped message.

Once all messages have timestamps, we synchronize them by sorting based on the timestamp. We sample image data at a fixed frequency (e.g., 2 Hz), and for each image, we search forward within a specified time window to find the first corresponding /cmd_vel, /plan, /odom, etc.—assuming these actions are reactions to that image observation. If the latency between the image and any associated action message exceeds a threshold, we discard the sample.

For the Nav2 dataset, we didn't use SDG. Instead, we recorded a ros2 bag with the required topics and applied the synchronization logic above during post-processing to construct the dataset.

[shreshtashetty]

@nv-liuw, thanks for the reply! This clears up a lot. I just want to confirm one last thing.

In your answer, you mentioned:

"For the Nav2 dataset, we didn't use SDG. Instead, we recorded a ros2 bag with the required topics and applied the synchronization logic above during post-processing to construct the dataset."

Does this mean that ISAAC Sim's Replicator is used to publish semantic data to a ROS2 topic instead of saving it to a local folder?

We tried using Replicator to publish semantic data to a ROS2 topic (by adding a ROS2 Camera Helper Node and configuring it further). We managed to get the 'perspective_semantic_image' and 'perspective_semantic_image_shape' fields from the semantic segmentation topic, but we couldn’t retrieve the 'perspective_semantic_image_labels' (the class number to class name mapping) field. SDG used to provide this directly in the local folder. How can we obtain this field over a ROS2 topic?

[huihuaNvidia2023]

@shreshtashetty thanks for the question.

Here is the screenshot of the camera node configuration, you have to specify the topic name.

[shreshtashetty]

@huihuaNvidia2023 Thanks, that worked!