Skip to content

Commit 8608fbc

Browse files
authored
Fixes template docs and restructure imitation learning docs (#3283)
# Description There have been two places where the template documentation has been placed (under Developers Guide and Workthrough), this PR unifies them into a new structure (see image below). Furthermore, the imitation learning examples were missing a grouping, this PR introduces a structure similar to the section about reinforcement learning Also some general docs fixes are included. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots <img width="588" height="1323" alt="image" src="https://github.com/user-attachments/assets/a17a6328-a9e3-44cd-a299-6aa62e0e422e" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
1 parent a80fa53 commit 8608fbc

File tree

16 files changed

+89
-131
lines changed

16 files changed

+89
-131
lines changed

docs/index.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,7 @@ Table of Contents
8989
:titlesonly:
9090

9191
source/setup/quickstart
92+
source/overview/own-project/index
9293
source/setup/walkthrough/index
9394
source/tutorials/index
9495
source/how-to/index
@@ -104,8 +105,7 @@ Table of Contents
104105
source/overview/core-concepts/index
105106
source/overview/environments
106107
source/overview/reinforcement-learning/index
107-
source/overview/teleop_imitation
108-
source/overview/augmented_imitation
108+
source/overview/imitation-learning/index
109109
source/overview/showroom
110110
source/overview/simple_agents
111111

docs/source/how-to/write_articulation_cfg.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ properties of an :class:`~assets.Articulation` in Isaac Lab.
1818
We will use the Cartpole example to demonstrate how to create an :class:`~assets.ArticulationCfg`.
1919
The Cartpole is a simple robot that consists of a cart with a pole attached to it. The cart
2020
is free to move along a rail, and the pole is free to rotate about the cart. The file for this configuration example is
21-
``source/isaaclab_assets/isaaclab_assets/robots/cartpole.py``.
21+
``source/isaaclab_assets/isaaclab_assets/robots/cartpole.py``.
2222

2323
.. dropdown:: Code for Cartpole configuration
2424
:icon: code

docs/source/overview/developer-guide/index.rst

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,3 @@ using VSCode.
1313
VS Code <vs_code>
1414
repo_structure
1515
development
16-
template

docs/source/overview/augmented_imitation.rst renamed to docs/source/overview/imitation-learning/augmented_imitation.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ Example usage for the cube stacking task:
8383
Running Cosmos for Visual Augmentation
8484
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
8585

86-
After converting the demonstrations to MP4 format, you can use a `Cosmos <https://github.com/NVIDIA/Cosmos?tab=readme-ov-file>`_ model to visually augment the videos. Follow the Cosmos documentation for details on the augmentation process. Visual augmentation can include changes to lighting, textures, backgrounds, and other visual elements while preserving the essential task-relevant features.
86+
After converting the demonstrations to MP4 format, you can use a `Cosmos`_ model to visually augment the videos. Follow the Cosmos documentation for details on the augmentation process. Visual augmentation can include changes to lighting, textures, backgrounds, and other visual elements while preserving the essential task-relevant features.
8787

8888
We use the RGB, depth and shaded segmentation videos from the previous step as input to the Cosmos model as seen below:
8989

@@ -99,7 +99,7 @@ We provide an example augmentation output from `Cosmos Transfer1 <https://github
9999
:align: center
100100
:alt: Cosmos Transfer1 augmentation output
101101

102-
We recommend using the `Cosmos Transfer1 <https://github.com/nvidia-cosmos/cosmos-transfer1/tree/e4055e39ee9c53165e85275bdab84ed20909714a>`_ model for visual augmentation as we found it to produce the best results in the form of a highly diverse dataset with a wide range of visual variations. You can refer to the installation instructions `here <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/INSTALL.md#environment-setup>`_, the checkpoint download instructions `here <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#download-checkpoints>`_ and `this example <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#example-2-multimodal-control>`_ for reference on how to use Transfer1 for this usecase. We further recommend the following settings to be used with the Transfer1 model for this task:
102+
We recommend using the `Cosmos Transfer1 <https://github.com/nvidia-cosmos/cosmos-transfer1/tree/e4055e39ee9c53165e85275bdab84ed20909714a>`_ model for visual augmentation as we found it to produce the best results in the form of a highly diverse dataset with a wide range of visual variations. You can refer to the `installation instructions <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/INSTALL.md#environment-setup>`_, the `checkpoint download instructions <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#download-checkpoints>`_ and `this example <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#example-2-multimodal-control>`_ for reference on how to use Transfer1 for this usecase. We further recommend the following settings to be used with the Transfer1 model for this task:
103103

104104
.. rubric:: Hyperparameters
105105

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
Imitation Learning
2+
==================
3+
4+
In this section, we show existing scripts for running imitation learning
5+
with Isaac Lab.
6+
7+
.. toctree::
8+
:maxdepth: 1
9+
10+
augmented_imitation
11+
teleop_imitation

docs/source/overview/teleop_imitation.rst renamed to docs/source/overview/imitation-learning/teleop_imitation.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -413,7 +413,7 @@ Annotations denote the end of a subtask. For the pick and place task, this means
413413
Each demo requires a single annotation between the first and second subtask of the right arm. This annotation ("S" button press) should be done when the right robot arm finishes the "idle" subtask and begins to
414414
move towards the target object. An example of a correct annotation is shown below:
415415

416-
.. figure:: ../_static/tasks/manipulation/gr-1_pick_place_annotation.jpg
416+
.. figure:: ../../_static/tasks/manipulation/gr-1_pick_place_annotation.jpg
417417
:width: 100%
418418
:align: center
419419

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
.. _own-project:
2+
3+
Build your Own Project or Task
4+
==============================
5+
6+
To get started, first create a new project or task with the template generator :ref:`template-generator`.
7+
For more detailed information on how your project is structured, see :ref:`project-structure`.
8+
9+
.. toctree::
10+
:maxdepth: 1
11+
:titlesonly:
12+
13+
template
14+
project_structure
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
.. _project-structure:
2+
3+
4+
Project Structure
5+
=================
6+
7+
There are four nested structures you need to be aware of when working in the direct workflow with an Isaac Lab template
8+
project: the **Project**, the **Extension**, the **Modules**, and the **Task**.
9+
10+
.. figure:: ../../_static/setup/walkthrough_project_setup.svg
11+
:align: center
12+
:figwidth: 100%
13+
:alt: The structure of the isaac lab template project.
14+
15+
The **Project** is the root directory of the generated template. It contains the source and scripts directories, as well as
16+
a ``README.md`` file. When we created the template, we named the project *IsaacLabTutorial* and this defined the root directory
17+
of a git repository. If you examine the project root with hidden files visible you will see a number of files defining
18+
the behavior of the project with respect to git. The ``scripts`` directory contains the ``train.py`` and ``play.py`` scripts for the
19+
various RL libraries you chose when generating the template, while the source directory contains the python packages for the project.
20+
21+
The **Extension** is the name of the python package we installed via pip. By default, the template generates a project
22+
with a single extension of the same name. A project can have multiple extensions, and so they are kept in a common ``source``
23+
directory. Traditional python packages are defined by the presence of a ``pyproject.toml`` file that describes the package
24+
metadata, but packages using Isaac Lab must also be Isaac Sim extensions and so require a ``config`` directory and an accompanying
25+
``extension.toml`` file that describes the metadata needed by the Isaac Sim extension manager. Finally, because the template
26+
is intended to be installed via pip, it needs a ``setup.py`` file to complete the setup procedure using the ``extension.toml``
27+
config. A project can have multiple extensions, as evidenced by the Isaac Lab repository itself!
28+
29+
The **Modules** are what actually gets loaded by Isaac Lab to run training (the meat of the code). By default, the template
30+
generates an extension with a single module that is named the same as the project. The structure of the various sub-modules
31+
in the extension is what determines the ``entry_point`` for an environment in Isaac Lab. This is why our template project needed
32+
to be installed before we could call ``train.py``: the path to the necessary components to run the task needed to be exposed
33+
to python for Isaac Lab to find them.
34+
35+
Finally, the **Task** is the heart of the direct workflow. By default, the template generates a single task with the same name
36+
as the project. The environment and configuration files are stored here, as well as placeholder, RL library dependent ``agents``.
37+
Critically, note the contents of the ``__init__.py``! Specifically, the ``gym.register`` function needs to be called at least once
38+
before an environment and task can be used with the Isaac Lab ``train.py`` and ``play.py`` scripts.
39+
This function should be included in one of the module ``__init__.py`` files so it is called at installation. The path to
40+
this init file is what defines the entry point for the task!
41+
42+
For the template, ``gym.register`` is called within ``isaac_lab_tutorial/source/isaac_lab_tutorial/isaac_lab_tutorial/tasks/direct/isaac_lab_tutorial/__init__.py``.
43+
The repeated name is a consequence of needing default names for the template, but now we can see the structure of the project.
44+
**Project**/source/**Extension**/**Module**/tasks/direct/**Task**/__init__.py

docs/source/overview/developer-guide/template.rst renamed to docs/source/overview/own-project/template.rst

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
.. _template-generator:
22

3-
Build your Own Project or Task
4-
==============================
3+
4+
Create new project or task
5+
==========================
56

67
Traditionally, building new projects that utilize Isaac Lab's features required creating your own
78
extensions within the Isaac Lab repository. However, this approach can obscure project visibility and

docs/source/setup/walkthrough/concepts_env_design.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ reference frame is what defines the world.
2424

2525
"Above" the world in structure is the **Sim**\ ulation and the **App**\ lication. The **Application** is "the thing responsible for
2626
everything else": It governs all resource management as well as launching and destroying the simulation when we are done with it.
27-
When we :ref:`launched training with the template<walkthrough_project_setup>`, the window that appears with the viewport of cartpoles
27+
When we :ref:`launched training with the template<template-generator>`, the window that appears with the viewport of cartpoles
2828
training is the Application window. The application is not defined by the GUI however, and even when running in headless mode all
2929
simulations have an application that governs them.
3030

0 commit comments

Comments
 (0)