tutorials/6.expandation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extend tasks based on VLABench"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.Add new entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Introduce to entity logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VLABench has integrated numerous 3D objects, each of which has been further annotated and processed. All the `obj`, `stl` and corresponding `xml` files are stored in [`assets/obj/meshes`](../VLABench/assets/obj/meshes) directory, objects of the same category are stored in the same directory. All the processed 3d assets can be downloaded by running script `scripts/download_assets.py`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To facilitate the use of these 3D objects, we have encapsulated these assets into different classes, with base class [`Entity`](../VLABench/tasks/components/entity.py). The `Entity`s are divided into two main classes: `GraspedObject` and `Container`(Same conception to Receptacle). For specific categories, you can write a class to inherit from `GraspedObject` or `Container`, such as the class [`Toy`](../VLABench/tasks/components/specific_entities/common_entities.py). Each entity class will be registered by decorator [`@register.add_entity`](../VLABench/utils/register.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the registered entities will be recorded to [`name2class_xml`](../VLABench/configs/constant.py), which maps the entity name to its entity class and the file path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We annotated the `xml` files by some rules:\n",
    "1. The entity that belongs to `GraspedObject`: add `<default class=\"grasppoint\"> <site type=\"sphere\" size=\"0.01\" group=\"4\" rgba=\"0 0 1 1\"/> </default>` to xml and annote the target points to grasp.\n",
    "2. The entity that belongs to `Container`: add `<default class=\"placepoint\"> <site type=\"sphere\" size=\"0.01\" group=\"2\" rgba=\"0 0 1 0\" /> </default>` and annotate the point for placement. \n",
    "3. The entity that belongs to `Container`: add `<default class=\"keypoint\"> <site type=\"sphere\" size=\"0.01\" group=\"3\" rgba=\"1 0 0 1\" /> </default>`and annotate the bounding box point for containing judgement.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Extend new 3d assets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to setup a new tasks with new 3d assets, register the assets as the following steps:\n",
    "1. Prepare the 3D assets with `obj` format. You can try [Blender](https://www.blender.org/) or [Trimesh](https://pypi.org/project/trimesh/)(just run `pip install trimesh`) for pre-processing.\n",
    "2. Generate the `xml` files from `obj` assets. We use obj2mjcf for this step, an example command is like `obj2mjcf --verbose --obj-dir /your/path/to/object  --compile-model --save-mjcf --decompose`.\n",
    "3. Make proper annotation. Such as the `grasppoint` for `GraspedObject` and `placepoint` for `Container`.  \n",
    "4. If this object is a new entity category that needs more instance method, inherit from our existed classes and write some new codes.\n",
    "5. Register the new entity in [`name2class_xml`](../VLABench/configs/constant.py) and [register](../VLABench/utils/register.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Add new tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Add new config_manager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[ConfigManager](../VLABench/tasks/config_manager.py) is a management class used for generating task randomness. It's one of the key components of [Task](../VLABench/tasks/dm_task.py) class.This class is responsible for randomly selecting target objects from predefined options, randomly generating object combinations, selecting object initialization positions within a certain range, and generating the initialization parameters for the conditions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a config manager class and register it by [`@register.add_config_manager`](../VLABench/utils/register.py). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Add new task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[LM4ManipBaseTask](../VLABench/tasks/dm_task.py) is the core class responsible for generating various scene and object instances from the config, modifying simulation parameters, updating the information required for the environment, and maintaining the values needed for metric calculation. Any newly defined tasks in VLABench inherit from `LM4ManipTask` and can be modified with minimal changes as needed, such as the calculation method for metrics, additional processing before and after each step, and the refined expert trajectory sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new task class and register it by ['@register.add_task'](../VLABench/utils/register.py)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}