Linear program object placement #307
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
## Summary Closedloop GN1.5 observed low SR in multi-episode rollout, esp in parallel-env where more contacts are being introduced / more episodes are being observed. ## Detailed description ### Static manip - Issue: At the beginning of episode, hands have close-open motions in recorded trajectories. Given microwave joint is not stiff enough, small deviations during first few inferences cause the door closed by mistake. And this closed door is hard to pull with static GR1, causing it to fail the task. [Screencast from 12-02-2025 03:16:36 PM.webm](https://github.com/user-attachments/assets/da06de60-8f01-47e7-ae26-a48e08cb523f) - Fix: a. Shorten task_episode_length_s to introduce more frequent resets once the door is closed. Tradeoff is introducing more episodes. b. Also tried with shorter `action_horizon` but ended up getting worse SR. My hypothesis is that it's hard for VLA to tell from visuals/states whether the door is closed to 0.2 (success) vs 0.21 (fail). > 16 -- Metrics: {'success_rate': 0.605, 'door_moved_rate': 0.955, 'num_episodes': 200} > 8 -- Metrics: {'success_rate': 0.225, 'door_moved_rate': 0.615, 'num_episodes': 200} > 1 -- Metrics: {'success_rate': 0.0, 'door_moved_rate': 0.985, 'num_episodes': 200} c. Switching to CPU PhyX does not solve above issues. So keep it on GPU for faster parallelization (in theory). ### Loco manip - Issue: After each reset, the left arm tends to have fast motions and the box is tilted. Also observed significant penetration among fingers. See 00:15 VS 0:30 for 5 parallel env closedloop in below video. [Screencast from 11-25-2025 03:42:36 PM.webm](https://github.com/user-attachments/assets/c4934817-65fa-412f-a88c-af143d25d7c2) - Fix: switch to CPU phyX, keep the policy on GPU Arms open first and G1 starts moving, box is placed with expected pose. [Screencast from 12-02-2025 10:15:59 PM.webm](https://github.com/user-attachments/assets/4a02e6cd-7baf-441b-8c0f-7146051e5c9a) ### Minor fixes Update doc on cmds & metrics.
## Summary expose env spacing parameter
## Summary Modify docs to show that this is manual annotation
## Summary Add ground plane and light objects
## Summary Users might want to modify env cfg components such as sim config. This lets them do it.
## Summary Move our CI infra to public runners ## Detailed description - As part of our open-source release, we can no longer run on internal infra. - This MR moves our runners to public runners. - I also took the chance to refactor and modularize the workflow file.
## Summary Update link to the docs in README.md to the new public location. ## Detailed description - Docs url has changed now that the repo is public.
## Summary Fixes an issue that our tags requesting for public CI were incorrect. ## Detailed description - Corrects the tag `[gpu]` -> `[self-hosted, gpu]`
## Summary Revert to mapping the whole repo in the dev docker. ## Detailed description - Previously we changed to mapping only specific folders in the repo. - This was done for docker build speed (I believe?) - The issue is that we want (even if occasionally) to work on all folders in the repo, within the dev docker. - This reverts to mapping the whole repo.
## Summary Re-enables pre-commit in CI. ## Detailed description - During the refactoring and switch to public CI, `pre-commit` was broken. - This MR fixes it.
## Summary Language prompts are fetched from Task's data member, populated from ArenaEnv creation. ## Detailed description - `task_description` is automatically populated into atomic task, and users have the freedom to overwrite it when instantiating the task class - `Policy` sets its `task_description` attribute thru a setter func - `Policy_runner.py` connects the `task_description` from Task to `task_description` setter in `Policy` - GR00T consumes description either thru `task_description` data member or `policy_config`
## Summary All example environments.py are repackaged into isaaclab_arena_environments ## Reason In prep for multi-task evaluation, as we may introduce more example envs.
## Summary Move the multi-versioned docs now that we have multiple version of Isaac Lab Arena. ## Detailed description - Means users can read the version of the docs that shipped the release that they're using. <img width="1130" height="625" alt="version_sidebar" src="https://github.com/user-attachments/assets/b06372cd-9bed-4a1d-99b8-9480c279ebb4" />
## Summary Tear down simulation app func could be useful both in core & tests. Prep for it to be consumed. ## Detailed description - Add USD `get_new_stage()` to jupyter notebook example, resolving issue where USDs from previous run are not removed - Add optional `suppress_exception `to let exception raised by default, or ignored in tests - Move this func to `isaaclab_utils`
## Summary Fix git ownership issues in deployment pipeline. ## Detailed description - Multi-version docs now require git during documentation build. - This revealed git ownership issues in the page deployment pipeline (previously seen in our pre-merge pipeline) - This MR applies the same fix that's used in pre-merge.
## Summary Update object library to use ISAAC_NUCLEUS_DIR prefix for YCB object usd paths
## Summary
Refactor `mimic_env_cfg` building logic in `arena_env_builder`.
## Detailed description
- What was the reason for the change?
- Originally we need to maintain a list of embodiment_names in each
task's MimicEnvCfg, given its single_arm or dual_arm. It is not
efficient and scalable.
- What has been changed?
- creates a new enum class `MimicArmMode` to represent the arm mode for the mimic environment: can select from
["single_arm", "dual_arm", "left", "right"]
- assigns a property of 'mimic_arm_mode' to embodiment_base
- changes task.get_mimic_env_cfg() method's input from 'embodiment_name'
to 'mimic_arm_mode'
- refactors the SubTaskConfigs configuration logic in each MimicEnvCfg
based on embodiment.mimic_arm_mode
- What is the impact of this change?
- all existing embodiments and tasks with MimicEnvCfg.
## Summary Add settings file
## Summary `RigidObjectSet` inherited from `Object` to enable users provide a list of assets, and sim app spawn each `env_id` with one obj from this set. ## Detailed description - Introduced `RigidObjectSet(Obejct)` class for handle rigid body object set construction - The order of each obj in the set to load in each env_id could be configured as following func args order, or being random. - Introduced `--object_set` in `kitchen_pick_and_place.py` cli to allow spawning for each env_id - Added tests for empty/single/multi object sets & checker each env_id's usd is referenced in expected sequence ## TODO - Pipe clean & verify other task-centered obj metrics/ attributes access (Done in test) - Introduce this concept in other sample envs & multi-task eval ## Note - Naming to `set` instead of `collection` is to differentiate what [`RigidBodyCollection`](https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/assets/rigid_object_collection/rigid_object_collection_data.py) from IssacLab provides. In our use case, we need to spawn 1 obj from N objs, where `RigidBodyCollection` API is to spawn all N objs for each id. - MultiAssetSpawnerCfg for articulated objs will be tricky (/buggy) as PhyX APIs require it has the same joint prim path. It puts too much constraints on what could be added into the set - [MultiAssetSpawnerCfg](https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/sim/spawners/wrappers/wrappers_cfg.py#L16) for rigid objs require the same type of collision meshes, as written in Lab's doc. <img width="720" height="295" alt="image" src="https://github.com/user-attachments/assets/71983e83-d586-427b-a1dd-3eb047be817f" />
## Summary This PR adds initial support for composite sequential tasks via the SequentialTaskBase class. The SequentialTaskBase class takes a list of atomic tasks (TaskBase) and automatically assembles them into a composite task with unified termination/event configs. Adds: 1. SequentialTaskBase class 2. Test case to validate class methods 3. Test case with example task (sequential open door task) to validate unified success check and events 4. Two new functions in `isaac_arena/utils/configclass.py` to perform config transformation and duplicate checking
## Summary Fixes a typo which caused a misnaming of EEFs in the Mimic Env Configs of tasks. The name of the eefs was being set as an Enum instead of the value of the Enum. This caused data generation to fail using our existing datasets.
## Summary Implement `OpenDoorTask` and `CloseDoorTask` inherited from `RotateRevoluteJointTask` ## Detailed description - Generalize the task to common articulated objects with revolute joint. E.g. Cabinet door; Window panel (rotate outward or inward relative to the fixed frame using a hinge); Scissor blade at the pivot pin. - Open vs Close Door task differ in terminations & reset events, but share the same underlying logics handling revolute joint. - Add `is_closed` member function to `Openable` affordance, using threshold to decide either open or close. Basically use it as a bi-state object. ## TODO - Test with other articulated objects, than the overly-used microwave
## Summary Simplify device registry and add a retargeter registry
## Summary Add warning to docs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Short description of the change (max 50 chars)
Detailed description