Skip to content
Open
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
931802a
Added local envionment for Go2
clayton-littlejohn Jan 26, 2026
58bb6a4
Add generic local asset resolution system for offline training
clayton-littlejohn Jan 27, 2026
94f10b4
minor refactors for offline mode
clayton-littlejohn Jan 27, 2026
d335006
Added offline asset documentation
clayton-littlejohn Jan 27, 2026
14b7e50
offline mode to /play
clayton-littlejohn Jan 27, 2026
e98441f
offline to shared command-line arguments
clayton-littlejohn Jan 27, 2026
94d82d5
offline mode for all (sb3, SKRL, RL Games, Sim2Sim)
clayton-littlejohn Jan 27, 2026
a52df7c
better asset downloader feedback and resolver
clayton-littlejohn Jan 27, 2026
52a1dcd
documentation clarifications
clayton-littlejohn Jan 27, 2026
e45aae2
minor cleanup
clayton-littlejohn Jan 28, 2026
b792c50
gitignored
clayton-littlejohn Jan 28, 2026
3a27bcb
more useful comment
clayton-littlejohn Jan 28, 2026
325d991
formatting
clayton-littlejohn Jan 28, 2026
32059f9
more formatting
clayton-littlejohn Jan 28, 2026
7bf8d81
Merge branch 'isaac-sim:main' into offline_mode
clayton-littlejohn Jan 28, 2026
3f403e1
I have added my name to the CONTRIBUTORS.md
clayton-littlejohn Jan 28, 2026
9f7b50b
linting and formatting
clayton-littlejohn Jan 28, 2026
ed00a5c
import directly from asset_resolver
clayton-littlejohn Jan 28, 2026
1adaa33
Optional[str] for better compatibility
clayton-littlejohn Jan 28, 2026
e872600
forgot some Optional[str]
clayton-littlejohn Jan 28, 2026
09b5bb4
Keep the str | None syntax
clayton-littlejohn Jan 28, 2026
e067e30
Merge branch 'isaac-sim:main' into offline_mode
clayton-littlejohn Jan 28, 2026
f7253d3
Added global offline to AppLauncher, propogates to all
clayton-littlejohn Jan 28, 2026
aa252fc
Documentation updates
clayton-littlejohn Jan 28, 2026
d52286f
clarification
clayton-littlejohn Jan 28, 2026
6c05a80
nucleus mirror patches & offline strict/permissive flag
clayton-littlejohn Jan 29, 2026
f71d0d4
Merge branch 'isaac-sim:main' into offline_mode
clayton-littlejohn Feb 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,6 @@ tests/
# TacSL sensor
**/tactile_record/*
**/gelsight_r15_data/*

# Downloaded assets
offline_assets/*
1 change: 1 addition & 0 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ Guidelines for modifications:
* Cathy Y. Li
* Cheng-Rong Lai
* Chenyu Yang
* Clayton Littlejohn
* Connor Smith
* CY (Chien-Ying) Chen
* David Yang
Expand Down
235 changes: 235 additions & 0 deletions scripts/offline_setup/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
# Isaac Lab Offline Training Setup
> Complete guide for training Isaac Lab environments offline using locally downloaded assets.

## 🎯 Overview
#### The offline training system enables you to train Isaac Lab environments without internet connectivity by using locally downloaded assets. This system:
- ✅ Works with any robot - No hardcoded paths needed
- ✅ Single flag - Just add --offline to your training command
- ✅ Automatic fallback - Uses Nucleus if local asset is missing
- ✅ Maintains structure - Mirrors Nucleus directory organization locally

## 📦 Requirements
- Isaac Lab installed and working
- Isaac Sim 5.0 or later
- 2-20 GB free disk space (depending on assets downloaded)
- Internet connection for initial asset download

## 🚀 Quick Start
### 1. Download essential assets (one-time, `all` ~30 GB)
#### Assets download to the `~/IsaacLab/offline_assets` directory: `cd ~/IsaacLab`
```
./isaaclab.sh -p scripts/offline_setup/download_assets.py \
--categories all
```
#### _Alternative Note: Category fields can be specified separately_
```
./isaaclab.sh -p scripts/offline_setup/download_assets.py \
--categories Robots --subset Unitree
```
### 2. Train completely offline with any robot via the `--offline` flag (also works with `/play`)
#### Supported for: `rl_games`, `rsl_rl`, `sb3`, `skrl`, and `sim2transfer`
```
./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py \
--task Isaac-Velocity-Flat-Unitree-Go2-v0 \
--num_envs 128 \
--offline

./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/play.py \
--task Isaac-Velocity-Flat-Unitree-Go2-v0 \
--num_envs 128 \
--checkpoint logs/rsl_rl/<robot>_flat/<timestamp>/model_<num>.pt \
--video \
--video_length 1000 \
--offline
```
#### _Note: For offline training, assets that cannot be found in `offline_assets` will attempted to be fetched from the [Nucleus Server](https://docs.omniverse.nvidia.com/nucleus/latest/index.html)._

## 📁 Asset Layout
#### Offline assets are organized to mirror Nucleus (`ISAAC_NUCLEUS_DIR` & `ISAACLAB_NUCLEUS_DIR`) under the `offline_assets` directory, meaning that no code changes are required for offline running! We flatten `Isaac/IsaacLab/` to just the category names (`Robots/`, `Controllers/`, etc.) for cleaner local structure. This happens in `asset_resolver.py`, where the resolver maintains a 1:1 mapping between Nucleus and local storage.

```
IsaacLab/
├── source/isaaclab/isaaclab/utils/
│ └── asset_resolver.py # Core resolver
├── scripts/setup/
│ └── download_assets.py # Asset downloader
└── offline_assets/
├── ActuatorNets/
├── Controllers/
├── Environments/ # Ground planes
│ └── Grid/
│ └── default_environment.usd
├── Materials/ # Textures and HDRs
│ └── Textures/
│ └── Skies/
├── Mimic/
├── Policies/
├── Props/ # Markers and objects
│ └── UIElements/
│ └── arrow_x.usd
└── Robots/ # Robot USD files
├── BostonDynamics/
│ └── spot/
│ └── spot.usd
└── Unitree/
├── Go2/
│ └── go2.usd
└── H1/
└── h1.usd
```









### Proposal

Add offline training support to Isaac Lab that enables training environments without internet connectivity. The feature should automatically redirect asset paths from Nucleus/S3 servers to local storage while maintaining the same directory structure, requiring zero configuration changes to existing environments.

Core capabilities:
- One-time asset download script that mirrors Nucleus directory structure locally
- Automatic path resolution from Nucleus URLs to local filesystem
- Single `--offline` flag for all training scripts
- Graceful fallback to Nucleus if local asset is missing
- Works with any robot and environment without hardcoded paths

### Motivation

**Current Problem:**
Training Isaac Lab environments requires constant internet connectivity to load assets from Nucleus/S3 servers. This creates several critical issues:

1. **Airgapped Environments**: Impossible to train in secure/classified facilities that prohibit internet access
2. **Network Reliability**: Training fails or becomes extremely slow on unstable networks
3. **Reproducibility**: Repeated downloads slow iteration and depend on external server availability
4. **Development Workflow**: Researchers waste time waiting for assets during rapid prototyping
5. **DGX Spark is Rooted**: I should be able to take my Spark anywhere regardless of internet connectivity and live out Isaac Lab and Isaac Sim to it's fullest

**User Story:**
"I'm always frustrated when I need to train in an airgapped lab environment or on an unstable network connection. I have to create custom configs with hardcoded local paths for each robot, which results in unmaintainable code duplication and can break when I switch robots or update Isaac Lab. I would like to take Isaac Lab on the go and quickly demo anything in an independent localized ecosystem without nit-picking configurations and dealing with asset management."

**Core User Needs:**
- Train without internet connectivity
- Avoid hardcoded paths in environment configs
- Work seamlessly with any robot
- Maintain same configs for both online and offline modes

### Alternatives

**Alternative 1: Hardcoded Local Paths** (Current workaround)
```python
# Separate config file per robot
UNITREE_GO2_CFG = ArticulationCfg(
spawn=sim_utils.UsdFileCfg(
usd_path=f"{ISAACLAB_NUCLEUS_DIR}/Robots/Unitree/Go2/go2.usd"
```
**Problems:**
- ❌ Requires separate config for each robot
- ❌ ~30 lines of boilerplate per robot
- ❌ Breaks when Isaac Lab updates
- ❌ Not maintainable

**Alternative 2: Environment Variables**
```bash
export ISAAC_ASSETS_DIR="/local/path"
```
**Problems:**
- ❌ Doesn't work with S3 URLs
- ❌ Requires Isaac Sim core changes
- ❌ Can't have fallback to Nucleus
- ❌ Not transparent to users

**Alternative 3: Manual Asset Copying**
Copy assets manually and update configs.
**Problems:**
- ❌ Error-prone manual process
- ❌ No directory structure guidelines
- ❌ Still requires config changes

**Proposed Solution Benefits:**
- ✅ Zero code or config changes
- ✅ Works with all robots and environments automatically
- ✅ Single flag: `--offline` for all training environments (`rl_games`, `rsl_rl`, `sb3`, `skrl`, and `sim2transfer`)
- ✅ Automatic fallback
- ✅ Easy `offline_asset` plug and play following Nucleus structure
- ✅ 90% code reduction

### Build Info

- Isaac Lab Version: main branch (as of January 2026)
- Isaac Sim Version: 5.1.0

### Additional Context

**Use Cases:**
1. **Defense/Aerospace**: Training in classified airgapped facilities
2. **Remote Locations**: Field robotics research with limited connectivity
3. **Development**: Rapid iteration without network delays and firewall interruptions
4. **CI/CD**: Reproducible builds without external dependencies
5. **Workshops/Tutorials**: Teaching without relying on conference WiFi (i.e. localized everything on DGX Spark)

**Technical Approach:**
The implementation uses a dual-layer strategy:
- **Monkey patching**: Intercepts asset loads at spawn config instantiation (90% coverage)
- **Config patching**: Explicitly modifies pre-loaded configs (10% coverage)

This ensures ~100% asset coverage without modifying environment configs.

**Expected Directory Structure:**
```
IsaacLab/
├── source/isaaclab/isaaclab/utils/
│ └── asset_resolver.py # Core resolver
├── scripts/setup/
│ └── download_assets.py # Asset downloader
└── offline_assets/
├── ActuatorNets/...
├── Controllers/...
├── Environments/ # Ground planes
│ └── Grid/
│ └── default_environment.usd
├── Materials/ # Textures and HDRs
│ └── Textures/
│ └── Skies/
├── Mimic/...
├── Plocies/...
├── Props/ # Markers and objects
│ └── UIElements/
│ └── arrow_x.usd
└── Robots/ # Robot USD files
├── BostonDynamics/
│ └── spot/
│ └── spot.usd
└── Unitree/
├── Go2/
│ └── go2.usd
└── H1/
└── h1.usd
```

Dynamically pulls and mirrors Nucleus structure for seamless path resolution.

### Checklist

- [x] I have checked that there is no similar issue in the repo (**required**)

### Acceptance Criteria

- [x] Asset download script that mirrors Nucleus directory structure to local storage (offline_assets)
- [x] Automatic path resolver that redirects Nucleus URLs to local filesystem
- [x] Optional `--offline` flag added to all training scripts (RSL-RL, SB3, SKRL, RL Games)
- [x] Monkey patching of Isaac Lab spawn configs (UsdFileCfg, GroundPlaneCfg, PreviewSurfaceCfg)
- [x] Config patching for pre-loaded environment configs
- [x] Graceful fallback to Nucleus for missing assets
- [x] Support for versioned Nucleus URLs (e.g., `/Assets/Isaac/5.1/...`)
- [x] Documentation including setup guide, usage examples, and troubleshooting
- [x] Works with any robot without hardcoded paths
- [x] Zero breaking changes - existing code continues to work
- [x] Manual testing completed across multiple robots (Go2, H1, ANYmal)
- [x] Verification in complete offline mode (no internet connectivity)

**Definition of Done:**
A user can download all, or select assets by running `./isaaclab.sh -p scripts/offline_setup/download_assets.py --categories all`, then train any robot completely offline by simply adding `--offline` to their training command, with no code or config changes required.
Loading