Fix/rheo dockerfile

workflows/rheo/README.md

@@ -22,6 +22,7 @@ The workflow provides an end-to-end development pipeline for Physical AI in clin
 - [🚀 Quick Start](#-quick-start)
 - [🏠 Environment Requirements](#-environment-requirements)
 - [⚡ Running Workflows](#-running-workflows)
+- [🖥 Viewing Isaac Sim](#-viewing-isaac-sim)
 - [🛠 Troubleshooting](#-troubleshooting)
 - [📚 Attribution & Citation](#-attribution-and-citation)
 
@@ -105,6 +106,61 @@ hf download nvidia/GR00T-N1.5-RL-Rheo-AssembleTrocar --local-dir $HOME/models/GR
 
 ## ⚡ Running Workflows
 
+### 🖥 Viewing Isaac Sim
+
+Isaac Sim does **not** open when you only enter the Docker shell (`[RHEO] $`). You must run a runner script (for example `observe_runner.py`). The Rheo Docker image sets `HEADLESS=1` by default, so you need an explicit override to show the native GUI window.
+
+#### Step 1 — Host display setup (run once per desktop session)
+
+On the machine with a monitor (not a plain SSH session without X11), allow Docker to use your display:
+
+```bash
+xhost +local:docker
+echo $DISPLAY   # should print :0 or :1
+```
+
+#### Step 2 — Run with the Isaac Sim GUI window
+
+Prefix the `run_docker.sh` command with `HEADLESS=0`, pass `--enable_cameras`, and **do not** pass `--headless`. The window appears after the environment finishes loading (first launch can take several minutes).
+
+```bash
+cd i4h-workflows
+
+HEADLESS=0 ./workflows/rheo/docker/run_docker.sh \
+  python scripts/simulation/examples/observe_runner.py \
+    --num_steps 15000 \
+    --enable_cameras \
+    observe_object \
+    --object surgical_tray_no_lid \
+    --embodiment g1_wbc_pink
+```
+
+Apply the same pattern to other runners (policy evaluation, teleop, and so on): `HEADLESS=0` on the host, no `--headless` on the script.
+
+#### Browser stream (WebRTC alternative)
+
+If you are on SSH, have no local display, or the GUI crashes on startup, use headless mode with WebRTC and open `http://localhost:8080` in a browser:
+
+```bash
+./workflows/rheo/docker/run_docker.sh \
+  python scripts/simulation/examples/observe_runner.py \
+    --headless \
+    --num_steps 15000 \
+    --enable_cameras \
+    --webrtc_cam \
+    --webrtc_host 0.0.0.0 \
+    --webrtc_port 8080 \
+    --webrtc_fps 30 \
+    observe_object \
+    --object surgical_tray_no_lid \
+    --embodiment g1_wbc_pink
+```
+
+| Mode | Host | Script flags | Where you see the scene |
+| --- | --- | --- | --- |
+| GUI window | `HEADLESS=0`, `xhost +local:docker` | `--enable_cameras` (no `--headless`) | Isaac Sim / Kit window |
+| WebRTC | default | `--headless --webrtc_cam ...` | Browser at `http://localhost:8080` |
+
 ### Running Agent Workflow
 
 #### Run Physical Agent
@@ -126,17 +182,15 @@ This is a minimal example to run the physical agent in Isaac Sim, in which the c
     --embodiment g1_wbc_joint
 ```
 
-OR simply observe the surgical tray:
+OR simply observe the surgical tray — see [Viewing Isaac Sim](#-viewing-isaac-sim) for GUI vs WebRTC. GUI example:
 
 ```bash
-./workflows/rheo/docker/run_docker.sh -g1.6 \
+xhost +local:docker
+
+HEADLESS=0 ./workflows/rheo/docker/run_docker.sh \
   python scripts/simulation/examples/observe_runner.py \
     --num_steps 15000 \
     --enable_cameras \
-    --webrtc_cam \
-    --webrtc_host 0.0.0.0 \
-    --webrtc_port 8080 \
-    --webrtc_fps 30 \
     observe_object \
     --object surgical_tray_no_lid \
     --embodiment g1_wbc_pink
@@ -176,7 +230,8 @@ hf download nvidia/GR00T-N1.5-RL-Rheo-AssembleTrocar --local-dir $HOME/models/GR
 
 Notes:
 
-- **`--headless`** and **`--enable_cameras`** are Isaac Lab / AppLauncher options (pass them if you need cameras/rendering).
+- **GUI vs headless**: The Docker image sets `HEADLESS=1`. Use `HEADLESS=0` on the host and omit `--headless` to open the Isaac Sim window; see [Viewing Isaac Sim](#-viewing-isaac-sim). Use `--headless` (and optionally `--webrtc_cam`) for SSH or browser viewing.
+- **`--enable_cameras`** is required when you need camera observations or rendering.
 - **`--rl_ckpt`** automatically applies runtime patches to `Gr00tPolicy` to ensure consistency with RL post-training modifications made by RLinf. **If you are not using RL-trained checkpoints, DO NOT pass this flag.** The patch modifies:
   - **Eagle input padding**: Pads `eagle_input_ids` and `eagle_attention_mask` to a fixed length of 850
   - **Dropout removal**: Replaces all dropout layers with `nn.Identity()` for deterministic inference
@@ -386,6 +441,14 @@ After fine-tuning or reinforcement learning, you can evaluate the success rate o
 
 ## 🛠 Troubleshooting
 
+- **Isaac Sim window does not appear**:
+  - Entering the container alone (`./workflows/rheo/docker/run_docker.sh` with no command) only opens a shell; run a runner script such as `observe_runner.py`.
+  - The image defaults to `HEADLESS=1`. Set `HEADLESS=0` before `run_docker.sh` and do not pass `--headless` on the script.
+  - On the host, run `xhost +local:docker` and confirm `echo $DISPLAY` is set (`:0` or `:1`).
+  - The GUI may take several minutes to appear on first scene load; wait for `[INFO] Environment created:` in the logs.
+  - If the process segfaults during RTX startup (common on some RTX 50-series GPUs), use the WebRTC path in [Viewing Isaac Sim](#-viewing-isaac-sim) instead.
+  - Rheo auto-injects conservative Kit/render settings for Blackwell (RTX 50-series) and unsets `DISPLAY` in Docker unless `HEADLESS=0` and `RHEO_KEEP_DISPLAY=1`. Opt out with `RHEO_DISABLE_BLACKWELL_RENDER_PATCH=1`. Clear shader cache: `rm -rf ~/.cache/ov ~/.nv/ComputeCache`.
+
 - **Resetting the VLM Agent**:
   - After changing the agent configuration or a system restart, you need to reset the UI container to reload the agent configurations. Otherwise, it could lead to unexpected behavior by using the default agent in the VLM-Surgical-Agent-Framework repository.
   - To reset the UI container, you can run the following command: `./tools/env_setup/install_vlm_surgical_agent_fx.sh -r`

workflows/rheo/docker/Dockerfile.x86

@@ -24,7 +24,8 @@ ARG INSTALL_GROOT=false
 ARG WORKDIR="/workspaces"
 ENV WORKDIR=${WORKDIR}
 WORKDIR "${WORKDIR}"
-ENV PYTHONPATH=${WORKDIR}/workflows/rheo/scripts:${WORKDIR}/workflows/rheo/scripts/simulation/rl
+ENV PYTHONPATH=${WORKDIR}/workflows/rheo:${WORKDIR}/workflows/rheo/scripts:${WORKDIR}/workflows/rheo/scripts/simulation/rl
+ENV HEADLESS=1
 
 USER root
 
@@ -40,7 +41,10 @@ RUN apt-get update && apt-get install -y \
   cmake \
   sudo \
   ffmpeg \
-  python3-pip
+  python3-pip \
+  pkg-config \
+  libfreetype6-dev \
+  libpng-dev
 
 # Skip pip upgrade - not needed in Docker and conflicts with PEP 668 in Python 3.12
 # RUN pip3 install --upgrade pip
@@ -52,40 +56,89 @@ COPY ./third_party/IsaacLab ${WORKDIR}/third_party/IsaacLab
 ENV ISAACLAB_PATH=${WORKDIR}/third_party/IsaacLab
 ENV TERM=xterm
 RUN ln -s /isaac-sim/ ${WORKDIR}/third_party/IsaacLab/_isaac_sim
-# Install IsaacLab dependencies
-RUN for DIR in ${WORKDIR}/third_party/IsaacLab/source/isaaclab*/; do pip install --no-deps -e "$DIR" --break-system-packages; done
 # Set permissions for Isaac Sim directories that need write access
 RUN chmod 777 -R /isaac-sim/
-# Pre-install flatdict into kit Python: its setup.py imports pkg_resources which
-# is absent from pip's isolated build env, causing isaaclab core install to fail.
-RUN /isaac-sim/python.sh -m pip install --no-build-isolation flatdict==4.0.1
-# Install isaaclab
-RUN ${ISAACLAB_PATH}/isaaclab.sh -i
-# Verify isaaclab core was installed (isaaclab.sh silently swallows failures)
-RUN /isaac-sim/python.sh -c "import isaaclab; print(isaaclab.__file__)"
+# Build deps and apt packages required by Isaac Lab extensions (see IsaacLab/docker/Dockerfile.base)
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install toml
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p ${ISAACLAB_PATH}/tools/install_deps.py apt ${ISAACLAB_PATH}/source
+# flatdict must be pre-installed without build isolation (pkg_resources / setuptools issue).
+# Do NOT install setuptools/wheel here — it corrupts pip's vendored packaging in the kit.
+# Pre-install isaaclab deps in stages with pinned wheels. Use --no-deps for packages that
+# re-trigger pip matplotlib backtracking (pytransform3d, dex-retargeting) after deps exist.
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
+    flatdict==4.0.1 \
+    "numpy<2" \
+    matplotlib==3.10.3 \
+    scipy \
+    lxml \
+    pyyaml \
+    anytree \
+    trimesh \
+    nlopt==2.7.1 \
+    pin==2.7.0 \
+    qpsolvers==4.8.1 \
+    quadprog \
+    loop-rate-limiters
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary --no-deps \
+    pytransform3d==3.14.4
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary --no-deps \
+    pin-pink==3.1.0 && \
+    ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
+    daqp==0.7.2
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary --no-deps \
+    dex-retargeting==0.4.6
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
+    prettytable==3.3.0 \
+    gymnasium==1.2.1 \
+    pillow==11.3.0 \
+    hidapi==0.14.0.post2 \
+    starlette==0.49.1 \
+    "pyglet<2"
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary "onnx>=1.18.0"
+# Do NOT pip install warp-lang — Isaac Sim 5.1 ships Warp via omni.warp-1.8.2. A newer
+# warp-lang from PyPI shadows the kit copy and breaks wp.types.array / wp.context at startup.
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary einops
+# isaaclab_mimic runtime deps (install editable packages with --no-deps below)
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
+    tomli ipywidgets==8.1.5
+# isaaclab / isaaclab_tasks / isaaclab_rl runtime deps skipped by --no-deps editable installs
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
+    h5py \
+    hydra-core \
+    tensorboard \
+    moviepy \
+    numba \
+    "protobuf>=4.25.8,!=5.26.0" \
+    "packaging<24"
+# transformers omitted here — bulk install segfaults kit Python; GR00T stage installs it when needed.
+# Install Isaac Lab core and extensions with --no-deps (runtime deps pre-installed above).
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --no-deps -e "${ISAACLAB_PATH}/source/isaaclab" && \
+    /isaac-sim/python.sh -c "import isaaclab; print(isaaclab.__file__)" && \
+    for DIR in ${WORKDIR}/third_party/IsaacLab/source/isaaclab_*/; do \
+      ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --no-deps -e "$DIR" || exit 1; \
+    done && \
+    /isaac-sim/python.sh -c "import isaaclab; import isaaclab_mimic; print(isaaclab.__file__)"
 
 # Patch for osqp
-RUN if python -c "import qpsolvers; print(qpsolvers.available_solvers)" | grep -q "osqp"; then \
+RUN if /isaac-sim/python.sh -c "import qpsolvers; print(qpsolvers.available_solvers)" | grep -q "osqp"; then \
         echo "OSQP is installed."; \
     else \
         echo "OSQP missing, installing..."; \
-        /isaac-sim/python.sh -m pip install --upgrade pip; \
         /isaac-sim/python.sh -m pip install qpsolvers==4.8.1; \
     fi
 
-# Install pip dependencies
-RUN /isaac-sim/python.sh -m pip install --upgrade pip && \
-  /isaac-sim/python.sh -m pip install \
+# Workflow pip deps in one layer. Do not pin typing_extensions or install jupyter here —
+# they install standalone `packaging` and break pip's vendored copy in kit Python.
+# WebRTC stack (aiortc/av/aiohttp) — loaded lazily by runners when --webrtc_cam is set.
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation --prefer-binary \
   pytest \
-  jupyter \
-  typing_extensions==4.12.2 \
-  onnxruntime
-
-# lwlabs deps
-RUN /isaac-sim/python.sh -m pip install --upgrade pip && \
-  /isaac-sim/python.sh -m pip install \
+  onnxruntime \
   vuer[all] \
-  lightwheel-sdk
+  lightwheel-sdk \
+  aiohttp \
+  av==12.3.0 \
+  aiortc==1.10.1 && \
+  ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --force-reinstall --no-deps pip==24.3.1
 
 ENV LW_API_ENDPOINT="https://api-dev.lightwheel.net"
 
@@ -191,7 +244,8 @@ RUN if [ "$INSTALL_GROOT" = "true" ]; then \
 # Set CUDA 12.8 library path permanently for runtime (prepend to existing path)
 ENV LD_LIBRARY_PATH=/usr/local/cuda-12.8/lib64:/usr/local/cuda/lib64:/usr/lib/x86_64-linux-gnu
 
-################################
+
+#################################
 # Install IsaacLab Arena
 ################################
 COPY ./third_party/IsaacLab-Arena ${WORKDIR}/third_party/IsaacLab-Arena
@@ -207,7 +261,7 @@ COPY ./third_party/RLinf ${WORKDIR}/third_party/RLinf
 ################################
 COPY ./workflows/rheo ${WORKDIR}/workflows/rheo
 
-RUN echo "export PYTHONPATH=${WORKDIR}/workflows/rheo/scripts:${WORKDIR}/workflows/rheo/scripts/simulation/rl:\$PYTHONPATH" >> /etc/bash.bashrc
+RUN echo "export PYTHONPATH=${WORKDIR}/workflows/rheo:${WORKDIR}/workflows/rheo/scripts:${WORKDIR}/workflows/rheo/scripts/simulation/rl:\$PYTHONPATH" >> /etc/bash.bashrc
 
 # Set aliases
 RUN echo "alias python='/isaac-sim/python.sh'" >> /etc/bash.bashrc
@@ -218,6 +272,12 @@ RUN echo "alias pytest='/isaac-sim/python.sh -m pytest'" >> /etc/bash.bashrc
 RUN pip3 install debugpy --break-system-packages
 RUN echo "alias debugpy='python -Xfrozen_modules=off -m debugpy --listen localhost:5678 --wait-for-client'" >> /etc/bash.bashrc
 
+# Strip pip warp-lang after all pip installs — PyPI warp shadows omni.warp-1.8.2 and breaks Kit.
+# pip uninstall fails on kit Python (cross-device link / .pyc permissions) — remove directories as root.
+RUN KIT_SITE="/isaac-sim/kit/python/lib/python3.11/site-packages" && \
+    chmod -R u+w "${KIT_SITE}/warp" "${KIT_SITE}/~arp" 2>/dev/null || true && \
+    rm -rf "${KIT_SITE}/warp" "${KIT_SITE}"/warp_lang-*.dist-info "${KIT_SITE}"/~arp* 2>/dev/null || true
+
 # Prompt
 RUN echo "PS1='[RHEO] \[\e[0;32m\]~\u \[\e[0;34m\]\w\[\e[0m\] \$ '" >> /etc/bash.bashrc
 RUN echo "alias ll='ls -alF --color=auto'" >> /etc/bash.bashrc

workflows/rheo/docker/run_docker.sh

@@ -3,6 +3,11 @@
 # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0
 
+# Re-exec with bash when invoked as `sh run_docker.sh` (dash does not support [[ or arrays).
+if [ -z "${BASH_VERSION:-}" ]; then
+    exec bash "$0" "$@"
+fi
+
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -183,6 +188,11 @@ add_volume_if_it_exists() {
     [ -d "$src" ] && echo "-v $src:$dst"
 }
 
+# Lightwheel SDK cache (nested dirs required for version metadata .txt files)
+mkdir -p "$HOME/.cache/lightwheel_sdk/object/CoffeeMachine037"
+mkdir -p "$HOME/.cache/lightwheel_sdk/object/Microwave039"
+mkdir -p "$HOME/.cache/lightwheel_sdk/object"
+
 # Run container
 if [ "$NEW_CONTAINER" = false ] && [ "$( docker container inspect -f '{{.State.Running}}' $CONTAINER_NAME 2>/dev/null)" = "true" ]; then
     if [ $# -ge 1 ]; then
@@ -247,10 +257,29 @@ else
         fi
     fi
 
-    # X11
-    if [ -n "$DISPLAY" ]; then
-        echo "Allowing X11 connections"
-        xhost +local:docker > /dev/null
+    # RTX 50-series (Blackwell): forwarding host DISPLAY can destabilize headless RTX/Vulkan init.
+    if [ "${RHEO_DISABLE_BLACKWELL_RENDER_PATCH:-}" != "1" ] && [ "${RHEO_KEEP_DISPLAY:-}" != "1" ]; then
+        if [ -n "${DISPLAY:-}" ] && [ "${HEADLESS:-1}" != "0" ]; then
+            echo "Note: Unsetting DISPLAY for container (RTX 50-series headless stability)."
+            echo "      Use HEADLESS=0 RHEO_KEEP_DISPLAY=1 for Isaac Sim GUI."
+            unset DISPLAY
+        fi
+    fi
+
+    # X11 (optional). Do not abort if xhost fails (SSH/headless); unset DISPLAY so Isaac Sim
+    # runs headless instead of failing on an invalid display.
+    if [ -n "${DISPLAY:-}" ]; then
+        echo "Allowing X11 connections for DISPLAY=${DISPLAY}"
+        if xhost +local:docker > /dev/null 2>&1; then
+            echo "X11 forwarding enabled."
+        else
+            echo "Warning: xhost failed (no X server on ${DISPLAY}). Running headless."
+            echo "         Use --webrtc_cam and open http://localhost:8080 to view the scene."
+            unset DISPLAY
+        fi
+    fi
+    if [ -z "${DISPLAY:-}" ]; then
+        export HEADLESS=1
     fi
 
     if [ $# -ge 1 ]; then