Skip to content

Add Sensor PPISP controller support for camera post-processing #5947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kellyguo11 merged 2 commits into from
Jun 15, 2026
Merged

Add Sensor PPISP controller support for camera post-processing #5947

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ee02eae
Add PPISP controller support
moennen Jun 9, 2026
d8d9fd0
Add PPISP controller changelog fragments
moennen Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 49 additions & 43 deletions docs/source/overview/core-concepts/sensors/camera.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,18 +239,21 @@ the project page: https://research.nvidia.com/labs/sil/projects/ppisp .

PPISP is typically authored alongside a `ParticleField3DGaussianSplat
<https://openusd.org/release/user_guides/schemas/usdVol/ParticleField3DGaussianSplat.html>`__
USD asset: it carries a `RenderProduct
<https://openusd.org/release/user_guides/schemas/usdRender/RenderProduct.html>`__
whose target camera and PPISP `UsdShade.Shader
<https://openusd.org/release/api/class_usd_shade_shader.html>`__
(a shader prim named ``PPISP`` whose inputs follow the PPISP naming
convention) were calibrated against the real capture rig that produced
the splats. Configuring the camera with the matching PPISP coefficients
makes the rendered tile match the calibration target.

The pipeline applies, in order: responsivity → exposure → vignetting →
color homography → camera response function → uint8 clamp. It runs as a
single Warp kernel.
USD asset: its camera prim authors ``ppisp:*`` attributes calibrated against
the real capture rig that produced the splats. Configuring the camera with
the matching PPISP coefficients makes the rendered tile match the calibration
target.

With static coefficients, the image pass applies, in order: responsivity →
exposure → vignetting → color homography → camera response function → uint8
clamp. It runs as a single Warp image kernel.

When the camera also authors ``ppisp:controllerWeights``, Isaac Lab runs the
exported PPISP controller before the image pass. The controller reads the
current HDR image, prior exposure, and controller responsivity, then predicts
``exposureOffset`` and the four color-latent pairs for each camera view. The
image pass then uses those predicted exposure and color-latent values while
static PPISP inputs still provide responsivity, vignetting, and CRF.

Configuration
^^^^^^^^^^^^^
Expand All @@ -259,15 +262,18 @@ Configuration

* ``None`` (default) — ISP disabled.
* :class:`~isaaclab_ppisp.PpispCfg` — explicit PPISP coefficients
(:attr:`~isaaclab_ppisp.PpispCfg.inputs`), or
:attr:`~isaaclab_ppisp.PpispCfg.shader_prim_path` to import them from a
PPISP ``UsdShade.Shader`` already on the stage.
* :class:`~sensors.CameraISPMode` — auto-discover an ISP shader on the
(:attr:`~isaaclab_ppisp.PpispCfg.inputs`) and, optionally, controller
weights. Use :attr:`~isaaclab_ppisp.PpispCfg.camera_prim_path` to import
static coefficients and camera-authored controller weights from a USD camera
already on the stage.
* :class:`~sensors.CameraISPMode` — auto-discover ISP camera attributes on the
stage (see below).

The cfg applies once per Camera sensor batch. The PPISP Warp kernel takes
scalar coefficients, so every cloned view in a tiled batch shares the same
ISP configuration — there is no per-view ISP today.
The cfg applies once per Camera sensor batch. Static PPISP inputs are scalar
coefficients shared by every cloned view in a tiled batch. When controller
weights are present, the controller predicts per-view exposure and color-latent
values from each HDR image, while responsivity, vignetting, and CRF remain
shared by the batch.

.. code-block:: python

Expand All @@ -280,8 +286,8 @@ ISP configuration — there is no per-view ISP today.
# explicit coefficients
cfg = CameraCfg(..., isp_cfg=PpispCfg(inputs={"exposureOffset": 1.5}))

# import coefficients from a USD shader path
cfg = CameraCfg(..., isp_cfg=PpispCfg(shader_prim_path="/World/Render/PPISP"))
# import coefficients from a USD camera path
cfg = CameraCfg(..., isp_cfg=PpispCfg(camera_prim_path="/World/Camera_ppisp"))

# auto-discover from the stage
cfg = CameraCfg(..., isp_cfg=CameraISPMode.AUTO_ANY)
Expand All @@ -293,22 +299,19 @@ Auto-discovery is opt-in via :class:`~sensors.CameraISPMode`. Discovery runs
once at camera construction using the first matched camera prim in the Camera
sensor batch:

1. Walk the stage for a USD ``RenderProduct`` whose ``camera`` relationship
targets the first matched camera prim **and** that has a child
``UsdShade.Shader`` prim named ``PPISP``. If found, import its inputs as a
:class:`~isaaclab_ppisp.PpispCfg`.
2. ``AUTO_ANY`` only: if step 1 finds nothing, fall back to the first
``UsdShade.Shader`` prim named ``PPISP`` anywhere on the stage.
1. Check the first matched camera prim for ``ppisp:*`` attributes.
2. ``AUTO_ANY`` only, or when no camera path is available: fall back to the
first camera anywhere on the stage with ``ppisp:*`` attributes.
3. Otherwise the ISP stays disabled for the whole Camera sensor batch.

In practice this means: if the stage carries a ``ParticleField3DGaussianSplat``
together with a ``RenderProduct`` that binds a ``PPISP`` shader child to the
batch's first matched camera prim, the Camera sensor picks up the matching ISP
automatically and no Python-side coefficient authoring is required.
together with a camera that authors ``ppisp:*`` attributes, the Camera sensor
picks up the matching ISP automatically, including controller weights when
authored, and no Python-side coefficient authoring is required.

``AUTO_CAMERA`` runs step 1 only — useful when the stage carries multiple
PPISP shaders and you want the Camera sensor batch to use exactly the one bound
to its first matched camera prim.
``AUTO_CAMERA`` runs only the camera-local discovery steps — useful when the
stage carries multiple PPISP cameras and you want the Camera sensor batch to use
the attributes authored on its first matched camera prim.

Renderer support
^^^^^^^^^^^^^^^^
Expand All @@ -320,7 +323,8 @@ internally: the Isaac RTX renderer sources HDR from the Replicator
Newton Warp renderer from its native scene-linear color buffer. Each
backend allocates its own HDR scratch buffer when the user did not request
``"rgb_hdr"`` in :attr:`~sensors.CameraCfg.data_types`, and dispatches the
PPISP kernel into ``rgb`` / ``rgba`` after every render tick.
PPISP pipeline into ``rgb`` / ``rgba`` after every render tick. For controller
configs, this is the controller pass followed by the PPISP image pass.

Usage example
^^^^^^^^^^^^^
Expand All @@ -337,28 +341,30 @@ absolute-difference images.

Use ``--renderer isaac_rtx`` to run the same workflow with Isaac RTX. Pass
``--input_scene`` for a custom scene and ``--camera_prim_path`` if the stage
contains multiple PPISP-bound cameras. If a config or command selects a visualizer,
force-disable all visualizers with ``--visualizer none`` or ``--viz none``. Images are
written to ``scripts/demos/sensors/output/ppisp_camera`` unless ``--output_dir`` is set.
contains multiple cameras with PPISP attributes. If a config or command selects
a visualizer, force-disable all visualizers with ``--visualizer none`` or
``--viz none``. Images are written to
``scripts/demos/sensors/output/ppisp_camera`` unless ``--output_dir`` is set.

Known limitations
^^^^^^^^^^^^^^^^^

* The ISP writes back into the ``rgb`` / ``rgba`` buffers. If neither is
requested, configuring ``isp_cfg`` raises at camera init.
* PPISP inputs are static for the lifetime of the camera. Animated USD
shader inputs are collapsed to their first authored time sample.
* Coefficients are global per camera — no per-pixel or per-region
authoring beyond the radial vignetting term.
* Static PPISP inputs and controller weights are fixed for the lifetime of the
camera. Animated USD camera attributes are collapsed to their first authored
time sample.
* Static coefficients are global per Camera sensor batch — no per-pixel or
per-region authoring beyond the radial vignetting term.
* PPISP is the only ISP implementation today. Other ISP families would
need a new config type and discoverer entry.
* On the Isaac RTX and OVRTX backends, enabling ``isp_cfg`` forces RTX-side
tonemapping off (``/rtx/rtpt/gaussian/skipTonemapping/enabled=False``)
and authors a neutral ``OmniRtxCameraExposureAPI_1`` schema on each
camera prim so the post-render ISP is the only path that processes
color. Mixing this with RTX-side exposure authoring is not supported.
* Auto-discovery resolves at camera construction; later authoring of a
``RenderProduct`` or shader on the stage is not picked up.
* Auto-discovery resolves at camera construction; later authoring of
``ppisp:*`` camera attributes on the stage is not picked up.

Depth and Distances
~~~~~~~~~~~~~~~~~~~
Expand Down
99 changes: 28 additions & 71 deletions scripts/demos/sensors/ppisp_camera.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
"--camera_prim_path",
type=str,
default=None,
help="Optional camera prim path override. Omit to auto-select the first PPISP-bound camera.",
help="Optional camera prim path override. Omit to auto-select the first camera with PPISP attributes.",
)
parser.add_argument(
"--camera_time_code",
Expand Down Expand Up @@ -126,9 +126,14 @@
import matplotlib.pyplot as plt
import numpy as np
import torch
from isaaclab_ppisp.cfg import PpispCfg, ppisp_cfg_from_usd_shader
from isaaclab_ppisp._demo_utils import (
find_ppisp_camera_bindings,
format_available_ppisp_cameras,
order_ppisp_bindings_by_camera,
)
from isaaclab_ppisp.cfg import PpispCfg, ppisp_cfg_from_usd_camera

from pxr import Usd, UsdGeom, UsdShade
from pxr import Usd, UsdGeom

import isaaclab.sim as sim_utils
from isaaclab.assets import AssetBaseCfg, RigidObjectCfg
Expand Down Expand Up @@ -191,61 +196,11 @@ def make_sim_cfg() -> sim_utils.SimulationCfg:
)


def find_ppisp_camera_bindings(stage: Usd.Stage) -> list[tuple[str, Usd.Prim, Usd.Prim]]:
"""Return camera, RenderProduct, and PPISP shader bindings from a source stage."""
bindings = []
for prim in stage.Traverse():
if prim.GetTypeName() != "RenderProduct":
continue
shader_prim = stage.GetPrimAtPath(prim.GetPath().AppendChild("PPISP"))
if not shader_prim or not shader_prim.IsValid():
continue
camera_rel = prim.GetRelationship("camera")
if not camera_rel:
continue
for target in camera_rel.GetTargets():
camera_prim = stage.GetPrimAtPath(target)
if camera_prim and camera_prim.IsValid():
bindings.append((str(target), prim, shader_prim))
return bindings


def order_ppisp_bindings_by_camera(
stage: Usd.Stage, ppisp_bindings: list[tuple[str, Usd.Prim, Usd.Prim]]
) -> list[tuple[str, Usd.Prim, Usd.Prim]]:
"""Return PPISP bindings ordered by source Camera prim traversal."""
binding_by_camera_path = {}
for binding in ppisp_bindings:
binding_by_camera_path.setdefault(binding[0], binding)

ordered_bindings = []
seen_paths = set()
for prim in stage.Traverse():
if prim.GetTypeName() != "Camera":
continue
camera_path = str(prim.GetPath())
binding = binding_by_camera_path.get(camera_path)
if binding is not None:
ordered_bindings.append(binding)
seen_paths.add(camera_path)

for binding in ppisp_bindings:
if binding[0] not in seen_paths:
ordered_bindings.append(binding)
seen_paths.add(binding[0])
return ordered_bindings


def format_available_ppisp_cameras(ppisp_bindings: list[tuple[str, Usd.Prim, Usd.Prim]]) -> str:
"""Format PPISP-bound camera paths for CLI error messages."""
return "\n ".join(dict.fromkeys(binding[0] for binding in ppisp_bindings))


def resolve_source_camera_binding(source_stage: Usd.Stage) -> tuple[str, Usd.Prim, Usd.Prim]:
"""Resolve the source camera and PPISP shader binding from CLI or source stage metadata."""
def resolve_source_camera_binding(source_stage: Usd.Stage) -> tuple[str, Usd.Prim | None, Usd.Prim]:
"""Resolve the source camera and PPISP camera binding from CLI or source stage metadata."""
ppisp_bindings = order_ppisp_bindings_by_camera(source_stage, find_ppisp_camera_bindings(source_stage))
if not ppisp_bindings:
raise RuntimeError("No PPISP RenderProduct bindings found in input scene.")
raise RuntimeError("No cameras with PPISP camera attributes found in input scene.")

if args_cli.camera_prim_path is not None:
camera_prim_path = args_cli.camera_prim_path
Expand All @@ -260,14 +215,14 @@ def resolve_source_camera_binding(source_stage: Usd.Stage) -> tuple[str, Usd.Pri
available = format_available_ppisp_cameras(ppisp_bindings)
raise RuntimeError(
f"Camera prim not found: {camera_prim_path}\n"
"Omit --camera_prim_path to auto-select a PPISP-bound camera, or use one of:\n"
"Omit --camera_prim_path to auto-select a camera with PPISP attributes, or use one of:\n"
f" {available}"
)
if camera_prim.GetTypeName() != "Camera":
available = format_available_ppisp_cameras(ppisp_bindings)
raise RuntimeError(
f"Prim is not a Camera: {camera_prim_path} ({camera_prim.GetTypeName()})\n"
"Omit --camera_prim_path to auto-select a PPISP-bound camera, or use one of:\n"
"Omit --camera_prim_path to auto-select a camera with PPISP attributes, or use one of:\n"
f" {available}"
)

Expand All @@ -277,8 +232,8 @@ def resolve_source_camera_binding(source_stage: Usd.Stage) -> tuple[str, Usd.Pri

available = format_available_ppisp_cameras(ppisp_bindings)
raise RuntimeError(
f"Selected camera has no PPISP RenderProduct binding: {camera_prim_path}\n"
"Omit --camera_prim_path to auto-select a PPISP-bound camera, or use one of:\n"
f"Selected camera has no PPISP camera attributes: {camera_prim_path}\n"
"Omit --camera_prim_path to auto-select a camera with PPISP attributes, or use one of:\n"
f" {available}"
)

Expand Down Expand Up @@ -353,8 +308,10 @@ def bake_source_camera_pose_to_envs(source_stage: Usd.Stage, source_camera_prim_
)


def get_render_product_resolution(render_product_prim: Usd.Prim) -> tuple[int, int] | None:
def get_render_product_resolution(render_product_prim: Usd.Prim | None) -> tuple[int, int] | None:
"""Return ``(width, height)`` from a RenderProduct ``resolution`` attribute."""
if render_product_prim is None:
return None
resolution_attr = render_product_prim.GetAttribute("resolution")
if not resolution_attr:
return None
Expand All @@ -364,7 +321,7 @@ def get_render_product_resolution(render_product_prim: Usd.Prim) -> tuple[int, i
return int(resolution[0]), int(resolution[1])


def resolve_image_shape(render_product_prim: Usd.Prim) -> tuple[int, int]:
def resolve_image_shape(render_product_prim: Usd.Prim | None) -> tuple[int, int]:
"""Resolve demo output ``(width, height)`` preserving source aspect when height is omitted."""
width = args_cli.image_width
height = args_cli.image_height
Expand All @@ -380,14 +337,14 @@ def resolve_image_shape(render_product_prim: Usd.Prim) -> tuple[int, int]:
return width, height


def make_ppisp_cfg(shader_prim: Usd.Prim, num_ppisp_bindings: int) -> PpispCfg:
"""Parse the selected source PPISP shader into an explicit cfg for duplicated envs."""
ppisp_cfg = ppisp_cfg_from_usd_shader(UsdShade.Shader(shader_prim))
# The duplicated stage does not carry the source /Render scope; keep the
# parsed inputs as explicit values instead of resolving this source path later.
ppisp_cfg.shader_prim_path = None
def make_ppisp_cfg(camera_prim: Usd.Prim, num_ppisp_bindings: int) -> PpispCfg:
"""Parse the selected source PPISP camera into an explicit cfg for duplicated envs."""
ppisp_cfg = ppisp_cfg_from_usd_camera(camera_prim)
# The duplicated stage can remap source camera paths; keep the parsed inputs
# as explicit values instead of resolving the original camera path later.
ppisp_cfg.camera_prim_path = None
if args_cli.ppisp_responsivity is None:
print(f"[INFO] Using USD-authored PPISP values from {num_ppisp_bindings} PPISP shader(s).", flush=True)
print(f"[INFO] Using USD-authored PPISP values from {num_ppisp_bindings} PPISP camera(s).", flush=True)
else:
ppisp_cfg.inputs["responsivity"] = float(args_cli.ppisp_responsivity)
print(
Expand Down Expand Up @@ -572,8 +529,8 @@ def main() -> None:
source_stage = Usd.Stage.Open(args_cli.input_scene)
if source_stage is None:
raise RuntimeError(f"Failed to open input scene: {args_cli.input_scene}")
source_camera_prim_path, render_product_prim, shader_prim = resolve_source_camera_binding(source_stage)
ppisp_cfg = make_ppisp_cfg(shader_prim, len(find_ppisp_camera_bindings(source_stage)))
source_camera_prim_path, render_product_prim, ppisp_camera_prim = resolve_source_camera_binding(source_stage)
ppisp_cfg = make_ppisp_cfg(ppisp_camera_prim, len(find_ppisp_camera_bindings(source_stage)))
camera_prim_path = source_camera_path_to_env_regex(source_stage, source_camera_prim_path)
width, height = resolve_image_shape(render_product_prim)

Expand Down
Loading
Loading