Skip to content

[Bug Report] TiledCamera outputs incorrect normals data due to buffer shift #4239

@Rabbit-Hu

Description

@Rabbit-Hu

Describe the bug

When using TiledCamera with "normals" enabled in data_types, the normals image obtained by camera.data.output["normals"] is incorrect. The incorrect image looks like vertical color stripes.

After comparing the implementation of TiledCamera (source/isaaclab/isaaclab/sensors/camera/tiled_camera.py) with that of Camera, I have located the issue: the normals annotator returns a databuffer of 4 channels (x, y, z, _), where the 4th channel is discarded by Camera._process_annotator_output to obtain the correct output of 3 channels (x, y, z). This process was not implemented in TiledCamera; instead, the 4-channel data buffer is directly reshaped into [B, H, W, C=3], leading to the incorrect output.

Image

Steps to reproduce

To reproduce the bug, you can add "normals" to the data_types of the camera in source/isaaclab_tasks/isaaclab_tasks/manager_based/classic/cartpole/cartpole_camera_env_cfg.py, and render one frame of the normals output.

In the environment config:

camera: TiledCameraCfg = TiledCameraCfg(
      prim_path="{ENV_REGEX_NS}/Camera",
      offset=TiledCameraCfg.OffsetCfg(pos=(-7.0, 0.0, 3.0), rot=(0.9945, 0.0, 0.1045, 0.0), convention="world"),
      data_types=["rgb", "normals"],
      spawn=sim_utils.PinholeCameraCfg(
          focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
      ),
      width=100,
      height=100,
  )

In the rendering script:

# intialize and step the simulation once
normals_data = camera_data["normals"][0].cpu().numpy()  # Shape: (H, W, 3)
# then visualize normals_data

System Info

  • Commit: 44b9f1
  • Isaac Sim Version: 5.1.0.0
  • OS: Ubuntu 22.04.5 LTS
  • GPU: NVIDIA GeForce RTX 4090
  • CUDA: 12.6
  • GPU Driver: 560.28.03

Additional context

I will submit a Pull Request containing a simple fix later.

Checklist

  • I have checked that there is no similar issue in the repo (required)
  • I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

  • When TiledCamera outputs correct normals data by discarding the 4-th channel before reshaping the tensor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions