Add native OpenCV lens-distortion cameras for OVRTX#6608
Open
liorbenhorin wants to merge 11 commits into
Open
Conversation
Add OpenCvPinholeDistortionCfg and OpenCvFisheyeDistortionCfg and a distortion field on PinholeCameraCfg so a real camera calibration (fx/fy/cx/cy plus distortion coefficients) can be reproduced in sim. The model is authored at spawn as the omni:lensdistortion USD API, which the RTX/OVRTX renderer honors natively. Make the intrinsics readback distortion-aware so camera.data.intrinsic_matrices reflects a non-square or off-center calibration instead of assuming fx == fy and a centered principal point. The Newton renderer does not yet apply the model and is left as a documented no-op with a warning.
Propagate OpenCvDistortionCfg, OpenCvPinholeDistortionCfg and OpenCvFisheyeDistortionCfg through the isaaclab.sim.spawners and isaaclab.sim lazy-export stubs, so they are importable as isaaclab.sim.<Cfg> like the sibling PinholeCameraCfg and FisheyeCameraCfg.
liorbenhorin
requested review from
AntoineRichard,
Mayankm96,
StafaH,
daniela-hase,
huidongc,
kellyguo11,
matthewtrepte,
mmichelis,
ooctipus,
pascal-roth,
pbarejko and
vidurv-nvidia
as code owners
July 19, 2026 11:03
github-actions
Bot
added
documentation
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+666
to
+669
| f_x = float(raw_prim.GetAttribute(f"{prefix}:fx").Get()) | ||
| f_y = float(raw_prim.GetAttribute(f"{prefix}:fy").Get()) | ||
| c_x = float(raw_prim.GetAttribute(f"{prefix}:cx").Get()) | ||
| c_y = float(raw_prim.GetAttribute(f"{prefix}:cy").Get()) |
Contributor
There was a problem hiding this comment.
None from missing distortion intrinsic attrs
GetAttribute(...).Get() returns None when the attribute is not authored on the prim; float(None) then raises TypeError on every subsequent update step. The imageSize read two lines below is already None-checked, but fx/fy/cx/cy are not. A USD scene that carries omni:lensdistortion:model without the matching intrinsic attributes (e.g., a hand-authored prim or a third-party export that only writes the schema token) will crash the camera sensor continuously from the first update call.
Comment on lines
+674
to
+682
| if not self._warned_distortion_image_size: | ||
| logger.warning( | ||
| "Camera lens-distortion 'imageSize' %s does not match the render resolution" | ||
| " (%d, %d). The reported intrinsic matrix is in the calibrated pixel space.", | ||
| (int(image_size[0]), int(image_size[1])), | ||
| width, | ||
| height, | ||
| ) | ||
| self._warned_distortion_image_size = True |
Contributor
There was a problem hiding this comment.
_warned_distortion_image_size is a per-Camera-instance flag, but one Camera object manages N sensor prims (one per environment). Once any single env's prim triggers the warning, the flag is set and all subsequent per-prim mismatches in the same instance — including those in different environments with different calibration sizes — are silently dropped. In a large multi-env setup a user would see one warning and have no indication that the mismatch also affects all other envs. Consider including the env index in the message, or emitting the warning per prim the first time it fires.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Fall back to the focal-length/aperture projection when a camera prim carries the omni:lensdistortion:model token without the fx/fy/cx/cy intrinsics, instead of dereferencing a None attribute value on every update step and raising TypeError continuously. Key the image-size mismatch warning on the authored size and include the prim path and environment index, so a genuinely different calibration in another environment is surfaced rather than masked by the first prim's one-time flag. Add regression tests covering the missing-intrinsics fallback and the per-size warning behavior.
Author
|
greptile issues rexsolved |
Extract the authored OpenCV lens-distortion readback out of the per-camera loop in Camera._update_intrinsic_matrices into a dedicated _read_authored_opencv_intrinsics helper. The loop body drops from four levels of nesting to a single authored-vs-fallback branch, keeping the warn-once guards and behavior unchanged. Also refresh the method docstring, which still claimed intrinsics always assume square pixels, and tidy the warning-guard comment.
The distortion field is defined on PinholeCameraCfg and inherited by FisheyeCameraCfg, the two cfg types spawn_camera accepts, so the defensive getattr lookup is unnecessary and inconsistent with the direct attribute access used elsewhere in the function.
The kit-less test_opencv_distortion.py carried only a skipif and no level marker, unlike every sibling in test/sensors/ (kit-less unit tests use pytest.mark.unit). Without the level marker the file is not categorized as a unit test in the JUnit report nor selected by '-m unit', so CI's unit lane skips it. Add pytest.mark.unit to the module-level pytestmark to match the convention.
The (height, width) viewport read is constant across the per-env loop in _update_intrinsic_matrices; compute it once before the loop rather than on every iteration.
Describe the Camera.data intrinsics change in the changelog without cross-referencing a private method. Add reciprocal See Also notes between FisheyeCameraCfg and OpenCvFisheyeDistortionCfg, and mention the optional OpenCV calibration in the sensors spawner module docstring. Broaden the Newton renderer warning to state that a non-square fx, the principal point, and the distortion coefficients are all dropped (Newton derives one field of view from fy), not only the distortion.
A camera with an authored OpenCV lens-distortion model owns its fx/fy/cx/cy through the omni:lensdistortion:* calibration, which the readback prefers; set_intrinsic_matrices only writes the square-pixel, centered focal-length/ aperture attributes and cannot express it. Skip such cameras per entry (with a warning) and leave their calibration untouched, while still updating any ordinary cameras selected in the same call. Regression-tested with a mixed distortion/plain batch.
Render the OpenCV fisheye model and an undistorted pinhole through OVRTX and assert the frames differ well beyond render noise, guarding that the renderer honors the OmniLensDistortionOpenCvFisheyeAPI schema.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Isaac Lab's stock camera cfgs (PinholeCameraCfg, FisheyeCameraCfg) can only express a focal-length + aperture projection. They cannot represent an OpenCV fx/fy/cx/cy + distortion-coefficient intrinsic model — the format real-camera and LeRobot calibrations come in — so a calibrated real camera (non-square pixels, off-center principal point, lens distortion) cannot be reproduced in simulation.
This PR adds first-class support for OpenCV lens-distortion cameras, rendered natively through the RTX/OVRTX renderer:
This upstreams a capability that has been proven out-of-tree in the SO-101 sim-to-real workshop; once merged, that workshop can delete its private OpenCV-pinhole spawner and use these APIs directly.
Dependencies: none (no new required or optional dependencies).
Fixes # (N/A — new feature)
Type of change
Screenshots
Rendering the grid-textured ground plane - Coefficients exaggerated (--k_scale 15) for visibility.
Checklist
(https://isaac-sim.github.io/IsaacLab/mainl)