Skip to content

Latest commit

 

History

History
229 lines (174 loc) · 9.77 KB

cameras.mdx

File metadata and controls

229 lines (174 loc) · 9.77 KB

Cameras

LeRobot offers multiple options for video capture:

Class Supported Cameras
OpenCVCamera Phone, built-in laptop, USB webcams
ZMQCamera Network-connected cameras
RealSenseCamera Intel RealSense (with depth)
Reachy2Camera Reachy 2 robot cameras

Tip

For OpenCVCamera compatibility details, see the Video I/O with OpenCV Overview.

Find your camera

Every camera requires a unique identifier to be instantiated, allowing you to distinguish between multiple connected devices.

OpenCVCamera and RealSenseCamera support auto-discovery. Run the command below to list available devices and their identifiers. Note that these identifiers may change after rebooting your computer or re-plugging the camera, depending on your operating system.

lerobot-find-cameras opencv # or realsense for Intel Realsense cameras

To view a live preview of each camera in separate windows:

lerobot-preview-cameras        # preview all detected OpenCV cameras
lerobot-preview-cameras 0 1 2  # preview cameras at indices 0, 1, 2

Press q in any preview window to quit.

The output of lerobot-find-cameras will look something like this if you have two cameras connected:

--- Detected Cameras ---
Camera #0:
  Name: OpenCV Camera @ 0
  Type: OpenCV
  Id: 0
  Backend api: AVFOUNDATION
  Default stream profile:
    Format: 16.0
    Width: 1920
    Height: 1080
    Fps: 15.0
--------------------
(more cameras ...)

Warning

When using Intel RealSense cameras in macOS, you could get this error: Error finding RealSense cameras: failed to set power state, this can be solved by running the same command with sudo permissions. Note that using RealSense cameras in macOS is unstable.

ZMQCamera and Reachy2Camera do not support auto-discovery. They must be configured manually by providing their network address and port or robot SDK settings.

Use cameras

Frame access modes

All camera classes implement three access modes for capturing frames:

Method Behavior Blocks? Best For
read() Waits for the camera hardware to return a frame. May block for a long time depending on the camera and SDK. Yes Simple scripts, sequential capture
async_read(timeout_ms) Returns the latest unconsumed frame from background thread. Blocks only if buffer is empty, up to timeout_ms. Raises TimeoutError if no frame arrives. With a timeout Control loops synchronized to camera FPS
read_latest(max_age_ms) Peeks at the most recent frame in buffer (may be stale). Raises TimeoutError if frame is older than max_age_ms. No UI visualization, logging, monitoring

Usage examples

The following examples show how to use the camera API to configure and capture frames from different camera types.

  • Blocking and non-blocking frame capture using an OpenCV-based camera
  • Color and depth capture using an Intel RealSense camera

Warning

Failing to cleanly disconnect cameras can cause resource leaks. Use the context manager protocol to ensure automatic cleanup:

with OpenCVCamera(config) as camera:
    ...

You can also call connect() and disconnect() manually, but always use a finally block for the latter.

from lerobot.cameras.opencv.configuration_opencv import OpenCVCameraConfig
from lerobot.cameras.opencv.camera_opencv import OpenCVCamera
from lerobot.cameras.configs import ColorMode, Cv2Rotation

# Construct an `OpenCVCameraConfig` with your desired FPS, resolution, color mode, and rotation.
config = OpenCVCameraConfig(
    index_or_path=0,
    fps=15,
    width=1920,
    height=1080,
    color_mode=ColorMode.RGB,
    rotation=Cv2Rotation.NO_ROTATION
)

# Instantiate and connect an `OpenCVCamera`, performing a warm-up read (default).
with OpenCVCamera(config) as camera:

    # Read a frame synchronously — blocks until hardware delivers a new frame
    frame = camera.read()
    print(f"read() call returned frame with shape:", frame.shape)

    # Read a frame asynchronously with a timeout — returns the latest unconsumed frame or waits up to timeout_ms for a new one
    try:
        for i in range(10):
            frame = camera.async_read(timeout_ms=200)
            print(f"async_read call returned frame {i} with shape:", frame.shape)
    except TimeoutError as e:
        print(f"No frame received within timeout: {e}")

    # Instantly return a frame - returns the most recent frame captured by the camera
    try:
        initial_frame = camera.read_latest(max_age_ms=1000)
        for i in range(10):
            frame = camera.read_latest(max_age_ms=1000)
            print(f"read_latest call returned frame {i} with shape:", frame.shape)
            print(f"Was a new frame received by the camera? {not (initial_frame == frame).any()}")
    except TimeoutError as e:
        print(f"Frame too old: {e}")
from lerobot.cameras.realsense.configuration_realsense import RealSenseCameraConfig
from lerobot.cameras.realsense.camera_realsense import RealSenseCamera
from lerobot.cameras.configs import ColorMode, Cv2Rotation

# Create a `RealSenseCameraConfig` specifying your camera’s serial number and enabling depth.
config = RealSenseCameraConfig(
    serial_number_or_name="233522074606",
    fps=15,
    width=640,
    height=480,
    color_mode=ColorMode.RGB,
    use_depth=True,
    rotation=Cv2Rotation.NO_ROTATION
)

# Instantiate and connect a `RealSenseCamera` with warm-up read (default).
camera = RealSenseCamera(config)
camera.connect()

# Capture a color frame via `read()` and a depth map via `read_depth()`.
try:
    color_frame = camera.read()
    depth_map = camera.read_depth()
    print("Color frame shape:", color_frame.shape)
    print("Depth map shape:", depth_map.shape)
finally:
    camera.disconnect()

Use your phone's camera

To use your iPhone as a camera on macOS, enable the Continuity Camera feature:

  • Ensure your Mac is running macOS 13 or later, and your iPhone is on iOS 16 or later.
  • Sign in both devices with the same Apple ID.
  • Connect your devices with a USB cable or turn on Wi-Fi and Bluetooth for a wireless connection.

For more details, visit Apple support.

If you want to use your phone as a camera using OBS, follow these steps to set up a virtual camera.

  1. (Linux only) Install v4l2loopback-dkms and v4l-utils. These packages create virtual camera devices and verify their settings. Install with:
sudo apt install v4l2loopback-dkms v4l-utils

  1. Install the DroidCam app on your phone. This app is available for both iOS and Android.

  2. Download and install OBS Studio.

  3. Download and install the DroidCam OBS plugin.

  4. Start OBS Studio.

  5. Add your phone as a source. Follow the instructions here. Be sure to set the resolution to 640x480 to avoid the watermarks.

  6. Adjust resolution settings. In OBS Studio, go to File > Settings > Video or OBS > Preferences... > Video. Change the Base(Canvas) Resolution and the Output(Scaled) Resolution to 640x480 by manually typing it.

  7. Start virtual camera. In OBS Studio, follow the instructions here.

  8. Verify the virtual camera setup and resolution.

    • Linux: Use v4l2-ctl to list devices and check resolution: 
      v4l2-ctl --list-devices  # find VirtualCam and note its /dev/videoX path
v4l2-ctl -d /dev/videoX --get-fmt-video  # replace with your VirtualCam path
      You should see VirtualCam listed and resolution 640x480.
    • macOS: Open Photo Booth or FaceTime and select "OBS Virtual Camera" as the input.
    • Windows: The native Camera app doesn't support virtual cameras. Use a video conferencing app (Zoom, Teams) or run lerobot-find-cameras opencv directly to verify.
Troubleshooting

The virtual camera resolution is incorrect.

Delete the virtual camera source and recreate it. The resolution cannot be changed after creation.

Error reading frame in background thread for OpenCVCamera(X): OpenCVCamera(X) frame width=640 or height=480 do not match configured width=1920 or height=1080.

This error is caused by OBS Virtual Camera advertising a 1920x1080 resolution despite rescaling. The only fix for now is to comment out the width and height check in _postprocess_image().

If everything is set up correctly, your phone will appear as a standard OpenCV camera and can be used with OpenCVCamera.