Enable selection of sensor format to avoid a cropped FoV

[larsll]

Work to resolve #99

It enables the raw role, and adjusts the size of it to match the desired input, this avoids cropping where libcamera by default selects a sensor mode that is not having the full FoV. This is a problem with RPi Camera V2 (see https://picamera.readthedocs.io/en/release-1.13/fov.html#sensor-modes).

Example:

ros2 run camera_ros camera_node --ros-args -p format:=BGR888 -p height:=120 -p width:=160 -p mode_width:=1640 -p mode_height:=1232

Output:

[INFO] [1740686517.425357897] [camera]: 1640x1232
[INFO] [1740686517.425436891] [camera]: Mode configuration:1640x1232-SBGGR12_CSI2P
[WARN] [1740686517.425989703] [camera]: stream configuration adjusted from "160x120-BGR888" to "160x120-BGR888"
[0:46:50.002516247] [3249]  INFO Camera camera.cpp:1033 configuring streams: (0) 160x120-BGR888 (1) 1640x1232-SBGGR10_CSI2P
[0:46:50.003349197] [3259]  INFO RPI vc4.cpp:512 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA
[INFO] [1740686517.429272329] [camera]: camera "/base/soc/i2c0mux/i2c@1/imx219@10" configured with 160x120-BGR888 stream

[christianrauch]

Thanks for the PR.

Can you add a before/after comparison to this PR? This will make it more clear what the new parameter do.

The new parameters and their behaviour and interaction with the other sizes and stream roles needs documentation, if this should make it into the upstream code. I think having two sets of sizes is confusing as it is not clear from the parameter names what the effect is.

[christianrauch]

Have you had a look at the std::optional<SensorConfiguration> sensorConfig; inside the CameraConfiguration? This seems to store the sensor configuration, such as the binning, size and crop region. Can you achieve the same effect by setting those values?

[christianrauch]

When you update the PR, can you rebase and squash your commits? I merged a change that avoids referring to the read-only parameters multiple times. Have a look at this and try following the same with the new read-only parameters that you want to add.

[christianrauch]

Have you had a look at the std::optional<SensorConfiguration> sensorConfig; inside the CameraConfiguration? This seems to store the sensor configuration, such as the binning, size and crop region. Can you achieve the same effect by setting those values?

Setting the sensorConfig (SensorConfiguration)

  libcamera::SensorConfiguration sensor;
  sensor.outputSize = libcamera::Size(4608, 2592);
  sensor.bitDepth = 10;

  if (sensor.isValid())
    cfg->sensorConfig = sensor;
  else
    throw std::runtime_error("invalid sensor configuration");

before the CameraConfiguration is validated, indeed has the same effect:

INFO Camera camera.cpp:1202 configuring streams: (0) 160x120-XRGB8888
INFO RPI pisp.cpp:1484 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected CFE format: 4608x2592-PC1B

without the need to manage an additional stream.

But the size and bit depth have to match exactly for the validation to pass.

[larsll]

Back. I would then suggest to do a parameter sensor, which is width:height:bitdepth using what you wrote? I am not worried about the invalid sensor configuration, as it is something you set up once for a camera, in a given setting.

[christianrauch]

Back. I would then suggest to do a parameter sensor, which is width:height:bitdepth using what you wrote?

Yeah, fine with me. Can you link here where the format string is used that way? How do they incorporate the other parameters, such as binning and skipping, into the config?

An alternative would be to have the parameters width, height, bitdepth inside the parameter namespace sensor, i.e. such that you can set them via sensor.width, sensor.height, sensor.bitdepth on the command line. But that is quite verbose, since you have to set all three parameters at once if you want to change the sensor configuration.

I am not worried about the invalid sensor configuration, as it is something you set up once for a camera, in a given setting.

You just have to know the valid values for each of the parameter, since there is no automatic adjustment. And it is not so obvious where to get this information from.

I think it would be nice to show the valid values on the screen, similar to what cam -c1 -I --stream role=raw shows, if configuration fails. But this would again require to at least generate and read the raw stream configuration once (generateConfiguration) without using the stream. Not sure if there is another way to get this information.

[larsll]

Unfortunately SensorConfiguration is not an option for ROS Humble, as ros-humble-libcamera is version 0.1.0; see https://github.com/raspberrypi/libcamera/blob/v0.1.0/include/libcamera/camera.h, where SensorConfiguration is missing. (It seems to be included in some RPi specific patches, otherwise starting in v0.2.0.

[larsll]

I think it would be nice to show the valid values on the screen, similar to what cam -c1 -I --stream role=raw shows, if configuration fails. But this would again require to at least generate and read the raw stream configuration once (generateConfiguration) without using the stream. Not sure if there is another way to get this information.

Looking at picamera2 -- doing generateConfiguration is what they do:
https://github.com/raspberrypi/picamera2/blob/fb031cb49cb2af4dab7439be7a90784c9d3db1ab/picamera2/picamera2.py#L483

[christianrauch]

Unfortunately SensorConfiguration is not an option for ROS Humble, as ros-humble-libcamera is version 0.1.0; see https://github.com/raspberrypi/libcamera/blob/v0.1.0/include/libcamera/camera.h, where SensorConfiguration is missing. (It seems to be included in some RPi specific patches, otherwise starting in v0.2.0.

Either we make this an libcamrea >= 0.2 only feature or we just leave libcamea 0.1 and ROS humble support for bloom packages behind. libcamera has high requirements on the meson version, the build farm only allows using system packages for build tool, and the RHEL 8 support here is the lower bound of the meson support we have to deal with on the build farm.

Options:

1. bump the version requirement for libcamera to support SensorConfiguration and then advice ROS humble users to compile libcamea from source, using whatever meson version from PyPI required for libcamera
2. use preprocessor macros to disable that feature at compile time on libcamera <= 0.1 (see LIBCAMERA_VER_GE)

I am in favour of disabling that feature and its parameters at compile time via LIBCAMERA_VER_GE. This is already done a couple of times in the code to support multiple libcamera versions.

[christianrauch]

Btw, if this changes the FoV independently of the output resoltion, that property has to be encoded in the calibration file name.

Currently, the calibration file name is a concatenation of the model (property libcamera::properties::Model), camera name (libcamera::Camera::id()), and the stream resolution (libcamera::StreamConfiguration::size), e.g.: IntegratedCamera_IntegratedC___SB__PCI0_XHC__RHUB_HS08_8_1_0_5986_2115_1280x720.yaml. Having the same stream resolution with different sensor modes will create conflicts as this will map to the same file name. Hence, we should append the sensor configuration or anything else that affects the FoV and image centre to the file name to avoid this conflict.

[christianrauch]

I think it would be nice to show the valid values on the screen, similar to what cam -c1 -I --stream role=raw shows, if configuration fails. But this would again require to at least generate and read the raw stream configuration once (generateConfiguration) without using the stream. Not sure if there is another way to get this information.

Looking at picamera2 -- doing generateConfiguration is what they do: https://github.com/raspberrypi/picamera2/blob/fb031cb49cb2af4dab7439be7a90784c9d3db1ab/picamera2/picamera2.py#L483

Then let's do the same. When the selected SensorConfiguration is invalid or the configuration fails, show a warning with the supported sensor modes like for the other parameters.

[larsll]

Options:

1. bump the version requirement for libcamera to support SensorConfiguration and then advice ROS humble users to compile libcamea from source, using whatever meson version from PyPI required for libcamera
2. use preprocessor macros to disable that feature at compile time on libcamera <= 0.1 (see LIBCAMERA_VER_GE)

I am in favour of disabling that feature and its parameters at compile time via LIBCAMERA_VER_GE. This is already done a couple of times in the code to support multiple libcamera versions.

A third option is obviously to have two different implementations; using SensorConfiguration for higher versions of LibCamera, and the RAW stream for lower versions. It will be a bit of 'ifdef' spaghetti though.

My personal need is to have support for Ubuntu 22.04 (meaning ROS Humble), Compute Module 4 and Camera Module v2; where for some reason legacy support is no longer working, so I would need something compatible with that (or else I would have to maintain a fork, which I don't want to do.

[christianrauch]

Options:

1. bump the version requirement for libcamera to support SensorConfiguration and then advice ROS humble users to compile libcamea from source, using whatever meson version from PyPI required for libcamera
2. use preprocessor macros to disable that feature at compile time on libcamera <= 0.1 (see LIBCAMERA_VER_GE)

I am in favour of disabling that feature and its parameters at compile time via LIBCAMERA_VER_GE. This is already done a couple of times in the code to support multiple libcamera versions.

A third option is obviously to have two different implementations; using SensorConfiguration for higher versions of LibCamera, and the RAW stream for lower versions. It will be a bit of 'ifdef' spaghetti though.

I am not a fan of having two paths to the same function. That's just a additional maintainance burden. If a solution works for both ROS / libcamera versions, I would rather only maintain that one solution.

My personal need is to have support for Ubuntu 22.04 (meaning ROS Humble), Compute Module 4 and Camera Module v2; where for some reason legacy support is no longer working, so I would need something compatible with that (or else I would have to maintain a fork, which I don't want to do.

I see. If you cannot use the newer version and are the one goint to implement this, then you are leaving me no choice but to accept your implementation :-)

With "legacy support" do you mean support for the legacy camera stack? This isn't supported on any platform with newer Raspberry Pi OS any more. May I ask why you cannot use Ubuntu 24.04 on the Raspberry Pi Compute Module 4? Is compiling from source also not an option for you?

[christianrauch]

On second thought, if you are using the Camera Module v2 on Ubuntu, I assume you are already compiling libcamera from source. Or does the binary bloomed libcamera package in the ROS repo support the Camera Module v2? If you already compile libcamera from source, doing the same for the node should not be a problem.

[larsll]

With "legacy support" do you mean support for the legacy camera stack? This isn't supported on any platform with newer Raspberry Pi OS any more.

Yes. For some obscure reason the RPi4 / Ubuntu 22.04 combo supports Legacy Camera, so I can activate that and grab the pictures using a basic camera package provided by AWS.

As we want to reduce the footprint of the solution we also want to support the CM4 with a custom carrier board, and for some strange reason Legacy Camera is not working with the CM4. I thought I had a solution when I found camera_ros, only to see that libcamera ends up cropping the picture...

On second thought, if you are using the Camera Module v2 on Ubuntu, I assume you are already compiling libcamera from source. Or does the binary bloomed libcamera package in the ROS repo support the Camera Module v2? If you already compile libcamera from source, doing the same for the node should not be a problem.

ros-humble-libcamera works with Ubuntu 22.04/CM4/Camera Module v2 as-is, no need to compile it. The preferred solution is to use the package for simplicity / reduce future work. As the project patches and compiles up a full DeepRacer package-set and bundles it as a deb-package, we can easily compile a branch/fork of camera_ros (but comes with the future maintenance penalty).

May I ask why you cannot use Ubuntu 24.04 on the Raspberry Pi Compute Module 4? Is compiling from source also not an option for you?

We are looking to support that, as support for RPi5, CM5 and other cameras is desirable as well, but we have quite a bit of OS-level integration (network configuration, USB-OTG etc.) that will require work.

I see. If you cannot use the newer version and are the one goint to implement this, then you are leaving me no choice but to accept your implementation :-)

I suggest to do the following:

• Define the input parameter as sensor; initial values are width:height, but with future expansion to width:height[:somethingelse] being backward compatible. This will enable a change of implementation without breaking it for the user.
• At some future point have a separate ros-humble branch with the main branch supporting ros-jazzy and whatever comes next. This is in line with how other packages maintain backwards compatibility with older ROS versions.

[larsll]

Btw, if this changes the FoV independently of the output resoltion, that property has to be encoded in the calibration file name.

Currently, the calibration file name is a concatenation of the model (property libcamera::properties::Model), camera name (libcamera::Camera::id()), and the stream resolution (libcamera::StreamConfiguration::size), e.g.: IntegratedCamera_IntegratedC___SB__PCI0_XHC__RHUB_HS08_8_1_0_5986_2115_1280x720.yaml. Having the same stream resolution with different sensor modes will create conflicts as this will map to the same file name. Hence, we should append the sensor configuration or anything else that affects the FoV and image centre to the file name to avoid this conflict.

Ok. I will look at how cam is doing this and replicate it if suitable.

[larsll]

More dependency complexity;

• Raspberry Pi 4 + Pi Camera v2 + Ubuntu 22.04 + ros-humble-libcamera works.
• Raspberry Pi 5 + Pi Camera v2 + Ubuntu 24.04 + ros-jazzy-libcamera does NOT work.

Getting the latter combination to work indeed requires custom build of libcamera, off the Raspberry Pi fork.

Could you point me to where the building of the ros-*-libcamera happens? Some adjustments would probably be required there to make things work out of the box.

[christianrauch]

More dependency complexity;

• Raspberry Pi 4 + Pi Camera v2 + Ubuntu 22.04 + ros-humble-libcamera works.
• Raspberry Pi 5 + Pi Camera v2 + Ubuntu 24.04 + ros-jazzy-libcamera does NOT work.

Getting the latter combination to work indeed requires custom build of libcamera, off the Raspberry Pi fork.

You are probably aware of the different "libcamera variants". The package in the ROS repo is built from the official upstream version at https://git.libcamera.org/libcamera/libcamera.git. But the development of the Raspberry Pi support happens in the @raspberrypi fork at https://github.com/raspberrypi/libcamera. That support is supposed to be merged into the upstream at some point, but this process seems to take a long time. See for example the patch series "Raspberry Pi: Add support for Pi 5 (PiSP)" by @naushir, which is still under review. That means that the upstream repo, and hence the bloomed package, usually only supports older Raspberry Pi models and camera modules.

Could you point me to where the building of the ros-*-libcamera happens? Some adjustments would probably be required there to make things work out of the box.

That package is "bloomed" using ros-infrastructure/bloom#691 directly on the upstream repo. This generates a Debian package specification, stores it in the release repo at https://github.com/ros2-gbp/libcamera-release, and then the ROS build farm is generating the binary package from this specification. You can also use bloom locally to generate and build the Debian package: https://docs.ros.org/en/jazzy/How-To-Guides/Building-a-Custom-Deb-Package.html.

But if you want Raspberry Pi 5 support out-of-the-box in the upstream repo and hence the ROS repo, you should nag @naushir to get the Raspberry Pi 5 patches merged upstream :-)

[larsll]

But if you want Raspberry Pi 5 support out-of-the-box in the upstream repo and hence the ROS repo, you should nag @naushir to get the Raspberry Pi 5 patches merged upstream :-)

Thanks for the references. I will have a look. Once I get serious about doing Jazzy + Pi 5 support I might just build a drop-in replacement package for ros-jazzy-libcamera and put it in our APT repository together with our other patched community work for DeepRacer.

[christianrauch]

Thanks for your work. I am OK with the general idea and its implementation via the raw stream configuration. But there are still some ways to optimise the code to make this better maintainable in the future.

[larsll]

Updated as per request - left it for you to resolve the comments

[christianrauch]

Almost there. I just left comments about the wording in the documentation and the use of roles.front(). Also, can you squash and rebase your commits? I would like to keep the git history clean of "fixup" commits.

[larsll]

All comments built in. Rebased (I normally do it via the built-in GitHub functionality when merging, this is why I did not do it manually).

Final open question is #100 (comment)

[christianrauch]

In get_sensor_format, we have to either handle all the exceptions and give meaningful messages to the users, or not handle them and let the process quit with the default message of that unhandled exception.

[christianrauch]

Looks good now. Thanks a lot.