Ashraf/glsl pointcloud

[ashrafk93]

Following this PR
IntelRealSense/librealsense#13462 (comment)
and issue
#3206

While debugging pointcloud behavior with GLSL acceleration enabled, I noticed that in the RealSense Viewer, pointclouds were being generated and rendered correctly. However, in the ROS 2 wrapper (realsense2_camera), pointclouds would not appear under certain conditions — specifically when the parameter pointcloud.stream_profile was left at its default value: any (RS2_STREAM_ANY).

Digging deeper into the ROS pipeline, I observed that when stream_profile is set to any, the system is unable to find a valid texture frame to associate with the pointcloud.

realsense-ros/realsense2_camera/src/pointcloud_filter.cpp

Line 99 in 711d1f7

bool use_texture = texture_source_id != RS2_STREAM_ANY;

As a result GLSL-based pointcloud computation silently fails due to the lack of a texture stream.

Unlike CPU-based fallback implementations, GLSL requires a valid texture stream (e.g., color or IR) to drive the pointcloud generation pipeline.

This explains why realsense-viewer works while ros2 wrapper doesnt -- it explicitly assigns a texture stream.

To resolve this, I verified that explicitly setting the parameter pointcloud.stream_profile to a valid stream
successfully enables the pointcloud pipeline, even when GLSL acceleration is used (e.g.,):

1 → RS2_STREAM_DEPTH
2 → RS2_STREAM_COLOR
3 → RS2_STREAM_INFRARED

RS2_STREAM_DEPTH pointcloud

RS2_STREAM_COLOR pointcloud

RS2_STREAM_INFRARED pointcloud

[ashrafk93]

also the PR here:
IntelRealSense/librealsense#13462 (comment)

is a valid fallback to calculating point cloud when no texture is found
it Uses only depth intrinsics + the raw depth frame bypassing the need for a texture
and can be merged as well

[Nir-Az]

@remibettan can you please review and maybe try this code?

[remibettan]

It seems that you tell the user what is the error, and why the pointcloud cannot be displayed until the texture param is changed - great.
Let's change also the default value, so that the pointcloud will be displayed by default (I think COLOR looks great)

[ashrafk93]

the question is if we want to this merge this PR
#3356 (comment)

as default (same behavior as cpu - non glsl)
or just put a warning.
the first approach is more robust
@Nir-Az @remibettan

[ashrafk93]

@remibettan
providing more details on the issue
when glsl is not enabled everything is fine.
and whether pointcloud.stream_profile param provided or not the pointcloud data is never empty and there are fallbacks for all cases.

when using glsl there is an issue with stream_profile 0 and its confusing to users.

reproducing the issue:

build realsense ros branch development with glsl on cmake flag

 colcon build --cmake-args '-DBUILD_ACCELERATE_GPU_WITH_GLSL=ON' -DCMAKE_EXPORT_COMPILE_COMMANDS=ON

run realsense node without specifiying a stream profile (i.e default pointcloud.stream_profile when not provided is 0 -> STREAM_ANY)

source install/setup.bash
ros2 run realsense2_camera realsense2_camera_node --ros-args -p initial_reset:=True -p accelerate_gpu_with_glsl:=True -p pointcloud.enable:=True -p enable_color:=True -p enable_depth:=True -p enable_infra1:=True -p enable_infra2:=True -p clip_distance:=2.5 -p pointcloud.stream_filter:=0 --log-level info

you would see that point cloud data is empty

ros2 topic echo /camera/camera/depth/color/points

this PR (realsense ros repo) prints a warning to the user when this happens explaning what is the issue

and that he should specify the stream_profile param to something else than 0 (STREAM_ANY)

now regarding realsense fix PR that can complete this issue to make realsense ros utilize gpu and calculate pointcloud even with no stream_profile can be provided
this PR https://github.com/IntelRealSense/librealsense/pull/13462/files contributed by the public
handles the case where a stream_profile is not provided or 0.
and implements a new shader that can calculate the pointcloud without any texture (depth, color,infra)

we can just merge he warning msg for now and do some deeper tests on the realsense PR with the fallback suggestions

[remibettan]

one comment - great catch!