rmw_zenoh vs rmw_cyclonedds performance

[ciandonovan]

Setup

• Official Dockerhub ROS 2 Jazzy images
• Two separate Podman containers with shared host and IPC namespaces
• Using ROS 2 testing repo
• rmw_zenoh 0.2.6
• cyclonedds 2.2.3
• rtprio permission for realtime threads

Container 1

ros2 run v4l2_camera v4l2_camera_node

[INFO] [1756389373.858345646] [v4l2_camera]: Driver: uvcvideo
[INFO] [1756389373.858431858] [v4l2_camera]: Version: 396808
[INFO] [1756389373.858444642] [v4l2_camera]: Device: Laptop Camera: Laptop Camera
[INFO] [1756389373.858455522] [v4l2_camera]: Location: usb-0000:c1:00.4-1
[INFO] [1756389373.858466062] [v4l2_camera]: Capabilities:
[INFO] [1756389373.858481301] [v4l2_camera]:   Read/write: NO
[INFO] [1756389373.858493454] [v4l2_camera]:   Streaming: YES
[INFO] [1756389373.858507660] [v4l2_camera]: Current pixel format: YUYV @ 640x480
[INFO] [1756389373.858597108] [v4l2_camera]: Available pixel formats: 
[INFO] [1756389373.858609972] [v4l2_camera]:   MJPG - Motion-JPEG
[INFO] [1756389373.858620182] [v4l2_camera]:   YUYV - YUYV 4:2:2
[INFO] [1756389373.858629920] [v4l2_camera]: Available controls: 
[INFO] [1756389373.858644257] [v4l2_camera]:   Brightness (1) = 0
[INFO] [1756389373.858656540] [v4l2_camera]:   Contrast (1) = 0
[INFO] [1756389373.858667751] [v4l2_camera]:   Saturation (1) = 64
[INFO] [1756389373.858678902] [v4l2_camera]:   Hue (1) = 0
[INFO] [1756389373.858694341] [v4l2_camera]:   White Balance, Automatic (2) = 1
[INFO] [1756389373.858710451] [v4l2_camera]:   Gamma (1) = 100
[INFO] [1756389373.858721983] [v4l2_camera]:   Power Line Frequency (3) = 2
[INFO] [1756389373.859224136] [v4l2_camera]:   White Balance Temperature (1) = 4600 [inactive]
[INFO] [1756389373.859241228] [v4l2_camera]:   Sharpness (1) = 2
[INFO] [1756389373.859253210] [v4l2_camera]:   Backlight Compensation (1) = 3
[ERROR] [1756389373.859271625] [v4l2_camera]: Failed getting value for control 10092545: Permission denied (13); returning 0!
[INFO] [1756389373.859329844] [v4l2_camera]:   Camera Controls (6) = 0
[INFO] [1756389373.859341937] [v4l2_camera]:   Auto Exposure (3) = 3
[INFO] [1756389373.859833229] [v4l2_camera]:   Exposure Time, Absolute (1) = 156 [inactive]
[INFO] [1756389373.859846474] [v4l2_camera]:   Exposure, Dynamic Framerate (2) = 0
[WARN] [1756389373.860343718] [v4l2_camera]: Control type not currently supported: 6, for control: Camera Controls
[INFO] [1756389373.860480865] [v4l2_camera]: Starting camera
[WARN] [1756389374.101598880] [v4l2_camera]: Image encoding not the same as requested output, performing possibly slow conversion: yuv422_yuy2 => rgb8
[INFO] [1756389374.106154904] [v4l2_camera]: using default calibration URL
[INFO] [1756389374.106254401] [v4l2_camera]: camera calibration URL: file:///root/.ros/camera_info/laptop_camera:_laptop_camera.yaml
[ERROR] [1756389374.106342356] [camera_calibration_parsers]: Unable to open camera calibration file [/root/.ros/camera_info/laptop_camera:_laptop_camera.yaml]
[WARN] [1756389374.106406777] [v4l2_camera]: Camera calibration file /root/.ros/camera_info/laptop_camera:_laptop_camera.yaml not found

Container 2

ros2 topic hz /image_raw

Results

CPU is total average process CPU usage. Useful as relative measure only.

rmw_zenoh (shm: off)

/opt/ros/jazzy/lib/v4l2_camera/v4l2_camera_node                CPU 44.9
/usr/bin/python3 /opt/ros/jazzy/bin/ros2 topic hz /image_raw   CPU 14.2

average rate: 29.551
    min: 0.028s max: 0.040s std dev: 0.00217s window: 10000

rmw_zenoh (shm:on)

/opt/ros/jazzy/lib/v4l2_camera/v4l2_camera_node                CPU 40.7
/usr/bin/python3 /opt/ros/jazzy/bin/ros2 topic hz /image_raw   CPU 10.7


average rate: 29.551
	min: 0.028s max: 0.040s std dev: 0.00209s window: 10000

rmw_cyclonedds (shm:N/A)

/opt/ros/jazzy/lib/v4l2_camera/v4l2_camera_node                CPU 41.6
/usr/bin/python3 /opt/ros/jazzy/bin/ros2 topic hz /image_raw   CPU 5.1

average rate: 29.551
	min: 0.029s max: 0.040s std dev: 0.00205s window: 10000

Bonus Torture Test

for i in $(seq 1 100); do ros2 topic hz /image_raw > /dev/null 2>&1 & sleep 0.1; done

Running 100 such subscribers with rmw_cyclonedds works fine, slightly reduced topic hz, but with rmw_zenoh many of the 100 immediately abort with both shared-memory and without

Comments

From this admittedly rough benchmark, it seems rmw_cyclonedds over the localhost loopback has significantly lower overhead on the subscriber side, and surprisingly slightly tighter standard deviation/jitter than even rmw_zenoh with shared-memory enabled.

[EricX-Zhao]

For rmw_zenoh shared memory, there are a few points to check:

1. If you installed rmw_zenoh from the ROS jazzy package and then compiled the latest rmw_zenoh yourself, please ensure that the environment variables on all routers and nodes are set correctly (in LD_LIBRARY_PATH, the compiled rmw_zenoh lib path should be placed before the installed one).
2. Confirm that shared memory transfer is being used. For your application (with image messages), you can use ifconfig to see if the lo network card traffic is increasing rapidly.

After confirming the above two points, compare CPU overhead. In my tests, rmw_zenoh with shm performs the best.

[ciandonovan]

For rmw_zenoh shared memory, there are a few points to check:

1. If you installed rmw_zenoh from the ROS jazzy package and then compiled the latest rmw_zenoh yourself, please ensure that the environment variables on all routers and nodes are set correctly (in LD_LIBRARY_PATH, the compiled rmw_zenoh lib path should be placed before the installed one).

I'm using the builds from the ROS 2 Jazzy Testing repo, which as of last week has rmw_zenoh 0.2.6 and zenoh 1.5, full support for shared-memory, which was enabled in my test.

https://repo.ros2.org/status_page/ros_jazzy_default.html?q=zenoh

2. Confirm that shared memory transfer is being used. For your application (with image messages), you can use `ifconfig` to see if the `lo` network card traffic is increasing rapidly.

Yes I confirmed using sudo iftop -i lo. Also inspected /dev/shm.

After confirming the above two points, compare CPU overhead. In my tests, rmw_zenoh with shm performs the best.

rmw_zenoh with shm does perform better intra-host than without shm on the loopback, but not better than rmw_cyclonedds as shown in those results. I appreciate that CycloneDDS is a much older and mature project which brings with it years of optimisation and tuning—I'd like to see rmw_zenoh progress likewise.

For context we run an AMR with 5 RealSense depth cameras, 4 RGB cameras, LIDAR, and many more sensors on an Embedded Ryzen x86 fanless PC—CPU budget is something we are conscious of. Otherwise our testing with Zenoh has been great, and looking forward to being able to use it as our default RMW going forward. The ability to bridge remotely over VPN using routers is especially attractive vs CycloneDDS.

[Yadunund]

@ciandonovan thanks for raising this discussion and sharing your findings. And thank you @EricX-Zhao for the extra pointers.

rmw_zenoh with shm does perform better intra-host than without shm on the loopback, but not better than rmw_cyclonedds as shown in those results.

It's good to know that you found a performance improvement with SHM enabled.

I think what would be helpful if you could help profile the subscription. Looking at FlameGraphs or equivalent will be helpful to understand where extra CPU cycles are being used with rmw_zenoh.