Large messages (e.g. point clouds) are silently dropped

[roehling]

Generated by Generative AI

No response

Operating System:

Linux roehling 6.11.0-29-generic #29~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 26 14:16:59 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

ROS version or commit hash:

jazzy, rolling

RMW implementation (if applicable):

rmw_zenoh_cpp

RMW Configuration (if applicable):

default

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

ros2 doctor --report

   NETWORK CONFIGURATION
inet         : 127.0.0.1
inet4        : ['127.0.0.1']
inet6        : ['::1']
netmask      : 255.0.0.0
device       : lo
flags        : 73<RUNNING,LOOPBACK,UP>
mtu          : 65536
inet         : 128.x.y.z
inet4        : ['128.x.y.z']
ether        : 4c:d7:17:7f:27:0d
inet6        : ['fe80::9bab:2c04:2217:964%enp0s31f6']
netmask      : 255.255.255.0
device       : enp0s31f6
flags        : 4163<RUNNING,BROADCAST,MULTICAST,UP>
mtu          : 1500
broadcast    : 128.x.y.255
inet         : 172.17.0.1
inet4        : ['172.17.0.1']
ether        : 02:42:d1:f7:34:c0
netmask      : 255.255.0.0
device       : docker0
flags        : 4099<BROADCAST,MULTICAST,UP>
mtu          : 1500
broadcast    : 172.17.255.255

   PACKAGE VERSIONS
zenoh_test                                : latest=N/A, local=1.0.0
rosidl_generator_cpp                      : latest=4.6.5, local=4.6.5
rmw_cyclonedds_cpp                        : latest=2.2.3, local=2.2.3
rmw_implementation                        : latest=2.15.5, local=2.15.5
examples_rclcpp_minimal_action_client     : latest=0.19.5, local=0.19.5
console_bridge_vendor                     : latest=1.7.1, local=1.7.1
tf_transformations                        : latest=1.1.0, local=1.1.0
pcl_conversions                           : latest=2.6.2, local=2.6.2
python_cmake_module                       : latest=0.11.1, local=0.11.1
launch                                    : latest=3.4.5, local=3.4.5
tf2_sensor_msgs                           : latest=0.36.12, local=0.36.12
trajectory_msgs                           : latest=5.3.6, local=5.3.6
rcl_yaml_param_parser                     : latest=9.2.7, local=9.2.7
class_loader                              : latest=2.7.0, local=2.7.0
ros_workspace                             : latest=1.0.3, local=1.0.3
ros2bag                                   : latest=0.26.7, local=0.26.7
sros2                                     : latest=0.13.3, local=0.13.3
ament_lint_cmake                          : latest=0.17.2, local=0.17.2
pcl_ros                                   : latest=2.6.2, local=2.6.2
ament_cmake_include_directories           : latest=2.5.4, local=2.5.4
yaml_cpp_vendor                           : latest=9.0.1, local=9.0.1
rosidl_typesupport_cpp                    : latest=3.2.2, local=3.2.2
ros_environment                           : latest=4.2.1, local=4.2.1
fkie_mas_msgs                             : latest=N/A, local=3.2.0
ament_cmake_test                          : latest=2.5.4, local=2.5.4
rosidl_generator_type_description         : latest=4.6.5, local=4.6.5
gz_math_vendor                            : latest=0.0.8, local=0.0.8
ament_index_python                        : latest=1.8.1, local=1.8.1
resource_retriever                        : latest=3.4.4, local=3.4.4
ament_cmake_google_benchmark              : latest=2.5.4, local=2.5.4
robot_state_publisher                     : latest=3.3.3, local=3.3.3
ros2param                                 : latest=0.32.4, local=0.32.4
rosidl_default_runtime                    : latest=1.6.0, local=1.6.0
rviz_rendering                            : latest=14.1.12, local=14.1.12
tf2_ros                                   : latest=0.36.12, local=0.36.12
compressed_depth_image_transport          : latest=4.0.4, local=4.0.4
ros2launch                                : latest=0.26.8, local=0.26.8
ament_xmllint                             : latest=0.17.2, local=0.17.2
launch_testing                            : latest=3.4.5, local=3.4.5
rpyutils                                  : latest=0.4.1, local=0.4.1
ros2lifecycle                             : latest=0.32.4, local=0.32.4
orocos_kdl_vendor                         : latest=0.5.1, local=0.5.1
tf2_kdl                                   : latest=0.36.12, local=0.36.12
rosbag2_transport                         : latest=0.26.7, local=0.26.7
grid_map_msgs                             : latest=2.2.1, local=2.2.1
ament_cmake_libraries                     : latest=2.5.4, local=2.5.4
visualization_msgs                        : latest=5.3.6, local=5.3.6
ament_cmake_export_targets                : latest=2.5.4, local=2.5.4
rviz_ogre_vendor                          : latest=14.1.12, local=14.1.12
sros2_cmake                               : latest=0.13.3, local=0.13.3
rosidl_adapter                            : latest=4.6.5, local=4.6.5
ament_pep257                              : latest=0.17.2, local=0.17.2
point_cloud_transport                     : latest=4.0.4, local=4.0.4
libcurl_vendor                            : latest=3.4.4, local=3.4.4
rosidl_dynamic_typesupport_fastrtps       : latest=0.1.0, local=0.1.0
map_msgs                                  : latest=2.4.1, local=2.4.1
rosbag2_storage                           : latest=0.26.7, local=0.26.7
gps_tools                                 : latest=2.1.0, local=2.1.0
nav_msgs                                  : latest=5.3.6, local=5.3.6
ros2doctor                                : latest=0.32.4, local=0.32.4
fastrtps_cmake_module                     : latest=3.6.1, local=3.6.1
fkie_mas_pylib                            : latest=N/A, local=4.4.1
stereo_msgs                               : latest=5.3.6, local=5.3.6
rosbag2_py                                : latest=0.26.7, local=0.26.7
libstatistics_collector                   : latest=1.7.4, local=1.7.4
ros2node                                  : latest=0.32.4, local=0.32.4
rviz2                                     : latest=14.1.12, local=14.1.12
actionlib_msgs                            : latest=5.3.6, local=5.3.6
pcl_msgs                                  : latest=1.0.0, local=1.0.0
rosidl_cli                                : latest=4.6.5, local=4.6.5
gps_msgs                                  : latest=2.1.0, local=2.1.0
rmw_fastrtps_shared_cpp                   : latest=8.4.2, local=8.4.2
rosidl_typesupport_introspection_cpp      : latest=4.6.5, local=4.6.5
ortools_vendor                            : latest=9.9.0, local=9.9.0
rosidl_typesupport_c                      : latest=3.2.2, local=3.2.2
lifecycle_msgs                            : latest=2.0.2, local=2.0.2
geometry2                                 : latest=0.36.12, local=0.36.12
eigen3_cmake_module                       : latest=0.3.0, local=0.3.0
ament_cmake_gen_version_h                 : latest=2.5.4, local=2.5.4
rosidl_typesupport_fastrtps_c             : latest=3.6.1, local=3.6.1
urdf_parser_plugin                        : latest=2.10.0, local=2.10.0
rosbag2_interfaces                        : latest=0.26.7, local=0.26.7
rosbag2                                   : latest=0.26.7, local=0.26.7
message_filters                           : latest=4.11.6, local=4.11.6
ros2cli_common_extensions                 : latest=0.3.0, local=0.3.0
ros2multicast                             : latest=0.32.4, local=0.32.4
rosidl_typesupport_introspection_c        : latest=4.6.5, local=4.6.5
example_interfaces                        : latest=0.12.0, local=0.12.0
rosidl_typesupport_interface              : latest=4.6.5, local=4.6.5
urdf                                      : latest=2.10.0, local=2.10.0
rcl_logging_interface                     : latest=3.1.1, local=3.1.1
rosbag2_compression                       : latest=0.26.7, local=0.26.7
async_web_server_cpp                      : latest=2.0.0, local=2.0.0
rviz_assimp_vendor                        : latest=14.1.12, local=14.1.12
composition_interfaces                    : latest=2.0.2, local=2.0.2
ament_cmake_export_interfaces             : latest=2.5.4, local=2.5.4
ament_cmake_gmock                         : latest=2.5.4, local=2.5.4
rosidl_pycommon                           : latest=4.6.5, local=4.6.5
launch_ros                                : latest=0.26.8, local=0.26.8
camera_info_manager                       : latest=5.1.7, local=5.1.7
rcpputils                                 : latest=2.11.2, local=2.11.2
sensor_msgs                               : latest=5.3.6, local=5.3.6
rclcpp_action                             : latest=28.1.10, local=28.1.10
sqlite3_vendor                            : latest=0.26.7, local=0.26.7
ament_cmake_lint_cmake                    : latest=0.17.2, local=0.17.2
tf2_py                                    : latest=0.36.12, local=0.36.12
zstd_image_transport                      : latest=4.0.4, local=4.0.4
rcl_lifecycle                             : latest=9.2.7, local=9.2.7
liblz4_vendor                             : latest=0.26.7, local=0.26.7
rmw_implementation_cmake                  : latest=7.3.2, local=7.3.2
common_interfaces                         : latest=5.3.6, local=5.3.6
rclcpp                                    : latest=28.1.10, local=28.1.10
rosbag2_compression_zstd                  : latest=0.26.7, local=0.26.7
fkie_mas_daemon                           : latest=N/A, local=4.5.10
rclcpp_lifecycle                          : latest=28.1.10, local=28.1.10
shape_msgs                                : latest=5.3.6, local=5.3.6
ament_cmake_export_definitions            : latest=2.5.4, local=2.5.4
rclpy                                     : latest=7.1.4, local=7.1.4
theora_image_transport                    : latest=4.0.4, local=4.0.4
ament_cmake_pytest                        : latest=2.5.4, local=2.5.4
statistics_msgs                           : latest=2.0.2, local=2.0.2
grid_map_ros                              : latest=2.2.1, local=2.2.1
tf2_ros_py                                : latest=0.36.12, local=0.36.12
image_view                                : latest=5.0.11, local=5.0.11
ament_cpplint                             : latest=0.17.2, local=0.17.2
tf2_bullet                                : latest=0.36.12, local=0.36.12
rosbag2_storage_mcap                      : latest=0.26.7, local=0.26.7
tf2                                       : latest=0.36.12, local=0.36.12
osrf_pycommon                             : latest=2.1.6, local=2.1.6
grid_map_core                             : latest=2.2.1, local=2.2.1
examples_rclcpp_minimal_action_server     : latest=0.19.5, local=0.19.5
zstd_vendor                               : latest=0.26.7, local=0.26.7
usb_cam                                   : latest=0.8.1, local=0.8.1
geographic_msgs                           : latest=1.0.6, local=1.0.6
rcl_action                                : latest=9.2.7, local=9.2.7
ros2run                                   : latest=0.32.4, local=0.32.4
cv_bridge                                 : latest=4.1.0, local=4.1.0
ament_uncrustify                          : latest=0.17.2, local=0.17.2
std_srvs                                  : latest=5.3.6, local=5.3.6
domain_coordinator                        : latest=0.12.0, local=0.12.0
ros_base                                  : latest=0.11.0, local=0.11.0
ament_cmake_gtest                         : latest=2.5.4, local=2.5.4
compressed_image_transport                : latest=4.0.4, local=4.0.4
ament_flake8                              : latest=0.17.2, local=0.17.2
rmw                                       : latest=7.3.2, local=7.3.2
rcutils                                   : latest=6.7.2, local=6.7.2
rosidl_core_runtime                       : latest=0.2.0, local=0.2.0
rosidl_default_generators                 : latest=1.6.0, local=1.6.0
rclcpp_components                         : latest=28.1.10, local=28.1.10
ros_core                                  : latest=0.11.0, local=0.11.0
rosidl_runtime_py                         : latest=0.13.1, local=0.13.1
launch_testing_ament_cmake                : latest=3.4.5, local=3.4.5
libyaml_vendor                            : latest=1.6.3, local=1.6.3
image_transport                           : latest=5.1.7, local=5.1.7
rosidl_typesupport_fastrtps_cpp           : latest=3.6.1, local=3.6.1
rcl                                       : latest=9.2.7, local=9.2.7
joy                                       : latest=3.3.0, local=3.3.0
ament_cmake_ros                           : latest=0.12.0, local=0.12.0
gz_cmake_vendor                           : latest=0.0.10, local=0.0.10
launch_yaml                               : latest=3.4.5, local=3.4.5
gz_utils_vendor                           : latest=0.0.5, local=0.0.5
fkie_mas_discovery                        : latest=N/A, local=4.1.4
ros2pkg                                   : latest=0.32.4, local=0.32.4
ament_cmake_target_dependencies           : latest=2.5.4, local=2.5.4
rosgraph_msgs                             : latest=2.0.2, local=2.0.2
mcap_vendor                               : latest=0.26.7, local=0.26.7
ament_cmake_copyright                     : latest=0.17.2, local=0.17.2
rosidl_generator_c                        : latest=4.6.5, local=4.6.5
tf2_geometry_msgs                         : latest=0.36.12, local=0.36.12
launch_xml                                : latest=3.4.5, local=3.4.5
action_msgs                               : latest=2.0.2, local=2.0.2
ament_lint_common                         : latest=0.17.2, local=0.17.2
nav2_msgs                                 : latest=1.3.7, local=1.3.7
rviz_common                               : latest=14.1.12, local=14.1.12
ament_cmake_version                       : latest=2.5.4, local=2.5.4
tf2_msgs                                  : latest=0.36.12, local=0.36.12
ament_lint                                : latest=0.17.2, local=0.17.2
python_orocos_kdl_vendor                  : latest=0.5.1, local=0.5.1
ament_cmake_python                        : latest=2.5.4, local=2.5.4
zenoh_cpp_vendor                          : latest=0.2.5, local=0.2.5
tracetools                                : latest=8.2.3, local=8.2.3
ament_cmake_export_include_directories    : latest=2.5.4, local=2.5.4
rviz_default_plugins                      : latest=14.1.12, local=14.1.12
builtin_interfaces                        : latest=2.0.2, local=2.0.2
grid_map_cv                               : latest=2.2.1, local=2.2.1
ament_cppcheck                            : latest=0.17.2, local=0.17.2
ament_cmake_export_libraries              : latest=2.5.4, local=2.5.4
rosidl_parser                             : latest=4.6.5, local=4.6.5
spdlog_vendor                             : latest=1.6.1, local=1.6.1
rosbag2_storage_default_plugins           : latest=0.26.7, local=0.26.7
ament_cmake_cppcheck                      : latest=0.17.2, local=0.17.2
rmw_fastrtps_cpp                          : latest=8.4.2, local=8.4.2
rosidl_runtime_cpp                        : latest=4.6.5, local=4.6.5
rosidl_core_generators                    : latest=0.2.0, local=0.2.0
rosidl_generator_py                       : latest=0.22.1, local=0.22.1
interactive_markers                       : latest=2.5.4, local=2.5.4
ament_package                             : latest=0.16.4, local=0.16.4
rosidl_dynamic_typesupport                : latest=0.1.2, local=0.1.2
service_msgs                              : latest=2.0.2, local=2.0.2
ament_cmake_export_link_flags             : latest=2.5.4, local=2.5.4
rosbag2_storage_sqlite3                   : latest=0.26.7, local=0.26.7
diagnostic_msgs                           : latest=5.3.6, local=5.3.6
rcl_interfaces                            : latest=2.0.2, local=2.0.2
tf2_eigen                                 : latest=0.36.12, local=0.36.12
tf2_tools                                 : latest=0.36.12, local=0.36.12
ament_lint_auto                           : latest=0.17.2, local=0.17.2
simple_actions                            : latest=0.4.0, local=0.4.0
tf2_eigen_kdl                             : latest=0.36.12, local=0.36.12
rcl_logging_spdlog                        : latest=3.1.1, local=3.1.1
keyboard_handler                          : latest=0.3.1, local=0.3.1
ament_cmake_cpplint                       : latest=0.17.2, local=0.17.2
sensor_msgs_py                            : latest=5.3.6, local=5.3.6
type_description_interfaces               : latest=2.0.2, local=2.0.2
filters                                   : latest=2.2.1, local=2.2.1
sdl2_vendor                               : latest=3.3.0, local=3.3.0
pluginlib                                 : latest=5.4.2, local=5.4.2
rosbag2_cpp                               : latest=0.26.7, local=0.26.7
rmw_zenoh_cpp                             : latest=0.2.5, local=0.2.5
ros2action                                : latest=0.32.4, local=0.32.4
ros2service                               : latest=0.32.4, local=0.32.4
ament_cmake_flake8                        : latest=0.17.2, local=0.17.2
ament_copyright                           : latest=0.17.2, local=0.17.2
launch_testing_ros                        : latest=0.26.8, local=0.26.8
uncrustify_vendor                         : latest=3.0.1, local=3.0.1
ros2topic                                 : latest=0.32.4, local=0.32.4
tinyxml2_vendor                           : latest=0.9.1, local=0.9.1
ament_cmake_auto                          : latest=2.5.4, local=2.5.4
ament_cmake_uncrustify                    : latest=0.17.2, local=0.17.2
velodyne_msgs                             : latest=2.5.1, local=2.5.1
ament_cmake_pep257                        : latest=0.17.2, local=0.17.2
ament_index_cpp                           : latest=1.8.1, local=1.8.1
ros2interface                             : latest=0.32.4, local=0.32.4
rosidl_runtime_c                          : latest=4.6.5, local=4.6.5
image_transport_plugins                   : latest=4.0.4, local=4.0.4
ament_cmake_xmllint                       : latest=0.17.2, local=0.17.2
pybind11_vendor                           : latest=3.1.3, local=3.1.3
camera_calibration_parsers                : latest=5.1.7, local=5.1.7
rmw_dds_common                            : latest=3.1.0, local=3.1.0
kdl_parser                                : latest=2.11.0, local=2.11.0
ament_cmake_core                          : latest=2.5.4, local=2.5.4
laser_geometry                            : latest=2.7.1, local=2.7.1
ament_cmake                               : latest=2.5.4, local=2.5.4
ros2component                             : latest=0.32.4, local=0.32.4
rosidl_cmake                              : latest=4.6.5, local=4.6.5
geometry_msgs                             : latest=5.3.6, local=5.3.6
unique_identifier_msgs                    : latest=2.5.0, local=2.5.0
ament_cmake_export_dependencies           : latest=2.5.4, local=2.5.4
ros2cli                                   : latest=0.32.4, local=0.32.4
std_msgs                                  : latest=5.3.6, local=5.3.6

   PLATFORM INFORMATION
system           : Linux
platform info    : Linux-6.11.0-29-generic-x86_64-with-glibc2.39
release          : 6.11.0-29-generic
processor        : x86_64

   QOS COMPATIBILITY LIST
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : _ros2cli_daemon_0_3f0cd82407344117b3dc9251336b37c0
subscriber node         : pong_node
compatibility status    : OK
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : _ros2cli_daemon_0_3f0cd82407344117b3dc9251336b37c0
subscriber node         : pingstat_node
compatibility status    : OK
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : pong_node
subscriber node         : pong_node
compatibility status    : OK
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : pong_node
subscriber node         : pingstat_node
compatibility status    : OK
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : pingstat_node
subscriber node         : pong_node
compatibility status    : OK
topic [type]            : /parameter_events [rcl_interfaces/msg/ParameterEvent]
publisher node          : pingstat_node
subscriber node         : pingstat_node
compatibility status    : OK
topic [type]            : /ping [zenoh_test/msg/Payload]
publisher node          : pingstat_node
subscriber node         : pong_node
compatibility status    : OK
topic [type]            : /pong [zenoh_test/msg/Ack]
publisher node          : pong_node
subscriber node         : pingstat_node
compatibility status    : OK

   RMW MIDDLEWARE
middleware name    : rmw_zenoh_cpp

   ROS 2 INFORMATION
distribution name      : jazzy
distribution type      : ros2
distribution status    : active
release platforms      : {'debian': ['bookworm'], 'rhel': ['9'], 'ubuntu': ['noble']}

   TOPIC LIST
topic               : /ping
publisher count     : 1
subscriber count    : 1
topic               : /pong
publisher count     : 1
subscriber count    : 1

Steps to reproduce issue

1. Publish a large message that serializes to more than 200 MB with publisher and subscriber both on the same host.

Expected behavior

The message should be received by the subscriber.

Actual behavior

Starting at around 200 MB serialized size, the message transmission becomes increasingly unreliable. Messages around 300 MB have a ~50% chance to go through. Messages larger than 350 MB are virtually never transmitted successfully.

Additional information

I stumbled upon this problem while working with a high resolution LiDAR scanner for 360° panoramic views. Out-of-the-box FastDDS and CycloneDDS behave even worse than Zenoh, but the FastDDS Large Data Mode (export FASTDDS_BUILTIN_TRANSPORTS=LARGE_DATA) works fine.

[JEnoch]

For large messages, a shared-memory mode is in preparation: #363 for rolling or #656 for jazzy. You could give it a try and give us your feedback.
You would need to set shared_memory/enabled=true in the configuration and to define the ZENOH_SHM_ALLOC_SIZE environment variable to a large size (in bytes, the default is 1610241024).

[roehling]

Thanks for the reply, and looking forward to having shared memory message transport as an option!
I ran a systematic test with payload sizes 1M, 2M, 3M, … up to 500M. Each payload size was transmitted ~500 times in total, and I recorded the percentage of received messages:

As I already observed initially, there is no hard size limit but a gradual degradation. Considering that the message is sent via loopback, there should not be any packet loss on the link layer at all (which would explain the pattern). The only other explanation I can think of is some sort of data race condition that makes it increasingly likely that the message becomes corrupted and is dropped. Note that at this point, messages might have become large enough that they can no longer be transmitted in a single CPU scheduling time slice, so this might trigger latent thread synchronization bugs which do not occur with shorter messages.

[wawatt]

Thanks for the reply, and looking forward to having shared memory message transport as an option!感谢回复，期待共享内存消息传输作为一个选项！ I ran a systematic test with payload sizes 1M, 2M, 3M, … up to 500M. Each payload size was transmitted ~500 times in total, and I recorded the percentage of received messages:我用 1M、2M、3M……直到 500M 的负载大小进行了一项系统测试。每个负载大小总共传输了约 500 次，我记录了接收消息的百分比：

As I already observed initially, there is no hard size limit but a gradual degradation. Considering that the message is sent via loopback, there should not be any packet loss on the link layer at all (which would explain the pattern). The only other explanation I can think of is some sort of data race condition that makes it increasingly likely that the message becomes corrupted and is dropped. Note that at this point, messages might have become large enough that they can no longer be transmitted in a single CPU scheduling time slice, so this might trigger latent thread synchronization bugs which do not occur with shorter messages.正如我最初观察到的，没有硬性大小限制，而是逐渐的退化。考虑到消息是通过回环发送的，链路层不应该有任何丢包（这可以解释该模式）。我能想到的另一个解释是某种数据竞争条件，这使得消息损坏并被丢弃的可能性越来越大。请注意，此时消息可能已经足够大，以至于无法在单个 CPU 调度时间片中传输，这可能触发潜在的线程同步错误，而这些错误在较短的消息中不会出现。

how to measure,what is your tool software

[roehling]

I wrote two simple nodes with a custom byte array message. One node cycles through different array lengths and publishes messages, the other subscribes to the topic and answers with on a different topic if it received an array and how large it was.

Now you just need to keep count of the messages that went through successfully, and you end up with a graph as above.

[wawatt]

I wrote two simple nodes with a custom byte array message. One node cycles through different array lengths and publishes messages, the other subscribes to the topic and answers with on a different topic if it received an array and how large it was.

Now you just need to keep count of the messages that went through successfully, and you end up with a graph as above.

How do you make sure that memory sharing is being used?

[roehling]

I didn't, this is just the default config

[JEnoch]

Considering that the message is sent via loopback, there should not be any packet loss on the link layer at all (which would explain the pattern). The only other explanation I can think of is some sort of data race condition that makes it increasingly likely that the message becomes corrupted and is dropped

No, the explanation is likely that you're not using RELIABLE + KEEP_ALL QoS. Thus rmw_zenoh applies a CongestionControl::Drop strategy that makes it drop messages it can't push to the network in a configurable amount of time.
By default this value is 1ms which is to low to publish a 350MB payload.

You can try 2 different changes:

1. Change your Publisher QoS to be RELIABLE + KEEP_ALL. rmw_zenoh will apply a CongestionControl::Block` strategy that will block each publication until it's fully pushed on the network. For a periodic Publisher, this might slow down its frequency.
2. Run your Publisher with ZENOH_CONFIG_OVERRIDE='transport/link/tx/queue/congestion_control/drop/wait_before_drop=1000000' environment variable. It will wait longer before to drop the message in case of congestion.

[roehling]

@JEnoch

1. Change your Publisher QoS to be RELIABLE + KEEP_ALL. rmw_zenoh will apply a CongestionControl::Block` strategy that will block each publication until it's fully pushed on the network. For a periodic Publisher, this might slow down its frequency.

I tried this and it worked! Thank you for pointing me in the right direction!

I've seen the blog post on reliability, scalability, and congestion control, which is informative but does not specify how ROS quality of service maps to Zenoh. Is there a different place where I can find this?

[YuanYuYuan]

Hi @roehling, we describe the details in this doc.

[mcp-ats]

I had similar problems when trying out rmw_zenoh 6MB imags and I can confirm that changing the drop parameter from congestion control makes it possible to leverage the full bandwidth of a network interface.

Thanks @JEnoch

[YuanYuYuan]

Glad to hear that it works for you. Let's close this issue.