Issues with multicast discovery across hosts

[christian-rauch]

Operating System:

Ubuntu 24.04

ROS version or commit hash:

jazzy

RMW implementation (if applicable):

rmw_zenoh

RMW Configuration (if applicable):

Trying to use ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true'.

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

ros2 doc --report

usage: ros2 [-h] [--use-python-default-buffering]
            Call `ros2 <command> -h` for more detailed usage. ...
ros2: error: argument Call `ros2 <command> -h` for more detailed usage.: invalid choice: 'doc' (choose from 'action', 'bag', 'component', 'control', 'daemon', 'doctor', 'extension_points', 'extensions', 'interface', 'launch', 'lifecycle', 'multicast', 'node', 'param', 'pkg', 'run', 'security', 'service', 'topic', 'wtf')

Steps to reproduce issue

Follow the instructions for "Enable multicast scouting (disabled by default) that allows ROS2 nodes to discover each other without requiring a Zenoh router:".

Setup: PC A (inet 10.71.1.208/24) and B (inet 10.71.1.207/24) in the same subnet.

# A
export ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp talker

# B
export ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp listener

Expected behavior

The listener on B should receive messages.

Actual behavior

The talker on A complains about a missing router: [rmw_zenoh_cpp]: Unable to connect to a Zenoh router. Have you started a router with ros2 run rmw_zenoh_cpp rmw_zenohd? but then continues to Publishing: 'Hello World: 1' etc.

The listener on B does not receive messages and shows the periodic warning:

WARN net-0 ThreadId(02) zenoh::net::runtime::orchestrator: Unable to connect to any locator of scouted peer 8dfe45adee11bab2eedd745f7fa24506: [tcp/127.0.0.1:42649]

Additional information

No response

[Yadunund]

Hi @christian-rauch, similar to #625 can you confirm that both hosts are using the same version of rmw_zenoh?

We're currently running Zenoh in ROS 2 nightly CI jobs with multicast enabled although all processes are within the same host: https://github.com/ros2/ros_buildfarm_config/blob/4e73515e6648eaa84909b4ff1edc2fad0d4e4b2c/rolling/ci-nightly-zenoh.yaml#L7-L8. So the issue could be either version mismatch or something on the network interface side of things.

Regarding the warning about the router, this happens because by default rmw_zenoh sessions check for a router once before proceeding with initialization. This can be disabled by additionally setting ZENOH_ROUTER_CHECK_ATTEMPTS=-1. But this should not affect anything in your case.

[christian-rauch]

dpkg -l ros-jazzy-rmw-zenoh-cpp shows on both PCs 0.2.3-1noble.20250429.203124.

The network configuration is different, of course. PC A is connected via Wi-Fi and PC B is connected via Ethernet. PC B additionally has the docker0 network interface. Not sure if this makes a difference.

Can someone reproduce this issue on a "real" setup?

[JEnoch]

The instructions for "Enable multicast scouting" in the README are intended for the basic scenario where all Nodes are running on the same host.

By default, Nodes open a TCP port only on the loopback interface ("tcp/localhost:0"). Therefore, to enable Node-to-Node communication between two Nodes on different hosts, you need to modify the listen/endpoints of at least one Node.

For instance to make the node on host A to also listen on its LAN interface :

# A
export ZENOH_ROUTER_CHECK_ATTEMPTS=-1
export ZENOH_CONFIG_OVERRIDE='listen/endpoints=["tcp/localhost:0", "tcp/10.71.1.208:0"];scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp talker

# B
export ZENOH_ROUTER_CHECK_ATTEMPTS=-1
export ZENOH_CONFIG_OVERRIDE='scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp listener

Or simply to make all the nodes to listen on all available interfaces (including the loopback):

# A
export ZENOH_ROUTER_CHECK_ATTEMPTS=-1
export ZENOH_CONFIG_OVERRIDE='listen/endpoints=["tcp/0.0.0.0:0"];scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp talker

# B
export ZENOH_ROUTER_CHECK_ATTEMPTS=-1
export ZENOH_CONFIG_OVERRIDE='listen/endpoints=["tcp/0.0.0.0:0"];scouting/multicast/enabled=true'
ros2 run demo_nodes_cpp listener

I will create a PR to clarify the README, and give detailed explanations in #395 .