Fix [rmw_cyclonedds_cpp]: rmw_create_node: failed to create domain, error

[xmfcx]

• Parent Issue: Support ROS 2 Jazzy Jalisco #6695

Error

Run planning sim demo in ROS 2 Jazzy.

[component_container_mt-16] 1769256670.700802 [0] component_: Failed to find a free participant index for domain 0
[component_container_mt-16] [ERROR 1769256670.700870533] [rmw_cyclonedds_cpp]: rmw_create_node: failed to create domain, error Error (check_create_domain() at ./src/rmw_node.cpp:1242)
[component_container_mt-16] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[component_container_mt-16]   what():  failed to initialize rcl node: error not set, at ./src/rcl/node.c:252
[component_container_mt-36] 1769256670.709949 [0] component_: Failed to find a free participant index for domain 0
[component_container_mt-36] [ERROR 1769256670.710007886] [rmw_cyclonedds_cpp]: rmw_create_node: failed to create domain, error Error (check_create_domain() at ./src/rmw_node.cpp:1242)
[component_container_mt-36] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'

System

• Ubuntu 24.04
• ROS 2 Jazzy
• RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
• CYCLONEDDS_URI=file:///home/mfc/cyclonedds.xml

cyclonedds.xml file

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain Id="any">
        <General>
            <Interfaces>
                <NetworkInterface name="lo" priority="default" multicast="default"/>
            </Interfaces>
            <AllowMulticast>default</AllowMulticast>
            <MaxMessageSize>65500B</MaxMessageSize>
        </General>
        <Internal>
            <SocketReceiveBufferSize min="10MB"/>
            <Watermarks>
                <WhcHigh>500kB</WhcHigh>
            </Watermarks>
        </Internal>
        <Tracing>
            <Verbosity>config</Verbosity>
            <OutputFile>/home/mfc/dds_logs/cdds.log.${CYCLONEDDS_PID}</OutputFile>
        </Tracing>
    </Domain>
</CycloneDDS>

multicast status

Enabled:

mfc@whale:~$ ifconfig lo
lo: flags=4169<UP,LOOPBACK,RUNNING,MULTICAST>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 8415231  bytes 3311458942 (3.3 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8415231  bytes 3311458942 (3.3 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

How to fix?

Add:

        <Discovery>
            <ParticipantIndex>auto</ParticipantIndex>
            <MaxAutoParticipantIndex>1000</MaxAutoParticipantIndex>
        </Discovery>

Full cyclonedds.xml file:

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain Id="any">
        <Discovery>
            <ParticipantIndex>auto</ParticipantIndex>
            <MaxAutoParticipantIndex>1000</MaxAutoParticipantIndex>
        </Discovery>
        <General>
            <Interfaces>
                <NetworkInterface name="lo" priority="default" multicast="default"/>
            </Interfaces>
            <AllowMulticast>default</AllowMulticast>
            <MaxMessageSize>65500B</MaxMessageSize>
        </General>
        <Internal>
            <SocketReceiveBufferSize min="10MB"/>
            <Watermarks>
                <WhcHigh>500kB</WhcHigh>
            </Watermarks>
        </Internal>
        <Tracing>
            <Verbosity>config</Verbosity>
            <OutputFile>/home/mfc/dds_logs/cdds.log.${CYCLONEDDS_PID}</OutputFile>
        </Tracing>
    </Domain>
</CycloneDDS>

References

https://cyclonedds.io/docs/cyclonedds/0.9.1/config.html#controlling-port-numbers

When it is auto, Eclipse Cyclone DDS probes UDP port numbers on start-up, starting with PI = 0, incrementing it by one each time until it finds a pair of available port numbers, or it hits the limit. The maximum PI it will ever choose is Discovery/MaxAutoParticipantIndex as a way of limiting the cost of unicast discovery.

https://docs.picknik.ai/how_to/computer_configuration/customize_dds_configuration/#1-default-let-moveit-pro-configure-a-single-computer-cyclone-dds-configuration

https://cyclonedds.io/docs/cyclonedds/latest/config/config_file_reference.html#cyclonedds-domain-discovery-maxautoparticipantindex

//CycloneDDS/Domain/Discovery/MaxAutoParticipantIndex
Integer

This element specifies the maximum DDSI participant index selected by this instance of the Cyclone DDS service if the Discovery/ParticipantIndex is "auto". This also determines the range of port numbers pinged by default for unicast participant discovery.

The default value is: 99

(But true default in jazzy is even lesser than 50 or something empirically, not sure what it is)

How to get DDS participant count when a launch file is run?

- Terminate all ros2 nodes

rm /home/mfc/dds_logs/*  # this is set by <OutputFile>/home/mfc/dds_logs/cdds.log.${CYCLONEDDS_PID}</OutputFile> in cyclonedds.xml file

watch -n 0.2 "ls /home/mfc/dds_logs | wc -l"

In another terminal, run ros2 launch autoware_launch planning_simulator.launch.xml map_path:=/home/mfc/autoware_map/sample-map-planning vehicle_model:=sample_vehicle sensor_model:=sample_sensor_kit with your map path.

The count will be 50 if everything goes to plan.

mfc@whale:~$ ros2 topic list | wc -l
595
mfc@whale:~$ ros2 node list | wc -l
138

The discrepancy between node count and dds participants come from probably ROS 2 Composable Node Containers.

mfc@whale:~$ ros2 component list | wc -l
85
mfc@whale:~$ ros2 component list | grep -E '^\s+[0-9]+' | wc -l
70

138-85=53 not sure how 3 is accounted for lol.

Task

• Add this to the autoware documentation.
• Make announcement post (first in GH Discussions, then link that in Discord)

[paulsohn]

The issue actually persisted from Humble (or even before). I have investigated on this a bit ago and documented the ongoing issues including this one. See these PRs:

• [DO NOT MERGE] fix: resolve undefined symbol when LD_LIBRARY_PATH set (eclipse-cyclonedds/cyclonedds#1868) tier4/cyclonedds#1
• [DO NOT MERGE] feat: bump rmw-cyclonedds for ROS2 humble tier4/rmw_cyclonedds#1

The issues will be fixed if official ROS 2 cyclonedds is updated, but we don't know when.
In the meantime, I confirmed that using patched cycloneDDS with vcs import src < cyclonedds.repos eliminates the issue.

# cyclonedds.repos
repositories:
  # Use patch until cyclonedds v0.11.0 is supported by ROS 2 humble
  cyclonedds:
    type: git
    url: https://github.com/tier4/cyclonedds.git
    version: 69ab7d8aba9c660d9e492613f10d83edfa83861c
  # Use patch until cyclonedds v0.11.0 is supported by ROS 2 humble
  rmw_cyclonedds:
    type: git
    url: https://github.com/tier4/rmw_cyclonedds.git
    version: 82d812b81b5c148ae5469402993b1a85c8f12d90

Alternatively without hijacking middleware, TIER IV internally have been working on some parameter tuning similar to yours, and they haven't reported this creating node failure ever since then. I can't share the detail without permission though.

[KYabuuchi]

@xmfcx Thank you for sharing this issue. I ran into the same problem on ROS 2 Jazzy.
After looking into it, the behavior seems to come from this change in rmw_cyclonedds: ros2/rmw_cyclonedds@6510ad2.

That PR makes the RMW use ParticipantIndex=auto and MaxAutoParticipantIndex=32 when ParticipantIndex is not set in the config.

Caution

So the documented default value is not applied actually 🙄

The change is included in rmw_cyclonedds 1.6.0 and later;
Humble's rmw_cyclonedds_cpp is 1.3.4 while Jazzy's version is 2.2.3 , which matches the difference in behavior.

My Experiment

Default behavior on Humble v.s. Jazzy

I checked on both Humble and Jazzy with the same cyclonedds.xml (no section) and saw different defaults: CycloneDDS logs showed ParticipantIndex = none on Ubuntu 22.04 (Humble) and ParticipantIndex = auto on Ubuntu 24.04 (Jazzy).

---

(But true default in jazzy is even lesser than 50 or something empirically, not sure what it is)

From my experience, the value is overrided as 32 . 🕵️

Matching Humble’s config

I then tried explicitly setting in cyclonedds.xml:

<Discovery><ParticipantIndex>none</ParticipantIndex></Discovery>

With that, the config was applied correctly (CycloneDDS logs showed ParticipantIndex as none),

Note

Then I was able to run 200 nodes without the "Failed to find a free participant index" error. 👍

One question:

To match Humble’s behavior, adding <ParticipantIndex>none</ParticipantIndex> is enough.
Was there a particular reason you chose ParticipantIndex=auto with MaxAutoParticipantIndex=1000 instead of none.
I’d be interested to hear the use case.
Thank you.

[paulsohn]

FYI (likely you are aware of it): CycloneDDS released 11.0.0 recently. https://github.com/eclipse-cyclonedds/cyclonedds/releases/tag/11.0.0
I think it worth to consider bumping CycloneDDS as well.

[KYabuuchi]

One question:

To match Humble’s behavior, adding none is enough.
Was there a particular reason you chose ParticipantIndex=auto with MaxAutoParticipantIndex=1000 instead of none.
I’d be interested to hear the use case.
Thank you.

@xmfcx Hi, sorry to bother you. I just wanted to follow up on my question — do you have any thoughts on it?

The drawback of adding none is that unicast discovery would no longer work, but since we mainly rely on multicast, I think it shouldn't be a problem.

On the other hand, setting MaxAutoParticipantIndex to 1000 with auto might introduce a lot of announcement traffic.

[xmfcx]

On the other hand, setting MaxAutoParticipantIndex to 1000 with auto might introduce a lot of announcement traffic.

I thought this would make it act similarly to what we had back in Ubuntu 22.04 with humble. Does this really make things worse?

I didn't test <ParticipantIndex>none</ParticipantIndex> option yet. But if you say that this matches the humble behavior, I would prefer it. Can you confirm it?

[xmfcx]

Also could you update:
https://github.com/autowarefoundation/autoware-documentation/blob/eea589df2de20fdff10467ee9f6d6b5a0448882f/docs/installation/additional-settings-for-developers/network-configuration/dds-settings.md?plain=1#L62-L85

With the recommended lines so we can close this issue?

Also https://github.com/autowarefoundation/autoware-documentation/tree/main/docs/community/support/troubleshooting it'd be great if you could create a page here for "runtime-troubleshooting". This error could be added as a subpage.

[KYabuuchi]

On the other hand, setting MaxAutoParticipantIndex to 1000 with auto might introduce a lot of announcement traffic.

Does this really make things worse?

Actually, Autoware has a large number of nodes, so this might not necessarily cause any noticeable degradation...

My mention of “a lot of announcement traffic” was just based on this comment in the source code:
https://github.com/ros2/rmw_cyclonedds/blob/7cd457de5825d4cb46ec7b081aa00a5392e388d0/rmw_cyclonedds_cpp/src/rmw_node.cpp#L1193-L1199

Also could you update:
autowarefoundation/autoware-documentation@eea589d/docs/installation/additional-settings-for-developers/network-configuration/dds-settings.md?plain=1#L62-L85

Yes, I can. I’ll create a PR for it. 👍
Once we add a note about this to the documentation, I think we can close this.