Create 3 Freezes and Reboots Regularly with Firmware G.5.3

[acochrane]

Hi All,
I see a couple other posters having issues with reboots and red rings.

#387
#215

It would be great to not have the same problem, I see out of memory in my attached log. is there a firmware far enough back that does not suffer from this issue or could it be a result of some interaction with my method for driving them?

oom_log2.txt

[alsora]

Hi @acochrane, I think that your issue is not related to the other discussions you linked (those are either very old or referring to ROS 2 Humble, and related to CPU usage, not RAM usage).

From the log, I see that the robot is going out of memory and indeed it looks like the RAM usage suddenly grows, before causing the oom crash.

[INFO] [1705534252.978843837] [mb.c3po.system_health]: CPU usage: max 100 [%] mean 62 [%] RAM usage: 31/59 [MB]
[INFO] [1705535572.912451892] [mb.c3po.system_health]: CPU usage: max 87 [%] mean 63 [%] RAM usage: 44/59 [MB]
[INFO] [1705535632.815688563] [mb.c3po.system_health]: CPU usage: max 73 [%] mean 63 [%] RAM usage: 49/59 [MB]

This is very strange, and not something we normally see.
It's particularly strange also because you are using cyclonedds as a middleware which uses less RAM than fast-dds.

Can you describe your setup and what are you trying to do?
In particular, I'm interested in knowing how many robots and ROS 2 applications you are running under the same WiFi network.

How reproducible is the issue?

If the issue is reproducible, I would also like to see another log (the one you linked is not complete as it shows the app immediately crashing and then most of the content is about the following run), please try to save the log immediately after the app crashes.

Thank you

[acochrane]

I have a residential grade wifi access point serving 6 create3s and a 1-3 laptops, usually we are only actively using 3 create3s. Each create has it's own namespace, if that's relevant. We have one laptop reading odom from each, doing some math and publishing to cmd_vel for each. Read and publish rate is ~20 Hz.

I'm not sure how to quantify the number of ROS 2 applications, but there are several nodes/publishers/subscribers that aren't involved with create3s at all, they just exist for the purpose of relegating business logic to functional domains. I suppose all that data hits the wifi network anyway? I could look at network usage while the application is running.

This happens every day we go to work with them, and it happened 3 times in 45 minutes yesterday.

New log since the reverting the firmware
oom_G.4.1.txt

What else can I tell you?

[alsora]

An example of XML configuration that may be suitable for your application is the following.
Here I'm using Fast-DDS for the example, but the same thing is doable with cyclone (but I don't have the XML files at hand)

• On the robots, disable multicast and use a passive discovery approach (i.e. robots won't discover other machines, but will let other machines discover them, see the 127.0.0.1 IP address to indicate that the robot will only discover itself)

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <participant profile_name="disable_multicast" is_default_profile="true">
        <rtps>
            <builtin>
                <metatrafficUnicastLocatorList>
                    <locator/>
                </metatrafficUnicastLocatorList>
                <initialPeersList>
                    <locator>
                        <udpv4>
                            <address>127.0.0.1</address>
                        </udpv4>
                    </locator>
                </initialPeersList>
            </builtin>
        </rtps>
    </participant>
</profiles>

• on the laptop use a unicast active discovery protocol. this will communicate only with the IP addresses listed in the XML profile and you could list all your robot IPs (here I used 192.168.1.103 as an example)

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <participant profile_name="unicast_discovery" is_default_profile="true">
        <rtps>
            <builtin>
                <metatrafficUnicastLocatorList>
                    <locator/>
                </metatrafficUnicastLocatorList>
                <initialPeersList>
                    <locator>
                        <udpv4>
                            <address>192.168.1.103</address>
                        </udpv4>
                    </locator>
                </initialPeersList>
            </builtin>
        </rtps>
    </participant>
</profiles>

the idea is that each robot will only see the laptop, and the laptop will see all robots

[acochrane]

Ok thanks for your insight!

This means that for my application, I will need some way to aggregate info from the different DOMAIN_IDs in order to perform my business logic, then redistribute the results individually to each create3? So I also need to have a separate odom/cmd-vel application dedicated to each create3?

Has anyone written up a straightforward way to achieve this?

Is there a way for a ROS 2 application to communicate with multiple DOMAIN_IDs?
I will spend some time with the referenced docs and try to get a feel for a working solution.

[alsora]

Usually, a ROS 2 application uses a single domain ID, so if you go down this route, the simplest approach would be to run multiple, separate, applications (one for each create 3s).
Accessing multiple domain IDs from the same application should be doable by creating multiple ROS 2 contexts in the same application.
This is something I never tried nor seen examples but the APIs are there.
ROS 2 contexts take a rclcpp::InitOptions argument and this allows to set the domain ID: https://github.com/ros2/rclcpp/blob/humble/rclcpp/include/rclcpp/init_options.hpp#L89-L92
Working with a non-default rclcpp context is not a standard use-case, but it should work.
The context should be explicitly passed to the nodes constructor via the rclcpp::NodeOptions and then entities created from this node will use it.
So you will need N rclcpp::Context and at least N rclcpp::Node objects, where N is the number of domain IDs.
All this can coexist in a single C++ application.

Another option to consider may be the Fast-DDS discovery server: https://docs.ros.org/en/humble/Tutorials/Advanced/Discovery-Server/Discovery-Server.html .
Create 3 supports it, but I never tried advanced configurations with it, so I don't know if it would lead to a complete isolation of the robots (similarly to a domain ID)

[brianabouchard]

Just to chime in - we did some testing with >10 robots on a rather finicky university network about a year ago (an arguably less optimal setup that what you have now). With the configurations available at the time, we also found Domain IDs to be the right solution as @alsora as suggested. A colleague ended up writing some Python code that allowed users to cycle through subscriptions to the /battery_state topic on each domain ID. Note we did also use namespaces even though they were on separate domain IDs simply to better keep track of the robots.

import time

import rclpy
from rclpy.node import Node
from rclpy.qos import qos_profile_sensor_data
from sensor_msgs.msg import BatteryState

class battery(Node):
    def __init__(self, name):
        super().__init__('battery_subscriber')
        self.subscription = self.create_subscription(BatteryState, 
                    '/' + name + '/battery_state', self.callback, qos_profile_sensor_data)
    
    def callback(self, msg: BatteryState):
        charge = 100*msg.percentage
        print(' %0.1f ' % charge, end='')


names = ['rogers','Syndrome','Dory','Nemo','Remy','EdnaMode','Woody','Buzz','MikeWazowski','LightningMcQueen','Mater','Dash','ElastGirl','Pig','PotatoHead','Dot','Sully','Carl']


rclpy.init()

for i in range(len(names)):
    bs = battery(names[i])
    print('checking %s (%d)' % (names[i],i), end='')
    os.environ['ROS_DOMAIN_ID'] = str(i)
    try:
        rclpy.spin_once(bs, timeout_sec = 10)
    except KeyboardInterrupt:
        print('\nCaught keyboard interrupt')
        break
    finally:
        print(" Done")
        bs.destroy_node()
    
rclpy.shutdown()

I wanted to share that this past summer we also found the Fast DDS discovery server to be immensely helpful in getting multiple robots successfully running on the same Domain ID on busy Wi-Fi networks - noting that while you have an isolated network, it sounds like you are running multiple ROS 2 processes on it. We went from never being able to connect without disabling multicast when using Fast DDS to connecting quickly, easily, and reliably. So I would highly recommend trying that if you're experiencing any network related issues. I can't speak to if this would improve memory issues with 3 robots because I haven't tested this particular use case, but we have been able to run 2 robots using this method on the same Domain ID with different namespaces without issue.

[mdombrowsky]

Is there a easy way to implement multiple Fast DDS server for different ports (for its own domain) at the same time?
Is it better to run all the server instances on the server itself or should each robot run its own Fast DDS server if possible?