ros2 communication failures

[gjf2a]

How are you connecting to your Create 3?

USB

Computer(s) Model(s) and Operating System(s)

Libre Computer Board AML-S905X-CC

Which version of ROS 2 is installed on your computer?

Humble

Which firmware version is installed on your robot?

H.1.2

Which RMW is your robot running?

FastRTPS

Does your robot have an assigned namespace? If so, please share.

No response

Is the robot connected to a network? If so, what is the network type?

Yes, work network

Are there multiple Create 3 robots connected to your network?

No

Is multicast enabled?

None

What is the Adapter Board's USB/BLE Toggle currently switched to?

USB

Describe your question.

I have a Libre Computer Board (Raspberry Pi clone) sitting atop my iRobot Create3 robot. In general, I get bad results from ROS2 commands. I have tried several examples and they have all failed.

ROS2 is functioning and communication is happening. For instance, ros2 topic list yields:

/battery_state
/cmd_audio
/cmd_lightring
/cmd_vel
/dock_status
/hazard_detection
/imu
/interface_buttons
/ir_intensity
/ir_opcode
/kidnap_status
/mobility_monitor/transition_event
/mouse
/odom
/parameter_events
/robot_state/transition_event
/rosout
/slip_status
/static_transform/transition_event
/stop_status
/tf
/tf_static
/wheel_status
/wheel_ticks
/wheel_vels

But if I send:
ros2 action send_goal /drive_distance irobot_create_msgs/action/DriveDistance "{distance: 0.5,max_translation_speed: 0.15}"

I always get the following response:
Waiting for an action server to become available...

If I send:
ros2 topic pub --once /cmd_audio irobot_create_msgs/msg/AudioNoteVector "{append: false, notes: [{frequency: 100, max_runtime: {sec: 1,nanosec: 0}}, {frequency: 50, max_runtime: {sec: 1,nanosec: 0}}]}"

It endlessly prints:
Waiting for at least 1 matching subscription(s)...

If I send:
ros2 topic pub /cmd_lightring irobot_create_msgs/msg/LightringLeds "{override_system: true, leds: [{red: 255, green: 0, blue: 0}, {red: 0, green: 255, blue: 0}, {red: 0, green: 0, blue: 255}, {red: 255, green: 255, blue: 0}, {red: 255, green: 0, blue: 255}, {red: 0, green: 255, blue: 255}]}"

It repeatedly prints:
publishing #14: irobot_create_msgs.msg.LightringLeds(header=std_msgs.msg.Header(stamp=builtin_interfaces.msg.Time(sec=0, nanosec=0), frame_id=''), leds=[irobot_create_msgs.msg.LedColor(red=255, green=0, blue=0), irobot_create_msgs.msg.LedColor(red=0, green=255, blue=0), irobot_create_msgs.msg.LedColor(red=0, green=0, blue=255), irobot_create_msgs.msg.LedColor(red=255, green=255, blue=0), irobot_create_msgs.msg.LedColor(red=255, green=0, blue=255), irobot_create_msgs.msg.LedColor(red=0, green=255, blue=255)], override_system=True)

But the lights never actually change color.

The robot itself works fine if I use the playground, but ROS2 has been a disappointment thus far, to say the least.

Any advice or suggestions would be most welcome. Thanks!

[alsora]

@gjf2a I have a few questions/suggestions to try fix this problem.

1. can you double check that your robot and your Libre Computer Board are using the same RMW implementation?
2. what happens if you run ros2 topic echo /odom ?
3. are you using a custom XML profile to configure your RMW?
4. have you tried to run ros2 daemon stop and then try one of the commands that was failing?

Let me know how it goes!

[gjf2a]

1. Yes, they are both using rmw_fastrtps_cpp.
2. ros2 topic echo /odom just sits there, not printing anything or responding in any way.
3. No, I'm not using a custom XML profile for the RMW.
4. I tried running ros2 daemon stop followed by the failing commands, and there was no change in behavior.

Let me know what else I might try. Thanks for your help!

[alsora]

Does ros2 topic list work reliably?

Can you try to start tcpdump in a separate terminal (see instructions here #278 (comment)) while in another terminal you run a bunch of ROS 2 commands?
For example

ros2 daemon stop
ros2 topic list
ros2 topic echo /odom
ros2 action send_goal /drive_distance irobot_create_msgs/action/DriveDistance "{distance: 0.5,max_translation_speed: 0.15}"

[gjf2a]

ros2 topic list is somewhat reliable. Initially, it just prints /parameter_events and /rosout, but if I run it five or six times usually all the topics appear.

I set up tcpdump. It doesn't display anything. Even when ros2 topic list returns a good topic list, I still don't see anything on tcpdump.

Rather mysteriously, the robot started driving after I waited on the action server for a minute or two. I was able to repeat this three or four times - again there was always a long wait. Once I waited as long as six minutes but it did not start driving.

But even when it started driving, I still did not see any TCP traffic.

Although my intention was to run it over USB, I think it is actually running over wifi - when I took out the wifi dongle, it doesn't connect at all - nothing from ros2 topic list at that point. The wifi dongle has been performing poorly in general - perhaps that plays a role.

[kwujciak]

From what it sounds like, there might be wifi connection issues. If you have to run the topic list a few times before you get a full list, it seems like the connection is inconsistent. ROS 2 + wifi can cause annoying problems if the set up isn't "ideal." Did you connect your robot and your Libre board to wifi? If so, this could be the source of some issues. If you'e controlling your robot through your board, you only need your robot connected to the board through USB, and the board connected to wifi. It may be bypassing the board and trying to talk to you through the wifi. In order to get your robot to not automatically connect to wifi, you'll need to reset it (if you had previously connected it). As a warning, factory reset (found on the webserver) will remove all user configuration. What is your wifi dongle doing? Is it connected to your personal computer?

[gjf2a]

The wifi dongle is plugged into the Libre board - unlike a Raspberry Pi, those boards don't come with wifi.

One thing I realized was slowing it down was that, to try to understand how things worked, I was plugging in a monitor and keyboard into the Libre. It turns out that running full desktop Ubuntu was impairing wifi performance. If I run things from a remote shell, it works better.

At this point, I have a simple Python program that changes light ring colors in response to bump sensors. It takes up to 25 seconds for the ROS2 callbacks to begin, and sometimes bumper events get queued up and ignored for a while. Once it has been running for about 40 seconds, it usually performs very well.

While the startup times are extremely annoying, the fact that it works consistently after "warming up" does make it usable. I'm planning to use this as a template for some other things, so hopefully we'll have some more interesting behaviors working soon.

[kwujciak]

Hm interesting. At one point it was taking an unusual amount of time to run actions in particular for me. Once I put it on a more consistent and less busy network, that solved my problem. My network seemed to get confused with the middleware. However, I've never used a Libre board so I'm not totally sure what would help speed it up. Using a Raspberry Pi also eliminated some issues (although I know they're hard to come by at the moment).