Losing Connection after sending a few ros2 commands

[aroller]

How are you connecting to your Create 3?

Wi-Fi (ROS 2)

Computer(s) Model(s) and Operating System(s)

Mac OS 14.1, multipass 1.12.2+mac (VM), ubuntu 22.04 / ROS2 Humble

Which version of ROS 2 is installed on your computer?

Humble

Which firmware version is installed on your robot?

H.2.3

Which RMW is your robot running?

FastRTPS

Does your robot have an assigned namespace? If so, please share.

no

Is the robot connected to a network? If so, what is the network type?

home network, firewall disabled

Are there multiple Create 3 robots connected to your network?

No

Is multicast enabled?

Yes

What is the Adapter Board's USB/BLE Toggle currently switched to?

Bluetooth (default)

Describe your question.

I'm successfully connecting and sending ros2 commands from the API docs. After a few commands (around 5) the commands no longer work and it appears the nodes have crashed. The ros2 topic list still presents the topics, but ros2 topic hz shows no longer publishing.

I'm sending the rotate command in sequence upon each success:

ros2 action send_goal /rotate_angle irobot_create_msgs/action/RotateAngle "{angle: 3.14,max_rotation_speed: 0.5}"

See the
ros2 commands.pdf showing successes ending in failure:

Waiting for an action server to become available...

The exported logs
messages.txt will show the entire history since my recent upgrade to Humble (the problem was happening with Galactic also).

There are no errors reported to the logs. The commands just stop being received.

Nov 29 15:42:20 Beaker user.notice create-platform: [WARN] [1701272540.333562849] [motion_control]: rotate_angle goal succeeded: rotated commanded angle
Nov 29 15:42:29 Beaker user.notice create-platform: [INFO] [1701272549.663931649] [motion_control]: Received new rotate_angle goal
Nov 29 15:42:29 Beaker user.notice create-platform: [INFO] [1701272549.705774386] [motion_control]: rotate_angle goal with angle 3.140000, max_speed 0.500000, per wheel 58
Nov 29 15:42:36 Beaker user.notice create-platform: [INFO] [1701272556.215243940] [mobility_monitor]: Received mobility notification: ROBOT_NOTIFY_GYRO_ROTATION_TEST_DONE (56) with arg 0
Nov 29 15:42:36 Beaker user.notice create-platform: [WARN] [1701272556.233523236] [motion_control]: rotate_angle goal succeeded: rotated commanded angle
Nov 29 15:42:43 Beaker user.notice create-platform: [INFO] [1701272563.420784984] [motion_control]: Received new rotate_angle goal
Nov 29 15:42:43 Beaker user.notice create-platform: [INFO] [1701272563.422782227] [motion_control]: rotate_angle goal with angle 3.140000, max_speed 0.500000, per wheel 58
Nov 29 15:42:49 Beaker user.notice create-platform: [INFO] [1701272569.008720905] [system_health]: CPU usage: max 90 [%] mean 61 [%] RAM usage: 38/59 [MB]
Nov 29 15:42:49 Beaker user.notice create-platform: [INFO] [1701272569.863528034] [mobility_monitor]: Received mobility notification: ROBOT_NOTIFY_GYRO_ROTATION_TEST_DONE (56) with arg 0
Nov 29 15:42:49 Beaker user.notice create-platform: [WARN] [1701272569.883160302] [motion_control]: rotate_angle goal succeeded: rotated commanded angle
Nov 29 15:42:58 Beaker user.notice create-platform: [INFO] [1701272578.516552099] [motion_control]: Received new rotate_angle goal
Nov 29 15:42:58 Beaker user.notice create-platform: [INFO] [1701272578.523373768] [motion_control]: rotate_angle goal with angle 3.140000, max_speed 0.500000, per wheel 58
Nov 29 15:43:00 Beaker user.notice ntpd: ntpd: timed out waiting for 192.168.186.3, reach 0x00, next query in 542s
Nov 29 15:43:05 Beaker user.notice create-platform: [INFO] [1701272585.016608706] [mobility_monitor]: Received mobility notification: ROBOT_NOTIFY_GYRO_ROTATION_TEST_DONE (56) with arg 0
Nov 29 15:43:05 Beaker user.notice create-platform: [WARN] [1701272585.035540410] [motion_control]: rotate_angle goal succeeded: rotated commanded angle
Nov 29 15:43:05 Beaker user.notice ntpd: ntpd: timed out waiting for 192.168.186.1, reach 0x00, next query in 512s
Nov 29 15:43:49 Beaker user.notice create-platform: [INFO] [1701272629.009968796] [system_health]: CPU usage: max 91 [%] mean 56 [%] RAM usage: 38/59 [MB]
Nov 29 15:44:49 Beaker user.notice create-platform: [INFO] [1701272689.000515137] [system_health]: CPU usage: max 66 [%] mean 52 [%] RAM usage: 38/59 [MB]
Nov 29 15:45:49 Beaker user.notice create-platform: [INFO] [1701272749.000234693] [system_health]: CPU usage: max 76 [%] mean 57 [%] RAM usage: 38/59 [MB]
Nov 29 15:46:49 Beaker user.notice create-platform: [INFO] [1701272809.004297592] [system_health]: CPU usage: max 73 [%] mean 59 [%] RAM usage: 38/59 [MB]

This problem is repeatable by restarting the application from the web console. The number of times an action can be executed varies +/- 2 and drive straight or rotate both result in the problem.

I'm new to ROS2 and the create3, but quite familiar with ROS. The robot was one of the first to ship over a year ago, but I just got it up and running today due to my schedule priorities.

Any assistance to troubleshoot would be appreciated.

About beaker.pdf

[alsora]

Hi @aroller, thank you for the detailed explanation.
From what I see, it looks to me that the nodes are not crashing.
The robot runs a single-process application, so the fact that your log doesn't show anything strange and that it keeps printing

Nov 29 15:46:49 Beaker user.notice create-platform: [INFO] [1701272809.004297592] [system_health]: CPU usage: max 73 [%] mean 59 [%] RAM usage: 38/59 [MB]

means that the app is happy and running.

It looks like you see topics via ros2 topic list, but you don't see data with ros2 topic hz and when sending the action request the server is not found.
This to me indicates a connectivity problem.
The only reason why ros2 topic list returns something is because ROS 2 uses a "discovery daemon" that caches discovery information.
When you reach this scenario, if you run ros2 topic list --no-daemon ( or run ros2 daemon stop before running your commands), I bet that you wouldn't see the topics.

Having said that, I'm not sure why the connection with the robot suddenly stops.

There's a few things I would like to suggest:

• can you verify that both the robot and the laptop are running with Fast-RTPS ? you can see the rmw used by the robot in the logs when the app starts or from the webserver. to set the rmw on the laptop you should use export RMW_IMPLEMENTATION=rmw_fastrtps_cpp.
• can you try to use an XML profile on the laptop to configure the ROS middleware to use unicast rather than multicast for discovery? you can find instructions here, look for the disable-multicast section.
• can you try to switch both robot and laptop to a different middleware, i.e cyclonedds?

If none of those work, I would suggest to try using a native Ubuntu machine to communicate with the robot, so you don't need a VM.

Please let me know if you need more information.

FYI: if you find yourself in this situation, you can use this quick curl command to restart the ROS application that runs on the robot.
Obviously it's not ideal and I want to get to the root of this, but dropping it here as it may make your life easier.

curl -X POST http://192.168.186.2/api/restart-app

[aroller]

This to me indicates a connectivity problem.

I think you are correct. Seemingly a problem on the robot side because the restart of the Application seems to recover the connection without modification on the laptop side. Notice during shutdown the nodes are itemized so I have no evidence that the nodes crashed and now there is evidence the nodes are seemingly available at termination.

I've updated the discussion title to reflect the likely problem.

Nov 30 07:49:01 Beaker user.notice create-platform: [INFO] [1701330541.785186956] [create-platform]: Shutting down nodes
Nov 30 07:49:01 Beaker user.notice create-platform: [INFO] [1701330541.908032867] [create-platform]: Shutdown mobility node
Nov 30 07:49:01 Beaker user.notice create-platform: [INFO] [1701330541.908434725] [_internal.mobility]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.125667155] [_internal.mobility]: Lifecycle callback: on_shutdown
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.141321361] [create-platform]: Shutdown kinematics_engine node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.202233051] [_internal.kinematics_engine]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.203682329] [_internal.kinematics_engine]: Lifecycle callback: on_shutdown
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.204765720] [create-platform]: Shutdown composite_hazard node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.237616000] [_internal.composite_hazard]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.282057503] [_internal.composite_hazard]: Lifecycle callback: on_shutdown
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.283118766] [create-platform]: Shutdown stasis node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.283934051] [_internal.stasis]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.285264967] [_internal.stasis]: Lifecycle callback: on_shutdown
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.286331905] [create-platform]: Shutdown mobility_monitor node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.287257181] [mobility_monitor]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.288868567] [create-platform]: Shutdown robot_state node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.342346211] [robot_state]: Lifecycle callback: on_deactivate
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.343958668] [create-platform]: Shutdown static_transform node
Nov 30 07:49:02 Beaker user.notice create-platform: [INFO] [1701330542.345230967] [create-platform]: Shutting down rclcpp
Nov 30 07:49:04 Beaker user.notice create-platform: [INFO] [1701330544.476006620] [create-platform]: Shutting down application!
Nov 30 07:49:04 Beaker user.notice create-platform: [INFO] [1701330544.480526323] [create-platform]: Joining threads!
Nov 30 07:49:04 Beaker user.notice create-platform: [INFO] [1701330544.481854745] [create-platform]: Shutting down audio!
Nov 30 07:49:04 Beaker user.notice create-platform: [INFO] [1701330544.678987820] [create-platform]: Application ready to terminate!
Nov 30 07:49:13 Beaker user.notice create-platform: terminate called without an active exception
Nov 30 07:49:14 Beaker user.notice create-platform: Server 'create-platform' was stopped gracefully.

[aroller]

Something unexpected is happening. When running ros2 topic list --no-daemon, the topics are never listed. The action commands do work for a period. I would have expected the topics to be listed correctly until the action fails. ros2 topic list without the --no-daemon does list the topics as expected.

ubuntu@create3-bridged:~$ ros2 action send_goal /rotate_angle irobot_create_msgs/action/RotateAngle "{angle: 3.14,max_rotation_speed: 0.5}"
Waiting for an action server to become available...
Sending goal:
     angle: 3.14
max_rotation_speed: 0.5

Goal accepted with ID: 29bf11293f9c409abd4935f68bcad4b5

Result:
    pose:
  header:
    stamp:
      sec: 1701333641
      nanosec: 87221932
    frame_id: odom
  pose:
    position:
      x: 0.01237259991466999
      y: 0.009147662669420242
      z: 3.25250681498801e-07
    orientation:
      x: -0.0008351126827787221
      y: 0.0007984426240869225
      z: 0.9999790746921748
      w: 0.006365159382532971

Goal finished with status: SUCCEEDED
ubuntu@create3-bridged:~$ ros2 topic list --no-daemon
/parameter_events
/rosout
ubuntu@create3-bridged:~$ ros2 action send_goal /rotate_angle irobot_create_msgs/action/RotateAngle "{angle: 3.14,max_rotation_speed: 0.5}"
Waiting for an action server to become available...
Sending goal:
     angle: 3.14
max_rotation_speed: 0.5

Goal accepted with ID: a7950b0b301f44e4a5653887e7aab63a

Result:
    pose:
  header:
    stamp:
      sec: 1701333666
      nanosec: 672831854
    frame_id: odom
  pose:
    position:
      x: -0.006500485353171825
      y: 0.007563842460513115
      z: -7.275630196090788e-06
    orientation:
      x: -0.0007613585033387207
      y: -0.0006163593250744081
      z: 0.002598264916686865
      w: 0.9999961447194856

Goal finished with status: SUCCEEDED
ubuntu@create3-bridged:~$ ros2 topic list --no-daemon
/parameter_events
/rosout
ubuntu@create3-bridged:~$ ros2 action send_goal /rotate_angle irobot_create_msgs/action/RotateAngle "{angle: 3.14,max_rotation_speed: 0.5}"
Waiting for an action server to become available...
Sending goal:
     angle: 3.14
max_rotation_speed: 0.5

Goal accepted with ID: 1fff72ede3444a66b3ce1d15a57ab1e0

Result:
    pose:
  header:
    stamp:
      sec: 1701333697
      nanosec: 669682167
    frame_id: odom
  pose:
    position:
      x: 0.013477456755936146
      y: 0.008821750059723854
      z: 2.857126855815295e-05
    orientation:
      x: 0.0005332153114068903
      y: -0.0007418037403948541
      z: -0.9999789098891543
      w: 0.0064300222037676

Goal finished with status: SUCCEEDED
ubuntu@create3-bridged:~$ ros2 topic list --no-daemon
/parameter_events
/rosout

[aroller]

I tried rmw_cyclonedds_cpp without successfully connecting from my laptop. I didn't exhaustively try, but I confirmed the logs was using the new middleware config. One attempt without success and I went back to the default for Humble. rmw_fastrtps_cpp

[aroller]

can you try to use an XML profile on the laptop to configure the ROS middleware to use unicast rather than multicast for discovery? you can find instructions here, look for the disable-multicast section.

Disabling multicast fixed the problem. I followed the instructions setting the IP address in the config file and set the environment variable and now I successfully send the connections for a couple of sessions for 10 minutes each without failure.

NOTE: The problem was introduced by my home network for an unknown reason. Direct connect via access point and mobile hotspot did not reproduce the problem.

Thanks @alsora for your assistance.

[aroller]

It seems my Home Network is the prime suspect. I repeated a timing test 3 times by restarting the application and running a repeat command every 10 seconds:

while sleep 10; do     
  date; 
  ros2 topic pub --once /cmd_audio irobot_create_msgs/msg/AudioNoteVector "{append: false, notes: [{frequency: 100, max_runtime: {sec: 1,nanosec: 0}}, {frequency: 50, max_runtime: {sec: 1,nanosec: 0}}]}";  
  done

The results on my home network consistently failed right near the 3 minute mark. I ran the command when I heard the startup chime.

timing test - home network - 1.txt
timing test - home network - 2.txt

Then I connected via the access point, following the setup instructions, which never failed to communicate. Notice I had to restart my multipass VM to get a new IP address.

Aarons-MacBook-Air:~ rollermacair$ multipass info create3-bridged
Name:           create3-bridged
State:          Running
IPv4:           10.211.67.5
                192.168.0.153
Release:        Ubuntu 22.04.3 LTS
Image hash:     054db2d88c45 (Ubuntu 22.04 LTS)
CPU(s):         2
Load:           0.01 0.02 0.00
Disk usage:     2.8GiB out of 19.3GiB
Memory usage:   197.4MiB out of 951.1MiB
Mounts:         --
Aarons-MacBook-Air:~ rollermacair$ multipass stop  create3-bridged && multipass start create3-bridged
Aarons-MacBook-Air:~ rollermacair$ multipass info create3-bridged
Name:           create3-bridged
State:          Running
IPv4:           10.211.67.5
                192.168.10.3
Release:        Ubuntu 22.04.3 LTS
Image hash:     054db2d88c45 (Ubuntu 22.04 LTS)
CPU(s):         2
Load:           0.21 0.05 0.02
Disk usage:     2.8GiB out of 19.3GiB
Memory usage:   147.3MiB out of 951.1MiB
Mounts:         --
Aarons-MacBook-Air:~ rollermacair$ 

Then I successfully sent commands over 12 minutes until manually quitting.

timing test - access pont - 1.txt

beaker logs - access point - 1.txt

I confirmed the 3 minute max still applied when reconnecting to the home network after the successful run on the access point.

Not yet a solution, but the prime suspect is the home network.

[aroller]

Rebooting the home network didn't resolve the problem. I wanted to make sure the firewall settings that I disabled really were applied. This seems like a firewall is applying some protection after 3 minutes of observing traffic perhaps, but that is just a guess.

[aroller]

Good news. I tried restarting the router again, perhaps with some changes to the firewall that I applied again getting past some poor user interface design. I had this successful session using my home network that lasted over 10 minutes. I did find this error posted in the log which seems very network relevant.

timing test - home network - after reboot -3.txt

robot logs during successful home network session.txt

Nov 30 11:07:07 Beaker user.notice ntpd: ntpd: send failed: Network unreachable
Nov 30 11:07:12 Beaker daemon.err udhcpc[28480]: sending discover
Nov 30 11:07:12 Beaker daemon.err udhcpc[28480]: sending select for 192.168.0.179
Nov 30 11:07:13 Beaker daemon.err udhcpc[28480]: lease of 192.168.0.179 obtained, lease time 122
Nov 30 11:07:13 Beaker kern.warn kernel: [74388.238773] MtAsicUpdateProtect: unknown protection mode(9)
Nov 30 11:07:13 Beaker daemon.info avahi-daemon[1231]: Joining mDNS multicast group on interface wlan0.IPv4 with address 192.168.0.179.
Nov 30 11:07:13 Beaker daemon.info avahi-daemon[1231]: New relevant interface wlan0.IPv4 for mDNS.
Nov 30 11:07:13 Beaker daemon.err network[1477]: [ERROR] (src/aos_net.c:449 errno:Address not available) route does not belong to main routing table
Nov 30 11:07:13 Beaker daemon.err network[1477]: [ERROR] (src/aos_net.c:249 errno:Address not available) aos_net_parse_routes failed
Nov 30 11:07:13 Beaker daemon.notice network[1477]: [INFO] (src/wifi_control.c:1620) have IP but have not read GW addr
Nov 30 11:07:13 Beaker daemon.err network[1477]: [ERROR] (src/aos_net.c:449 errno:Address not available) route does not belong to main routing table
Nov 30 11:07:13 Beaker daemon.err network[1477]: [ERROR] (src/aos_net.c:249 errno:Address not available) aos_net_parse_routes failed

I don't know that the problem is solved, but at least it worked once.

[aroller]

Two more attempts resulted in eventual failures, but longer than the 3 minute previously observed.

timing test - home network - after reboot - 5.txt
timing test - home network - after reboot -4.txt

Given the problem exists and I don't have any other settings on my home network, I will re-attempt on the access point and attempt on my mobile hotspot network to see if either of those will ever demonstrate this same failure. I don't want to mess with the middleware without understanding if the problem exists only on a specific LAN.

[aroller]

The access point and mobile hotpsot seem to work indefinitely without failure. over 20 minutes on my hotspot. So it seems the home network is the problem, although the specific problem with the home network was not identified.