Joint State Broadcaster fails to configure on Raspberry Pi 5

[louietouie]

Description
When running demos like example 10, the rrbot.launch.py launch file fails when running as instructed. The failures are occurring on a Raspberry Pi 5 with Jazzy (ros-jazzy-ros-base, ros-jazzy-ros2-control, ros-jazzy-ros2-controllers, ros-jazzy-xacro)

Reproduce

1. colcon build (in the folders example_10 and ros2_control_demo_description)
2. source install/local_setup.bash
3. ros2 launch ros2_control_demo_example_10 rrbot.launch.py

Expected behavior
Runs normally

Environment (please complete the following information):

1. Raspberry Pi 5
2. Ubuntu 24.04.1 LTS
3. Memory: 7.7 GiB memory
4. OS-Type: 64-bit
5. Kernel: Linux 6.8.0-1015-raspi
6. Jazzy

Errors

[spawner-5] [INFO] [1733117855.734730114] [spawner_joint_state_broadcaster]: Loaded joint_state_broadcaster
[ros2_control_node-1] [INFO] [1733117855.740230128] [controller_manager]: Configuring controller: 'joint_state_broadcaster'
[ros2_control_node-1] [INFO] [1733117855.740658192] [joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published
[ros2_control_node-1] Exception thrown during init stage with message: Resource temporarily unavailable 
[ros2_control_node-1] [WARN] [1733117855.755595928] [joint_state_broadcaster]: Error occurred while doing error handling.
[ros2_control_node-1] [ERROR] [1733117855.756544168] [controller_manager]: After configuring, controller 'joint_state_broadcaster' is in state 'unconfigured' , expected inactive.
[spawner-5] [ERROR] [1733117855.762393727] [spawner_joint_state_broadcaster]: Failed to configure controller
[ERROR] [spawner-5]: process has died [pid 9693, exit code 1, cmd '/opt/ros/jazzy/lib/controller_manager/spawner joint_state_broadcaster --ros-args'].

[ros2_control_node-1] [INFO] [1733117925.183339910] [controller_manager]: Loading controller : 'joint_state_broadcaster' of type 'joint_state_broadcaster/JointStateBroadcaster'
[ros2_control_node-1] [INFO] [1733117925.183398244] [controller_manager]: Loading controller 'joint_state_broadcaster'
[ros2_control_node-1] [INFO] [1733117925.204228872] [controller_manager]: Controller 'joint_state_broadcaster' node arguments: --ros-args --params-file /home/[name]/Documents/ros2_control_demos/example_10/install/ros2_control_demo_example_10/share/ros2_control_demo_example_10/config/rrbot_controllers.yaml 
[ros2_control_node-1] Stack trace (most recent call last) in thread 9864:
[ros2_control_node-1] #3    Object "", at 0, in 
[ros2_control_node-1] #2    Object "/opt/ros/jazzy/lib/liblifecycle_msgs__rosidl_typesupport_introspection_c.so", at 0xffffbd6ba7df, in 
[ros2_control_node-1] #1    Object "/opt/ros/jazzy/lib/liblifecycle_msgs__rosidl_typesupport_introspection_c.so", at 0xffffbd6591d3, in 
[ros2_control_node-1] #0    Object "/opt/ros/jazzy/lib/liblifecycle_msgs__rosidl_typesupport_introspection_c.so", at 0xffffbd658f97, in 
[ros2_control_node-1] signal 11
[ERROR] [ros2_control_node-1]: process has died [pid 9830, exit code -11, cmd '/opt/ros/jazzy/lib/controller_manager/ros2_control_node --ros-args --params-file /home/[name]/Documents/ros2_control_demos/example_10/install/ros2_control_demo_example_10/share/ros2_control_demo_example_10/config/rrbot_controllers.yaml'].
[spawner-5] [WARN] [1733117935.206845319] [spawner_joint_state_broadcaster]: Failed getting a result from calling /controller_manager/load_controller in 10.0. (Attempt 1 of 3.)

Also of note, when doing ctrl-c, I often get the error...

^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[robot_state_publisher-2] [INFO] [1733118012.387450808] [rclcpp]: signal_handler(signum=2)
[ros2_control_node-1] [INFO] [1733118012.387450845] [rclcpp]: signal_handler(signum=2)
[ros2_control_node-1] Stack trace (most recent call last):
[ros2_control_node-1] #7    Object "/usr/lib/aarch64-linux-gnu/ld-linux-aarch64.so.1", at 0xffffffffffffffff, in 
[ros2_control_node-1] #6    Object "/opt/ros/jazzy/lib/controller_manager/ros2_control_node", at 0xaaaae2dd51ef, in _start
[INFO] [robot_state_publisher-2]: process has finished cleanly [pid 9981]
[ros2_control_node-1] #5    Source "../csu/libc-start.c", line 360, in __libc_start_main_impl [0xffffab268597]
[ros2_control_node-1] #4    Source "../sysdeps/nptl/libc_start_call_main.h", line 58, in __aarch64_ldadd4_relax [0xffffab2684c3]
[ros2_control_node-1] #3    Object "/opt/ros/jazzy/lib/controller_manager/ros2_control_node", at 0xaaaae2dd48bf, in main
[ros2_control_node-1] #2    Object "/opt/ros/jazzy/lib/controller_manager/ros2_control_node", at 0xaaaae2dd5527, in 
[ros2_control_node-1] #1    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0xffffab8da8e3, in rclcpp::Executor::~Executor()
[ros2_control_node-1] #0    Object "/opt/ros/jazzy/lib/librclcpp.so", at 0xffffab8d24fc, in 
[ros2_control_node-1] Segmentation fault (Address not mapped to object [0xffffa25ee028])

Additional context
I also made it a stack overflow question, but moved it to this when I realized that it wasn't only happening on my code, but on official examples too.

[louietouie]

Quick note, those 3 errors all occurred on separate runs; they did not occur together. The bug that happens seems to be random.

[saikishor]

Hello @louietouie

This seems to be an ABI issue. Can you please update all the packages on Raspi to their latest version?

Thank you!

[louietouie]

Thanks for the help @saikishor.

Updated all packages with the 5 commands... sudo apt update/sudo apt upgrade/sudo apt full-upgrade/sudo apt autoremove/sudo reboot.

Unfortunately still having issues, the only other thing I can think that is unique to my situation is that I have a RS485 CAN shield on the RPi5.

After running a few more times, I am seeing the error Resource temporarily unavailable a lot. Also just got this new error when running my launch file (not example 10)...

[ros2_control_node-2] [INFO] [1733123792.259891679] [my_joint_state_broadcaster]: 'joints' or 'interfaces' parameter is empty. All available state interfaces will be published
[ros2_control_node-2] 
[ros2_control_node-2] >>> [rcutils|error_handling.c:108] rcutils_set_error_state()
[ros2_control_node-2] This error state is being overwritten:
[ros2_control_node-2] 
[ros2_control_node-2]   'Type support not from this implementation. Got:
[ros2_control_node-2]     Handle's typesupport identifier (rosidl_typesupport_cpp) is not supported by this library, at ./src/type_support_dispatch.hpp:114
[ros2_control_node-2]     Could not load library libcontrol_msgs__rosidl_typesupport_introspection_cpp.so: dlopen error: libcontrol_msgs__rosidl_typesupport_introspection_cpp.so: failed to map segment from shared object, at ./src/shared_library.c:99, at ./src/type_support_dispatch.hpp:79
[ros2_control_node-2] while fetching it, at ./src/TypeSupport_impl.cpp:212'
[ros2_control_node-2] 
[ros2_control_node-2] with this new error message:
[ros2_control_node-2] 
[ros2_control_node-2]   'failed to register type object with incompatible type control_msgs::msg::dds_::DynamicJointState_, at ./src/publisher.cpp:196'
[ros2_control_node-2] 
[ros2_control_node-2] rcutils_reset_error() should be called after error handling to avoid this.
[ros2_control_node-2] <<<
[ros2_control_node-2] Exception thrown during init stage with message: could not create publisher: failed to register type object with incompatible type control_msgs::msg::dds_::DynamicJointState_, at ./src/publisher.cpp:196, at ./src/rcl/publisher.c:117 
[ros2_control_node-2] [WARN] [1733123792.292329757] [my_joint_state_broadcaster]: Error occurred while doing error handling.
[ros2_control_node-2] [ERROR] [1733123792.292450497] [controller_manager]: After configuring, controller 'my_joint_state_broadcaster' is in state 'unconfigured' , expected inactive.
[spawner-4] [ERROR] [1733123792.294592133] [spawner_my_joint_state_broadcaster]: Failed to configure controller

[saikishor]

Hello @louietouie!

Can you try setting the parameter lock_memory: false to the controller_manager? (FYI: ros-controls/ros2_controllers#1389). This is already addressed in #1890.

I would like to know if you have a real-time kernel on your Raspi setup or not?

Thank you!

[louietouie]

lock_memory: false fixed it, thanks @saikishor! I apologize for opening an issue that's already been addressed... I hadn't found those threads you linked to.

I haven't heard of a real-time kernel, but if it is what is described in the jazzy docs, I have not set that up.

[saikishor]

lock_memory: false fixed it, thanks @saikishor! I apologize for opening an issue that's already been addressed... I hadn't found those threads you linked to.

I haven't heard of a real-time kernel, but if it is what is described in the jazzy docs, I have not set that up.

Awesome. Don't worry. Good to hear that this is fixed. We are trying to make the lock_memory default for the kernels with realtime-patches (#1896).

I'll go ahead and close the issue for now.

Thank you!