Launching a paused simulation makes spawners from the same launchfile fail

[christophfroehlich]

After the refactoring done by ros-controls/ros2_control#1638, the switch_controller service has a working timeout for each controller. This time is hardcoded within the spawner.py to 5s.

If the simulation is started paused, like with omitting the -rargument for gz_sim.launch.py

        IncludeLaunchDescription(
            PythonLaunchDescriptionSource(
                [PathJoinSubstitution([FindPackageShare('ros_gz_sim'),
                                       'launch',
                                       'gz_sim.launch.py'])]),
            launch_arguments=[('gz_args', [' -v 4 empty.sdf'])]),

then the CM services are started but the update method is not called yet. This leads to running into the timeout

[ruby $(which gz) sim-1] [ERROR] [1727769677.868599778] [controller_manager]: Switch controller timed out after 5.000000 seconds!
[spawner-4] [ERROR] [1727769677.869656433] [spawner_joint_state_broadcaster]: Failed to activate controller

The same happens if a running simulation is paused and a service call to the CM is done.

Same as ros-controls/gazebo_ros2_control#380

[wltjr]

Is there a work around for this?

There also seems to be some side effects, not sure the lidar sensor starts and slam toolbox never seems to, even when I unpause in under 5 seconds. Here are some log files from startup under Iron/Fortress and then under Jazzy/Harmonic, but it seems its more about the changes to this package than Jazzy or Harmonic. Maybe same issue if I update to latest in Iron.
ros2_iron_fortress.txt
ros2_jazzy_harmonic_paused.txt
ros2_jazzy_harmonic_unpaused.txt

Atm I am unable to use Gazebo Harmonic under Jazzy, sticking with older not current ( available updates not applied ) Gazebo Fortress under Iron which is EOL now. Its moot, just trying to move to currently supported env that is LTS.

Thank you for your efforts!

[christophfroehlich]

the fix is the linked PR with setting a higher timeout. have you tried that?
I don't think that the issue in the first post effects any other plugins or toolboxes, however, it can be that the slam toolbox expects odometry/joint_states at launch but those aren't sent before unpausing.

[wltjr]

I assume you mean --controller-manager-switch-timeout, and I pass that as an argument when I am launching a CM controller. I will try that, but timeout aside, even when I fast unpause Gazebo never hitting the timeout, slam toolbox never starts. There is some other missing slam toolbox output from earlier on, usually before gz_ros2_control output starts from slam toolbox that is missing. Lidar service does seem up and running, I can see that data in rviz2. I just cannot get things going with slam toolbox for some reason. It seems slam toolbox is waiting for something.

I never see these lines from slam toolbox before gz_ros2_control starts
[async_slam_toolbox_node-6] [INFO] [1730817215.838625905] [slam_toolbox]: Node using stack size 40000000 [async_slam_toolbox_node-6] [INFO] [1730817215.868619487] [slam_toolbox]: Using solver plugin solver_plugins::CeresSolver [async_slam_toolbox_node-6] [INFO] [1730817215.868884230] [slam_toolbox]: CeresSolver: Using SCHUR_JACOBI preconditioner.

Or this after I unpause and CM/joint broadcaster starts, avoiding the timeout
[async_slam_toolbox_node-6] Registering sensor: [Custom Described Lidar]

[wltjr]

Looks like I am on an older version per the comments, I have an older argument.

[gazebo-1] [INFO] [1730821015.714333422] [controller_manager]: Activating controllers: [ joint_state_broadcaster ]
[gazebo-1] [WARN] [1730821017.620083592] [gz_ros_control]:  Desired controller update period (0.1 s) is slower than the gazebo simulation period (0 s).
[gazebo-1] [ERROR] [1730821020.714456731] [controller_manager]: Switch controller timed out after 5.000000 seconds!
[spawner-11] [ERROR] [1730821020.717520662] [spawner_joint_state_broadcaster]: Failed to activate controller
[ERROR] [spawner-11]: process has died [pid 14911, exit code 1, cmd '/opt/ros/jazzy/lib/controller_manager/spawner joint_state_broadcaster -c /controller_manager --controller-manager-timeout 10 --ros-args'].

The issue with slam toolbox maybe unrelated and specific to changes in that package for jazzy or newer not sure. There maybe two unrelated issues, or the issue might be related.

[BengIslam7]

Any solution?

[christophfroehlich]

@BengIslam7 have you tried the --controller-manager-switch-timeout as mentioned above?

[BengIslam7]

@christophfroehlich yes but always the same error!

[christophfroehlich]

sure, it will timeout unless you unpause the simulation. set the timeout high enough (your log writes that you have set it to 5.0s), wait for the simulation to finish startup and then unpause it.

[github-actions]

This issue is being labeled as stale because it has been open 45 days with no activity. It will be automatically closed after another 45 days without follow-ups.