Sharing camera_sensor_broadcaster, lidar_sensor_broadcaster, and related controllers — looking for help integrating them upstream

[GilmarCorreia]

Hi ROS 2 community,

About five years ago I started working on a project to deliver online simulation for students and instructors, built on top of CoppeliaSim simulator. The project streamed the full 3D simulation to users' browsers via WebSockets and WebGL, so they could develop and run simulations directly from the browser. Early on, I was responsible for building the communication layer between ROS 2 and the simulation platform. That's when I started working with ROS 2 Foxy and the ros2_control framework, with an eye toward eventually integrating with real robotic platforms.

Some of our applications had a direct dependency on cameras and other sensors. In 2022 I began developing what I called camera_sensor_broadcaster, which later grew into a lidar_sensor_broadcaster, a time_sensor_broadcaster, and a state_machine_command_controller. A year later I started porting this work to Humble, replicating what had been done for Foxy.

Around that time I became the maintainer and owner of the DropS project (video here, if you'd like to check it out: https://www.youtube.com/watch?v=5cxuWUQ_SjQ), and we always intended to share these results with the community. Over time, though, investor interest in the project faded, and I ended up developing most of this on my own. My attempt to keep sharing progress was to try porting everything to Jazzy, but it's been about a year now and I haven't had the time to move that forward.

I don't know whether these controllers/broadcasters would be as useful to the community as they've been to us, but if it makes sense, I'd really appreciate help integrating these packages into the official repositories. If any maintainer or contributor is willing to support this transfer, it would mean a lot to see this work benefit other developers.

The repositories are public and referenced below:

https://github.com/NTU-AI/ros2_controllers/tree/humble (Foxy and Humble versions)
https://github.com/NTU-AI/ros2_control/tree/humble (Foxy and Humble versions)
https://github.com/GilmarCorreia/ros2_control (attempt to migrate to Jazzy)
https://github.com/GilmarCorreia/ros2_controllers (attempt to migrate to Jazzy)

We also have a couple of other packages we'd like to contribute: sim_ros2_manager (a plugin for managing scenes and spawns in CoppeliaSim — https://github.com/NTU-AI/sim_ros2_manager) and sim_ros2_control (a plugin that integrates the ros2_control framework with the CoppeliaSim simulator — https://github.com/NTU-AI/sim_ros2_control). My idea it was to split some development made in sim_ros2_control to sim_ros2_manager, for the manager be more focused on working with the simulation control.

Thanks in advance to anyone who can help. Feel free to reach out on LinkedIn to discuss next steps or new ideas. Really appreciate it.

[christophfroehlich]

Can you tell us more about the broadcasters, why do you need Camera/Lidar inside the realtime loop?

And I'm not sure if we have resources to maintain another simulation plugin. I never considered using this simulator, maybe @MarqRazz or @saikishor has an opinion on that?

[GilmarCorreia]

Good morning, and thanks for the question!

To clarify: these broadcasters weren't designed for hard real-time guarantees in the strict sense. The motivation was closer to keeping sensor data synchronized with the same update cycle as other hardware states — particularly useful in simulation, where all sensor data (joint states, camera frames, LIDAR scans) is generated at the same simulation step, and exposing them as state interfaces gives a unified, consistent snapshot per cycle rather than an out-of-band topic with its own timing.

For the camera broadcaster specifically, the intended use case leans toward lightweight, low-resolution sensors — e.g. simple color/lighting sensors that only need a handful of pixels to operate — where the per-cycle overhead is small enough not to threaten loop timing.

For LIDAR, I'll admit the real-time justification is weaker — it's more about 2D scan data being available through the same state-interface mechanism as other sensors in simulation, rather than a strict RT requirement. I'm open to feedback on whether that belongs better outside the control loop as a regular non-RT node.

Regarding the simulator support question — completely understandable, and I don't expect ros2_control to take on maintaining a CoppeliaSim plugin as part of the core project. I'll open a separate discussion in the relevant simulator-adjacent repositories to gauge interest there; I mentioned it here mainly for context on the ecosystem this was originally built for.

[saikishor]

Hello!

I personally don't think camera_sensor_broadcaster or the lidar_sensor_broadcaster makes sense in ros2_control, most of the manufacturers just provide drivers that publish the data, and moreover, it gets tricky when there are multiple cameras or sensors connected. Additionally, these type of sensors are not real-time as you mentioned. For those reasons, it would be better to not integrate them into the official repositories.

[GilmarCorreia]

thanks for the feedback, I appreciate it. The use case I had in mind was more simulation-specific. Thank you.

[saikishor]

thanks for the feedback, I appreciate it. The use case I had in mind was more simulation-specific. Thank you.

Sure, but for the simulation specific why not directly from the simulator? And why via ros2_control?

[GilmarCorreia]

Good question. At the start of the project, we were developing simulation support across multiple backends — CoppeliaSim, Gazebo Classic, and Ignition.

Initially, it seemed reasonable to build simulator-specific controllers and broadcasters for each one individually. But as the project grew, and given the need to spawn robots/sensors dynamically (the scene were edited on-the-fly, with several components names), we moved toward making sensor and joint spawning as generic as possible across simulators. That's where ros2_control's interfaces fit naturally.

We knew building broadcaster support for these sensor types from scratch would take work, but it seemed more robust to replicate a unified sensor-spawning pattern through the framework rather than maintaining a separate plugin per simulator. With this approach, adding a new sensor instance is just a matter of adjusting the URDF and namespace — no need to write and maintain a new plugin/file per component per simulator.

As a side benefit, this also makes the same robot/sensor description portable between simulations.

[saikishor]

@GilmarCorreia I understand, but that not a good enough reason in my opinion. Moreover, I don't think the state and command interfaces right now support cameras and LIDAR sensors. i assume you are taking the pointer and then extracting all the information, right?. This pointer approach is deprecated now and will no more be available in future, so the cameras and lidars won't be supported IMO.

[GilmarCorreia]

Yes, that's correct — in the implementation, I extended the ReadOnlyHandle to accept a pointer to a std::vector< double > (or another container type), so the state interface could expose array-like data rather than just a single scalar per interface.

I understand the pointer-based approach is being phased out, but wouldn't it be worth considering array/vector support as a general addition to the state/command interface types?

That would open the door to other sensor types that naturally return arrays (e.g.: joysticks).