colcon parallel build race condition

[balthasarschuess]

In our build pipeline we encountered a race condition when building multiple rust projects with multiple build jobs enabled.

The setup was similar to the following:

• A is a rust project built through colcon-ros-cargo
• B is a ros package like std_msgs which message definitions are generated for.
• C is a rust project built through colcon-ros-cargo and depends on A and B

The build was triggered with colcon build --packages-up-to C.
It would fail if:

• Job 1 would start building B first and register itself to config.toml
• Job 2 would start building A, cargo ament build would discover package B in config.toml, try to look for the Cargo.toml, but this was not generated by Job 1 yet and the build fails.

I think this can be avoided in two ways:

• modify config.toml on successful subpackage build
• cargo ament-build ignores entries in config.toml that it does not depend on

So this might not be the correct repository to report this for.

[luca-della-vedova]

Thanks for the detailed report!

I think you proposed valid fixes, my 2c (with a disclaimer that I haven't worked in this for some time):

modify config.toml on successful subpackage build

This might be a bit tricky and might require moving the config.toml generation into colcon-cargo as well. It seems this package does a very rudimentary package identification to populate the file instead. I think the aim at least for colcon-cargo is to move away from this generated config file since it's not really a good practice to rely on hidden files in the root of a workspace.

cargo ament-build ignores entries in config.toml that it does not depend on

How would this work? I haven't tried this in a long time but doesn't the error actually come from rustc as soon as you try to refer to a package that doesn't exist in the config.toml?

I wonder as an alternative if we could ensure that the config.toml only ever contains packages that have been built, rather than all the existing cargo packages by doing something like this, for each colcon-ros-cargo package that gets built:

• Look at its dependencies
• Open the config.toml for read / write.
• Iterate through each dependency, if it is a rust package add it to the config.toml.

This should ensure the correct build order because colcon itself would make sure to build each dependency before the package itself.
What do you think?

[balthasarschuess]

Regarding the cargo ament-build proposal, I haven't given much though about it and was under the assumption that this plugin triggers the build process, but apparently it only installs binaries to ros. So let's just discard this.

I think for now we have to rely on config.toml to configure the rust build process in a standard way. It may be possible to write a plugin for cargo, that extends dependency graph resolution to include ros packages declared in a package.xml, but I could not find out with a quick google search if this is technically possible right now.

Regarding your proposed solution. If I understand it correctly this should move the cargo.toml generation, from before building dependencies to after building dependencies.
I think this will work nicely.

As a temporary work-around we built our two rust-ros repostories with two build step and skipped the previous with packages-ski-build-finished. Otherwise the serial executor of colcon can also be used.

[maspe36]

Our message build story is in my mind one of the big items we have yet to fully solve. We are trying to integrate two ecosystems with very different ideas on how build systems should work.

I believe my proposed approach to message generation would not have this issue (at least for shipped ROS msgs) as we will depend on cargo for this parallel dependency resolution. Let me know if you have any thoughts or insights.

ros2-rust/ros2_rust#471