ROS2 Monorepo best practices

[justink-hadrian]

I am working on a ROS2 project with both python and c++ packages. The project is structured as a monorepo. I have seen a number of discussions about pixi with monorepos on the internet, but some of the discussions are dated given the rapid pace of pixi development. I also saw workspace dependencies (discussed #5940 (comment)) just landed in recent versions. I want to make sure I am taking advantage of current best practices.

My current setup:
I have a workspace level pixi.toml with all package dependencies. pixi install from the top level builds everything and gives me a pixi environment I can work with any package from.

|-pixi-ws
| └─ libs
│   └── msgs
│       ├── pixi.toml (pixi-build-ros)
│       ├── package.xml
│       └── msg
│           └── msg_one.msg
│   └── py_node (build depend on msgs)
│       └── src
│           └── py_node.py
│       ├── pixi.toml (pixi-build-ros)
│       ├── package.xml
│       ├── setup.py
│       ├── setup.cfg
│   └── cpp_node (build depend on msgs)
│       └── src
│           └── cpp_node.cpp
|               └─pixi.toml
│       ├── pixi.toml (pixi-build-ros)
│       ├── package.xml
│       ├── CMakeLists.txt
| └─ apps
|       └─ app-1 (runtime depends on python_node/cpp_node from launch script)
|              └── launch
|                   └─ lauch.py
│              ├── pixi.toml (pixi-build-ros)
│              ├── package.xml
│              ├── setup.py
└─pixi.toml
└─pixi.lock

The pixi.toml is along the lines of:

[workspace]
channels = [
    "https://prefix.dev/robostack-kilted",
    "https://prefix.dev/conda-forge",
]
platforms = ["osx-arm64", "win-64", "linux-64"]
preview = ["pixi-build"]

[dependencies]
ros-kilted-msgs = { path = "libs/msgs" }
ros-kilted-py-node = { path = "libs/py_node" }
ros-kilted-app-1 = { path = "apps/app-1" }
ros-kilted-rclpy = ">=9.1.4,<10"
ros-kilted-rclcpp-lifecycle = ">=29.5.6,<30"

[target.linux-64.dependencies]
ros-kilted-cpp-node = { path = "libs/cpp_node" }

The main issue with this setup is that individual apps, the context developers work in most frequently, only use a subset of packages, but they have to build everything to work with the environment. I would like to work with the packages independently and only have to build the necessary dependencies. Something along the lines of what was discussed here: #387 (comment).

It would be nice from a CI perspective to only test what you change (and downstream dependencies). I was also trying to figure out the best way to deploy a single application (+ dependencies) from the monorepo. Does anyone have examples of what this looks like (especially with c++ packages depending on other c++ packages in the monorepo) given the current pixi state of the art? I am currently using pixi-build-ros in each of the packages currently, a pixi.toml like:

[package.build.backend]
channels = ["https://prefix.dev/conda-forge"]
name = "pixi-build-ros"
version = "*"

[package.build.config]
distro = "kilted"

[package.build-dependencies]
ros-kilted-msgs = { path = "../msgs" }

[package.host-dependencies]
ros-kilted-msgs = { path = "../msgs" }

I appreciate any guidance. Excited about where pixi is headed!

[VRichardJP]

Maybe I am wrong, but I start to think pixi-build is not the correct tool for most ROS2 workspaces.

When I work on a ROS2 workspace, I typically modify ROS2 packages source files (code, config, launch), re-compile specific packages with extra compile flags. Sometimes I even break one package and but keep testing other packages with colcon build --continue-on-error. In other words, I need full access to the build files, run colcon commands and source my ROS2 workspace manually.

But this is not what pixi packaging system is designed for. With pixi-build you build ready-to-release packages and install them in the workspace environment so you can use them (not modify or tinker). So pixi-build makes sense for third-party dependencies you will never touch, like a ros-kilted-sensor-msgs package, but not for the packages your are actively developing.

Basically, there are 2 different worlds:

• I want to develop a ROS2 package and need full control -> use simple [workspace], run custom build commands (either manually or using [tasks])
• I want to release/publish pixi packages from a ROS2 workspace -> add [package] in each ROS2 package and list them in top-level workspace [dependencies]

Maybe the solution is to leverage pixi's multi-environment, and create a "publish" environment including all your workspace ROS2 packages into [feature.publish.dependencies]. When you want to develop, you use a "dev" environment. When you want to create release packages, you simply do pixi install --environment publish. Here is for example what this setup could look like (although it is broken): #6277 (comment)

[baszalmstra]

Hey @VRichardJP !

We are working hard on making this experience much better. We support a few different workflows at the moment:

You can use the "normal" pixi workflow and use things like pixi install --only my-package to only rebuild a specific package you touched. You have to use pixi run --as-is then to force pixi not to update the environment automatically. Arguably not the best experience.

The alternative method that we are actively working on is the [dev] feature. Packages in the [dev] dependencies table (or [feature.bla.dev]) are not built and installed directly. Instead, their host-, build-, and run-dependencies are added to the environment. This brings all the development tools needed to build the package into the environment (like compilers etc). But it also requires you to manually build the packages. In your case, you could simply invoke colcon build in that environment. You can set up tasks in this environment to streamline this process.

We are on a mission to replace colcon in this workflow as well. We are currently experimenting with allowing build backends to return "tasks" as well, these can then be invoked when they are included as a dev dependency. This would allow you to just invoke pixi run my-pkg#install without any additional configuration.

You can see an example of what is currently possible here. We are still very much in an experimental phase, but I would love to hear your feedback already!

[VRichardJP]

The [dev] dependencies look very nice indeed. So if I understand correctly, by putting the ROS2 workspace packages under the workspace pixi.toml's [dev] section, the workspace environment will automatically include all package dependencies (e.g. listed in package.xml files).

I wish the experience was simpler though:

• a pixi.toml needs to be created for each package. Since most info is already available in each package's package.xml, the pixi.toml could be reduced to:

[package.build.backend]
channels = ["https://prefix.dev/conda-forge"]
name = "pixi-build-ros"
version = "*"

• in the workspace pixi.toml, it is necessary to specify each package and paths manually
• building and publishing packages must be done manually

While above pixi.toml can be generated with a simple bash script, could there be an auto-discovery mechanism for ROS2 packages instead? Like a workspace option ros-package-auto-discovery = {path = "src"}, which would make pixi parse package.xml files directly. Similarly, I wish there was an option to automatically discover pixi packages in the workspace (like colcon automatically finding package.xml files), so there is no need to add each package path under the workspace dependencies for example:

[dependencies_options]
pixi-package-auto-discoverty = {path = ["./pixi_packages/", "./pixi_recipes/"]}

[justink-hadrian]

@baszalmstra To confirm, the dev features would pull in all dependencies throughout the workspace? Could you point the pixi install at a particular app in my example and it would only pull in dependencies for that package and its dependency packages?

I prototyped an approach where each package was it's own workspace. The idea was that I would have an S3 channel for my released internal packages and developers would mostly pull local dependencies from that and global dependencies when working on a package. For the workflow where a developer was modifying a package and its dependency (ex: adding a message to the msgs package then adding a node in the application package that used it), I figured I could use a local channel to stash the developed dependency then the downstream package would pull from the local channel. I would commit the local channel folder, but none of its packages and CI would build in dependency order, publishing to the local channel for any package that was modified and check in the lock for the downstream dependency to pull from the local cache. Then, when merging to main, CI would build and publish changed packages to the internal S3 channel and update the locks to point to the S3 channel. This would give me strict versioning of internal packages, which would be valuable for a large team working on different applications. The issue I ran into prototyping this approach is that the local channel lock references seem to be absolute path, so CI couldn't use a lock file pointing to the local cache that was generate on my machine. Is this on purpose? I believe if they were relative path to a local channel in my pixi workspace, I could make this workflow work.

Back to my original question about features, if it only pulled in dependencies of packages and their dependency packages, that would help me with the environment isolation and production deployment of just what you need between very different projects (python only application and c++ only applications). I still like the idea of package version management for packages in the monorepo, are there any ideas of how to manage that with dev features?