lyrical images pin the ROS package version but not the apt source, so installing any extra package can break ABI

[YuanYuYuan]

The ros:lyrical-* images pin the ROS package version but not the apt source. Installing any extra ROS package therefore mixes repo-fresh binaries with the older ones baked in. When the two disagree on ABI, the result is memory corruption rather than a link error.

Four lines reproduce it on a stock image, with the default RMW and no third-party code:

docker run --rm --network host ros:lyrical-ros-base bash -c '
  apt-get update -qq && apt-get install -y -qq ros-lyrical-demo-nodes-cpp
  source /opt/ros/lyrical/setup.bash
  ros2 run demo_nodes_cpp add_two_ints_server &
  sleep 5
  ros2 run demo_nodes_cpp add_two_ints_client
'
# double free or corruption (out)
# [ros2run]: Aborted            → exit 250

demo_nodes_cpp's own client crashes against demo_nodes_cpp's own server.

What is happening

Package	Source	Version	Built
ros-lyrical-rclcpp	baked into the image	32.0.0	2026-06-06
ros-lyrical-rclcpp	available in the repo	32.0.2	2026-07-30
ros-lyrical-demo-nodes-cpp	installed from the repo	0.37.9	2026-07-30

ros-core/Dockerfile installs ros2-apt-source. Its ros2.sources points at http://packages.ros.org/ros2/ubuntu, which is the live repository. The Dockerfile then installs ros-lyrical-ros-core=0.13.0-3*.

The version is pinned. The source is not. So the pin resolved against the repository of the build date, and every later apt-get install resolves against a different one.

apt has no reason to upgrade rclcpp, because the dependency is already satisfied. demo_nodes_cpp 0.37.9 therefore loads librclcpp.so 32.0.0, having been compiled against 32.0.2.

Diagnosis

rclcpp::ExecutorOptions holds a pimpl, so ~ExecutorOptions() performs delete impl_. demo_nodes_cpp allocates one on the stack in send_request. When the layouts differ, the destructor frees whatever sits at that offset — here a stack address:

Invalid free() / delete / delete[] / realloc()
   at operator delete(void*, unsigned long)
   by rclcpp::ExecutorOptions::~ExecutorOptions()   (librclcpp.so)
   by send_request(...)  →  main
 Address 0x1ffefffb60 is on thread 1's stack

glibc reports the same corruption as double free or corruption (out).

Under valgrind the request itself succeeds (Result of add_two_ints: 5). The data path is correct. The crash is on teardown.

Installing rclcpp explicitly makes apt upgrade it, and the crash disappears:

apt-get install -y ros-lyrical-demo-nodes-cpp ros-lyrical-rclcpp
# ros-lyrical-rclcpp 32.0.2-1resolute.20260730.192517
# [INFO] Result of add_two_ints: 5     → exit 0

No package has a code defect. This is an image/repository coherence problem.

A fix that works: pin the source, not just the version

ROS publishes snapshot repositories. http://snapshots.ros.org/lyrical/2026-06-08/ holds exactly the set this image ships: rclcpp 32.0.0 and demo_nodes_cpp 0.37.8, both built 2026-06-06. Pointing the image's apt source there makes the skew impossible by construction.

I verified this by rebuilding ros-core and ros-base with the source repointed before the first ROS install:

	stock image	patched build
ros-lyrical-rclcpp	32.0.0-1resolute.20260606.021228	identical
ros-lyrical-ros-core	0.13.0-3resolute.20260606.042109	identical
ros-lyrical-ros-base	0.13.0-3resolute.20260606.042427	identical

The patch changes what the source resolves to. It does not change what the image contains. The reproduction then passes against the patched image with no in-container modification: Result of add_two_ints: 5, exit 0.

Patching ros-core alone is enough. ros-base is FROM it and inherits both the rewritten source and the keyring. I verified this by building rather than assuming it.

Diff: osrf:master...YuanYuYuan:pin-lyrical-apt-source-to-snapshot. It is evidence that the approach works, not a merge candidate — see the questions below.

The whole change to ros/lyrical/ubuntu/resolute/ros-core/Dockerfile is one block, placed after ENV ROS_DISTRO and before the first ROS install (comments trimmed here; the diff carries them in full):

 ENV ROS_DISTRO=lyrical
 
+ARG ROS_SNAPSHOT=2026-06-08
+COPY ./ros-snapshots.asc /usr/share/keyrings/ros-snapshots.asc
+RUN printf 'Types: deb\nURIs: http://snapshots.ros.org/%s/%s/ubuntu\nSuites: resolute\nComponents: main\nSigned-By: /usr/share/keyrings/ros-snapshots.asc\n' \
+      "$ROS_DISTRO" "$ROS_SNAPSHOT" > /usr/share/ros-apt-source/ros2.sources \
+    && cat /usr/share/ros-apt-source/ros2.sources
+
 # install ros2 packages
 RUN apt-get update && apt-get install -y --no-install-recommends \
     ros-lyrical-ros-core=0.13.0-3* \

The second file is ros-snapshots.asc, the snapshot signing key, carried in the build context because it is not published on ros.org — see the note below.

Important

Two things make the naive version of this change fail:

1. snapshots.ros.org uses a different signing key from packages.ros.org: 4B63CF8FDE49746E98FA01DDAD19BAB3CBF125EA (ROS Snapshot builder) versus C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654 (Open Robotics). Repointing the URI alone gives NO_PUBKEY, then The repository … is not signed, then Unable to locate package. That key does not appear to be published as a file: snapshots.ros.org/ros.key returns 404, rosdistro does not carry it, and it is absent from ros-apt-source release assets. The only distribution point I found is keyserver.ubuntu.com.
2. Snapshot repositories serve no Sources index. Types: deb deb-src therefore warns on every apt-get update. It must be Types: deb.

Questions for maintainers

This is a policy change rather than a bug fix, so these look like your decisions:

1. The change belongs in the template, not the generated Dockerfile — docker_templates/templates/docker_images_ros2/create_ros_core_image.Dockerfile.em, plus a way to pass a per-distro snapshot date through images.yaml.em. That is why this is an issue and not a pull request.
2. Selecting the date is the hard part. It must be the newest snapshot at or before the sync the image was built from. The pipeline has no notion of that today (see Add ROS_SYNC_DATE to dockerfile busting cache for each sync #775). A snapshot newer than the image reintroduces the same skew.
3. Users lose newer packages and ROS security updates unless they opt out. Leaving the live source present but disabled may be better than overwriting it.
4. Snapshot retention matters. Only four lyrical snapshots exist today. If old ones are pruned, images pinned to a pruned date break outright.
5. Key provenance is weak. The only distribution point for the snapshot key is a keyserver. Publishing it at a canonical URL looks like a prerequisite, and may deserve its own issue.
6. ros2-apt-source owns /usr/share/ros-apt-source/ros2.sources. Overwriting it fights dpkg, because upgrading that package restores the live URL. A separate sources file, plus disabling the shipped one, would be more robust.

Related

• **undefined symbol** ERROR #828 — same root cause on rolling, with a louder symptom (undefined symbol: rcutils_strnlen instead of heap corruption).
• Python ABI mismatch: _rclpy_pybind11.cpython-312-*.so installed under lib/python3.14/site-packages/ #883 — image/repository ABI skew in the Python tree.
• Recent ros:kinetic-ros-base-xenial causes some ROS packages to crash #198 — the same shape in 2018 on kinetic.
• ZettaScaleLabs/hiroz#303 — where this was found and diagnosed, with the full valgrind session and the ruled-out hypotheses.

The problem recurs across distros and across years. That is the argument for pinning the source, rather than for rebuilding images more often.