feat(setup-dev-env.sh): add --locked option for reproducible dependency installation by youtalk · Pull Request #6934 · autowarefoundation/autoware

youtalk · 2026-03-24T18:33:33Z

Related to #6862

Description

Add an Ansible lockfile mechanism to pin package versions for reproducible builds using standard APT preferences (/etc/apt/preferences.d/).

Architecture

Lockfile format: Flat YAML dict (package: version), alphabetically sorted, one file per distro/arch combination (ansible/vars/locked-versions-{distro}-{arch}.yaml)
Version pinning: New version_lock Ansible role generates /etc/apt/preferences.d/autoware-lock with Pin-Priority: 1001, which makes APT automatically install the locked version (including downgrades)
Role changes: Each existing role only adds allow_downgrade (1 line per apt task). No version concatenation or conditional logic in roles
pip packages: Included in the same lockfile (e.g., gdown), referenced via locked_packages['gdown'] in the gdown role

Key design decisions

APT preferences instead of inline version pinning — roles stay close to upstream, minimal diff
Generator reads package names from the existing lockfile — no hardcoded package lists in the script
Generator fails immediately if any package (APT or pip) is not installed — prevents partial lockfiles
ROS_DISTRO must be explicitly set when running the generator — no implicit defaults

Current lockfile status

Lockfile templates are committed with empty versions. Run the generator on a fully provisioned environment to fill in actual versions:

ROS_DISTRO=jazzy ./ansible/scripts/generate_ansible_lockfile.sh

Files added

ansible/roles/version_lock/tasks/main.yaml — loads lockfile and manages APT preferences
ansible/roles/version_lock/templates/autoware-lock.pref.j2 — APT preferences template
ansible/scripts/generate_ansible_lockfile.sh — generates lockfile from installed versions
ansible/scripts/validate_lockfiles.sh — validates lockfile format (flat YAML dict)
ansible/vars/locked-versions-{humble,jazzy}-{amd64,arm64}.yaml — 4 lockfile templates

Usage

# Normal mode (existing behavior, unchanged)
./setup-dev-env.sh -y

# Locked mode (install pinned versions)
./setup-dev-env.sh -y --locked

# Generate lockfile from current environment
ROS_DISTRO=jazzy ./ansible/scripts/generate_ansible_lockfile.sh

# Validate lockfiles
./ansible/scripts/validate_lockfiles.sh

How was this PR tested?

pre-commit run --all-files passes (yamllint, shellcheck, shfmt, prettier, etc.)
ansible/scripts/validate_lockfiles.sh passes on all 4 lockfile templates
ansible-playbook --syntax-check passes for both locked and unlocked modes
ansible-playbook --check (dry run) confirms correct task flow:
- Unlocked: version_lock skips loading/pinning, attempts to remove stale pins
- Locked: version_lock loads lockfile and creates APT preferences
Generator correctly fails on missing packages without writing partial lockfiles

github-actions · 2026-03-24T18:33:45Z

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

You've checked our contribution guidelines.
Your PR follows our pull request guidelines.
All required CI checks pass before marking the PR ready for review.

…installation Add Ansible lockfile mechanism using APT preferences for version pinning: - Add --locked option to setup-dev-env.sh - Add version_lock role that generates /etc/apt/preferences.d/autoware-lock - Add lockfile templates (humble/jazzy x amd64/arm64) as flat YAML dicts - Add generate and validate scripts for lockfile management - Exclude ansible/vars/ from ansible-lint (package names as keys) Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

Add allow_downgrade conditioned on use_locked_versions to all apt tasks in roles managed by the lockfile. APT preferences handle version pinning, so roles only need this single-line addition. Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

Wrap ros-apt-source download tasks in a block with when condition to skip network access when using locked versions. The APT source is already configured in locked environments. Signed-off-by: Yutaka Kondo <yutaka.kondo@youtalk.jp>

Copilot

Pull request overview

Adds an opt-in “locked” installation path to the dev environment setup to support reproducible dependency versions via Ansible-managed APT pinning and a per-distro/arch lockfile.

Changes:

Add --locked flag to setup-dev-env.sh to enable locked mode and pass lockfile path into Ansible.
Introduce a version_lock role that loads a lockfile and writes APT preferences pins, and update existing roles to allow downgrades when locked mode is enabled.
Add lockfile templates plus scripts to generate/validate lockfiles.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
setup-dev-env.sh	Adds `--locked` flag and passes `use_locked_versions` + `lockfile_path` to Ansible.
ansible/vars/locked-versions-jazzy-arm64.yaml	Adds jazzy/arm64 lockfile template (empty versions).
ansible/vars/locked-versions-jazzy-amd64.yaml	Adds jazzy/amd64 lockfile template (empty versions).
ansible/vars/locked-versions-humble-arm64.yaml	Adds humble/arm64 lockfile template (empty versions).
ansible/vars/locked-versions-humble-amd64.yaml	Adds humble/amd64 lockfile template (empty versions).
ansible/scripts/validate_lockfiles.sh	Adds a script to validate lockfile YAML is a flat dict.
ansible/scripts/generate_ansible_lockfile.sh	Adds a script to generate pinned versions from installed packages.
ansible/roles/version_lock/tasks/main.yaml	Adds role tasks to load lockfile and manage `/etc/apt/preferences.d/autoware-lock`.
ansible/roles/version_lock/templates/autoware-lock.pref.j2	Adds APT preferences template for per-package version pins.
ansible/roles/ros2_dev_tools/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/roles/ros2/tasks/main.yaml	Skips dynamic `ros-apt-source` install in locked mode; allows downgrades for ROS metapackage install.
ansible/roles/rmw_implementation/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/roles/geographiclib/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/roles/gdown/tasks/main.yaml	Pins `gdown` version via lockfile in locked mode; allows downgrades for `pipx` APT install.
ansible/roles/docker_engine/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/roles/dev_tools/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/roles/build_tools/tasks/main.yaml	Allows downgrades when locked mode is enabled.
ansible/playbooks/universe.yaml	Adds `autoware.dev_env.version_lock` role to the universe playbook.
.cspell.json	Ignores `ansible/vars/` in cspell.
.ansible-lint	Excludes `ansible/vars/` from ansible-lint.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-26T05:40:33Z

+- name: Load lockfile
+  ansible.builtin.include_vars:
+    file: "{{ lockfile_path }}"
+    name: locked_packages
+  when: use_locked_versions | default(false) | bool
+
+- name: Create APT version pins
+  become: true
+  ansible.builtin.template:
+    src: autoware-lock.pref.j2
+    dest: /etc/apt/preferences.d/autoware-lock
+    mode: "0644"
+  when: use_locked_versions | default(false) | bool


When use_locked_versions is enabled, this role assumes lockfile_path is defined and that the loaded locked_packages contains non-empty versions. If either is missing (e.g., running the playbook directly, or using the committed template lockfiles), the failure mode will be an undefined-variable error or later package install failures. Consider adding an explicit assert/fail step under the same when: to (1) require lockfile_path and (2) ensure all locked versions are populated before templating pins.

Copilot · 2026-03-26T05:40:33Z

 - name: Install gdown to download files from CMakeLists.txt
  community.general.pipx:
-    name: gdown
+    name: "{{ 'gdown==' ~ locked_packages['gdown'] if use_locked_versions | default(false) | bool else 'gdown' }}"
    executable: /usr/bin/pipx


In locked mode this constructs gdown==<version> from locked_packages['gdown']. With the current committed lockfile templates this value loads as null and will render as None, producing an invalid requirement string and a confusing pipx error. Add a clearer early validation (e.g., in version_lock or this role) that locked_packages.gdown is defined and non-empty when use_locked_versions is true, so the failure is actionable.

Copilot · 2026-03-26T05:40:34Z

+- name: Download and install ros-apt-source package
+  when: not (use_locked_versions | default(false) | bool)
+  block:
+    - name: Get latest release information of ros-apt-source package
+      ansible.builtin.uri:
+        url: https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest
+        method: GET
+        return_content: true
+      register: ros2__ros_apt_release_info


In locked mode this skips installing ros-apt-source, but the subsequent apt install of ros-<distro>-<type> still relies on the ROS 2 APT sources being present. On a fresh machine, --locked is likely to fail because the ROS repository isn’t configured. Consider installing ros-apt-source in locked mode as well (pinned via the lockfile), rather than skipping the repo setup entirely.

Copilot · 2026-03-26T05:40:34Z

+# Check locked option
+ansible_args+=("--extra-vars" "use_locked_versions=${option_locked:-false}")
+if [ "$option_locked" = "true" ]; then
+    arch=$(dpkg --print-architecture)
+    lockfile="$SCRIPT_DIR/ansible/vars/locked-versions-${rosdistro:-humble}-${arch}.yaml"
+    if [ -f "$lockfile" ]; then
+        ansible_args+=("--extra-vars" "lockfile_path=${lockfile}")
+    else
+        echo -e "\e[31mError: Lockfile not found: $lockfile\e[0m" >&2
+        exit 1
+    fi
+fi


--locked sets use_locked_versions/lockfile_path, but the lock mechanism only takes effect in playbooks that include the autoware.dev_env.version_lock role (currently only universe.yaml). If users run setup-dev-env.sh <other-playbook> --locked, the flag will appear to work but won’t actually pin APT versions. Consider either adding the role to the other supported playbooks, or failing fast here when --locked is used with a playbook that doesn’t apply the lock.

Copilot · 2026-03-26T05:40:34Z

+    # Resolve all versions first, fail if any package is missing
+    declare -A versions
+    local has_error=false
+
+    # APT packages
+    for pkg in $packages; do
+        local ver
+        ver=$(dpkg-query -W -f='${Version}' "$pkg" 2>/dev/null) || true
+        if [[ -z $ver ]]; then
+            echo "Error: APT package '$pkg' is not installed." >&2
+            has_error=true
+        else
+            versions[$pkg]=$ver
+        fi
+    done


The generator currently treats every key in the lockfile as an APT package first. Since the lockfile includes pip-only entries like gdown, dpkg-query will fail for those keys and has_error will be set even when the pip package is installed, preventing lockfile generation in a fully provisioned environment. The generator should distinguish APT vs pip entries (e.g., try dpkg-query then fall back to pip metadata, or encode the package type in the lockfile) so mixed lockfiles can be generated successfully.

Copilot · 2026-03-26T05:40:35Z

+    # pip packages (included in the same lockfile)
+    local pip_pkgs=("gdown")
+    for pkg in "${pip_pkgs[@]}"; do
+        local ver
+        ver=$(pip3 show "$pkg" 2>/dev/null | grep '^Version:' | awk '{print $2}' || true)
+        if [[ -z $ver ]]; then
+            echo "Error: pip package '$pkg' is not installed." >&2
+            has_error=true
+        else
+            versions[$pkg]=$ver
+        fi
+    done


This hardcoded pip_pkgs list contradicts the stated design goal of “no hardcoded package lists” and means adding another pip dependency to the lockfile won’t be picked up by the generator. Consider deriving pip package names from the lockfile content (or a structured schema) so the generator’s behavior is fully driven by the lockfile.

xmfcx · 2026-03-26T06:39:43Z

This file will be removed as part of Simplify the docker images and workflows #6852

Could you postpone this dependency pinning task after that issue is completed?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

youtalk · 2026-03-26T13:05:13Z

@xmfcx Yes, I could.

youtalk force-pushed the feat/ansible-lockfile branch 2 times, most recently from 19f2255 to 8a10e7b Compare March 24, 2026 19:58

youtalk self-assigned this Mar 25, 2026

youtalk force-pushed the feat/ansible-lockfile branch from 3bf7650 to 966d6c9 Compare March 25, 2026 18:49

youtalk changed the title ~~feat(setup-dev-env): add --locked option for reproducible dependency installation~~ feat(setup-dev-env.sh): add --locked option for reproducible dependency installation Mar 25, 2026

youtalk force-pushed the feat/ansible-lockfile branch 2 times, most recently from 976416d to 27886d3 Compare March 26, 2026 03:56

youtalk added 3 commits March 25, 2026 21:02

youtalk force-pushed the feat/ansible-lockfile branch from 27886d3 to b21d086 Compare March 26, 2026 04:03

youtalk added the run:health-check Run health-check label Mar 26, 2026

youtalk marked this pull request as ready for review March 26, 2026 05:35

youtalk requested review from isamu-takagi, mitsudome-r, oguzkaganozt and xmfcx as code owners March 26, 2026 05:35

Copilot AI review requested due to automatic review settings March 26, 2026 05:35

Copilot started reviewing on behalf of youtalk March 26, 2026 05:36 View session

Copilot AI reviewed Mar 26, 2026

View reviewed changes

youtalk and others added 2 commits March 26, 2026 05:56

Update ansible/roles/version_lock/templates/autoware-lock.pref.j2

c51e4b6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update ansible/scripts/generate_ansible_lockfile.sh

1bce85b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

youtalk added the type:build Tooling and infrastructure around building the Autoware. label Mar 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(setup-dev-env.sh): add --locked option for reproducible dependency installation#6934

feat(setup-dev-env.sh): add --locked option for reproducible dependency installation#6934
youtalk wants to merge 5 commits intoautowarefoundation:mainfrom
youtalk:feat/ansible-lockfile

youtalk commented Mar 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Uh oh!

xmfcx commented Mar 26, 2026

Uh oh!

youtalk commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

youtalk commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Architecture

Key design decisions

Current lockfile status

Files added

Usage

How was this PR tested?

Uh oh!

github-actions Bot commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

xmfcx commented Mar 26, 2026

Uh oh!

youtalk commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

youtalk commented Mar 24, 2026 •

edited

Loading

github-actions Bot commented Mar 24, 2026 •

edited

Loading