ci: add sync-params workflow

[paulsohn]

Description

Tracking issue: autowarefoundation/autoware#7048

Introduce sync-params workflow that updates configurated *.param.yaml with custom overrides.

As described in sync-params.yaml in this PR, the configuration goes like this.

perception: # category
  - repository: autowarefoundation/autoware_universe
    ref: main
    files:
      - # Source param path in source repository (i.e. in autoware_universe).
        source: perception/autoware_image_object_locator/config/bbox_object_locator.param.yaml
        # List of param paths in current repository that are variants of the source param path.
        # Variant files should have the same yaml file layout as the source file and only differ in the parameters.
        variants:
          - path: autoware_launch/config/perception/object_recognition/detection/camera_vru_detection/near_range_camera_vru_detector.param.yaml

In this case, the python run with category python3 ./.github/scripts/sync_params.py perception clones the upstream repository to a temp directory, copies source (autoware_universe)/perception/autoware_image_object_locator/config/bbox_object_locator.param.yaml to each variant path - in this case only (autoware_launch)/autoware_launch/config/perception/object_recognition/detection/camera_vru_detection/near_range_camera_vru_detector.param.yaml with a comment header. and duplicates the original file content as a comment block at the bottom of the file

In variant files, leaf fields (i.e. scalar or list of scalars) can be marked overridden by having a comment starting with {OVERRIDE} or {OVERRIDE: reason to override} (the reason can be any text without newlines or braces). In this case, the marked leaf field is overlaid upon the original. So if you want to override a certain value, add # {OVERRIDE} or # {OVERRIDE: reason} comment (or prepend the existing one with it).

Example

suppose the upstream param file looks like:

foo: 42
bar: hoge
baz: 1

and the corresponding downstream variant:

foo: 42
bar: hogehoge # {OVERRIDE}
baz: 2 # {OVERRIDE}

now suppose the upstream param file changes:

-foo: 42
+foo: 40 # updated
-bar: hoge
+bar: fuga
-baz: 1 # removed
+qux: piyo # added

then after syncing, the downstream will be:

-foo: 42
+foo: 40 # updated
 bar: hogehoge # {OVERRIDE} # not updated
- baz: 2 # {OVERRIDE} # dropped (upstream is gone)
+qux: piyo # added

Check mode

The check-params workflow runs sync_params.py --check <category> on every PR. In check mode the script performs the same computation as update mode - cloning upstream, applying overrides, building the expected variant content - but writes nothing to disk, and exits non-zero if any variant has drifted from its expected state.

Check mode uses the SHA pinned in each variant's header (the # Source: line) rather than the current upstream HEAD. Upstream changes between the pinned SHA and HEAD are intentionally ignored, because those are the responsibility of update-params. A freshly synced variant will always pass check mode until someone edits it by hand without going through the sync workflow.

Rationales on designs

Q. Why do we have categories instead of bumping all param files at once?

perception / localization / etc have different codeowners, so one large PR updating all param files across all categories will need many approvals to sync. The exact granularity is a subject to change (e.g. we can have perception/object_recognition and perception/prediction categories split instead of one large perception category) but it is beyond the scope of this PR.

Q. Why shouldn't we only annotate the overridden value with the original one?

In the recent SW WG we decided not to keep upstream param file as footer, while my original suggestion was having a full copy footer. So the choice was all or none - not selectively marking overridden fields.

Maintaining 'shadowed value' marker has two limitations: one is the technical limitation of ruamel and the other is maintenance cost - if a maintainer should override a previously not overridden field, the maintainer should know the exact format of the shadowed value marker, and this can be tricky to define when some fields are actually matrices that span across multiple lines.

In our current implementation we can have only # {OVERRIDE} marker, which is always one-liner and much simpler.

My argument supporting having full footer than no footer (unaccepted for AWF)

This is for visibility of updates of overridden leaf fields.

We can of course sync upstream source to variants directly, without duplicating the original as a comment footer. However when the upstream changes a leaf field, then the syncing PR might be ignorant to the context of the upstream change because the diff is suppressed by overriding mechanism.

For example, if the upstream param file has a field

some_list: [1, 2]

and the downstream overrides it

some_list: [10, 20] # {OVERRIDE}

and later the upstream changes the field to some_list: [1,2,3,4] because the list size should be four, the sync PR will be signaled that the upstream has been changed, but that exact diff is suppressed so the reviewer might not realize or forget that the override value should be updated to length-4 list e.g. [10, 20, 30, 40]. Having a local copy on this repository exposes the exact diff to the sync PR.

Since the header contains the last SHA that upstream file has changed, we can actually get notified when it has changed, but to see what has been changed, we need to open the link and see the upstream file history.

For real-world integration (e.g. products in my company) I am considering to enable --footer option to have full footer, just as my original suggestion before AWF discussion.

TODOs before getting approval of this PR

• create an actual workflow to sync (both periodically or manually triggered. different category corresponds to different PR branch.)
• create a verification workflow that validates each variant: original file + override = variant, and the original file footer matches to the source (version pinned in the header, not the branch head)
  • Run python3 .github/scripts/sync_params.py --check perception for this.
  • I deferred 'shape consistency check' for now, i.e. whether the list of leaf field keys match between the original content and the variant. But on the second thought I concluded that the removal can be easily noticed, and it is only useful when we MUST remove a parameter which is overridden in the downstream, which should be a rare case (for backward compatibility purpose, the obsolete param should be kept unused)
• get confirmation from perception owners that the configured correspondences between autoware_universe and autoware_launch is correct
• get confirmation from perception owners that the applied override markers match to their intent

Future works beyond the scope of this PR

• Apply this workflow to the rest of perception, localization, map, etc.
• Make the workflow reusable in https://github.com/autowarefoundation/autoware-github-actions

How was this PR tested?

Notes for reviewers

Effects on system behavior

None.

[github-actions]

Thank you for contributing to the Autoware project!

🚧 If your pull request is in progress, switch it to draft mode.

Please ensure:

• You've checked our contribution guidelines.
• Your PR follows our pull request guidelines.
• All required CI checks pass before marking the PR ready for review.

[paulsohn]

memo: this aligns with autowarefoundation/sync-file-templates#56 .

[isamu-takagi]

@paulsohn This action looks very useful. I'm not sure if this is possible, but how about saving the original parameters as comments, like OVERRIDE: <original-parameter>? That way, this action wouldn't need to copy the original file, and the differences would be displayed very clearly.

- some_list: [10, 20] # OVERRIDE: [1, 2]
+ some_list: [10, 20] # OVERRIDE: [1, 2, 3, 4]

[sasakisasaki]

@paulsohn I reviewed a source code and put my understanding as this commit.

Let me consider a way about this @isamu-takagi's proposal.

[paulsohn]

I have seriously reviewed three plausible options:

• full footer (this PR)
• override-only footer (My compromise with Takagi-san's suggestion) -- Created PoC with Copilot aid. 52d5f52
• override-only inline (Takagi-san's suggestion)

And still after the comparison I concluded that full footer solution is genuinely a better choice if we consider maintenance cost as well. Imagine a downstream maintainer wants to override a field which was not previously overridden.

If we choose full footer, they can just add a comment # {OVERRIDE} (or to be more informative, # {OVERRIDE: reason}) and the verification passes. Now, what should they do if we chose override-only footer or inline? They should similarly add an override marker, plus either:

1. learn the correct format of shadowed value markers and manually add them, under OVERRIDDEN VALUE (DO NOT EDIT) pivot (see the commit link for details). If such the pivot didn't exist then the user should add it as well.

2. run sync-params.py locally to automate the above formatting, which binds the maintenance procedure to the local script run

If we inline the shadowed value marker comments then at least we don't have that pivot line, but the responsibility for providing correct marker format is still on the maintainer and we still are facing the matrix issue.

So the maintenance cost makes full footer as an optimal choice, rather than a non-ideal but easy-to-implement workaround.
Duplicate everything in the format footer is indeed noisy, but SSoT is always kept, and override-only alternatives impose a real maintenance burden.

Comparison by Copilot, if interested

One more note: the integration-side maintainer needs to have the full parameter actually applied to the runtime and the original values of overridden fields, not only overriding values. Integration engineers need to understand the upstream intent transparently in case the upstream parameter format or required fields has changed unexpectedly.

---

EDIT: the final consent has been made to have no footer, but I will leave this comment for the record.

[mitsudome-r]

Notes from Software WG: We will make the copy of original parameters optional and set it false for Autoware Launch repo.

[paulsohn]

At 6242fa5 I confirmed in my fork that this sync-params workflow gives the expected PR with information of shadowed changes by override. Upstream footers removed, and appending footer is now spun out as --footer which is disabled on AWF workflow.

paulsohn#14

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

clone_repository() uses git clone --branch <ref> ..., which fails when ref is a raw commit SHA. The script/docstring and config comment indicate ref may be a branch/tag/SHA, so this will break valid configurations. Consider cloning without --branch (or using --no-checkout + git fetch), then git checkout <ref> so SHA pins work reliably.

Suggested change

	run_git(["clone", "--branch", ref, f"https://github.com/{repo}.git", str(repo_dir)])
	# Do not use ``git clone --branch`` here: it accepts branch/tag names but not
	# raw commit SHAs, while ``ref`` is documented to support branch/tag/SHA.
	run_git(["clone", f"https://github.com/{repo}.git", str(repo_dir)])

[paulsohn]

ref is not expected to be a fixed SHA. The whole point of this workflow is to track a moving ref, so branch is expected. Skipping.

[Copilot]

Pull request overview

Copilot reviewed 44 out of 44 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[paulsohn]

@xmfcx @mitsudome-r
I think my recent updates address our latest discussions - removing upstream footer from the existing synced files, making upstream footer optional and disable in the CI. Plus as you can see in paulsohn#14 (this is how the generated PR would look like), shadowed upstream changes are now reported in PR body, so that we don't need any additional bot to track them.

Also DCO check is somehow broken again! Pushed again and worked.