Skip to content

aws_sdk_vendor: fetch aws-sdk-cpp live via ament_vendor(), drop the separate vendor repo - #437

Merged
Minipada merged 2 commits into
jazzyfrom
feature/434-aws-sdk-vendor-fetch-aws-sdk-cpp-live-fr
Aug 30, 2026
Merged

aws_sdk_vendor: fetch aws-sdk-cpp live via ament_vendor(), drop the separate vendor repo#437
Minipada merged 2 commits into
jazzyfrom
feature/434-aws-sdk-vendor-fetch-aws-sdk-cpp-live-fr

Conversation

@Minipada

@Minipada Minipada commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • aws_sdk_vendor fetches aws-sdk-cpp directly from https://github.com/aws/aws-sdk-cpp.git at the pinned tag via ament_cmake_vendor_package's ament_vendor() macro (VCS_TYPE git), instead of a hand-rolled ExternalProject_Add — the same idiom zmqpp_vendor/tinyxml_vendor/yaml_cpp_vendor already use successfully on the real ROS buildfarm. No aws-sdk-cpp source is vendored anywhere.
  • This supersedes aws_sdk_vendor: flattened source, no network fetch #425/PR feat(aws_sdk_vendor): flattened source, no network fetch #433's flattened-source approach, which rested on a false assumption: ros_buildfarm's own job-generation source runs binarydeb builds with docker run --net=host, and zmqpp_vendor has a real, currently-succeeding Jenkins job doing exactly this live-clone pattern.
  • crt/aws-crt-cpp's submodule chain (including the unrelated CBMC formal-verification model) is now let recurse in full via ament_vendor()'s vcs import --recursive, dropping the previous hand-rolled PATCH_COMMAND exclusion hack — harmless with real network available.
  • GLOBAL_HOOK is required on the ament_vendor() call: dc_bridge only ever calls find_package(AWSSDK), never find_package(aws_sdk_vendor) first, so the install prefix has to reach CMAKE_PREFIX_PATH via an environment hook, not a CMake config-extra.
  • aws_sdk_vendor keeps its own separate repo (github.com/Minipada/aws_sdk_vendor, mirroring vector_vendor's split, ADR-0002's amendment) — this repo's vendor packages live in their own repos as a rule, independent of whether a given package's own content (a thin CMakeLists.txt + package.xml here, no vendored source) is large enough to justify the split on git-bloat grounds alone. ros2_data_collection.repos gains an aws_sdk_vendor entry, the same vcs import path vector_vendor already uses. vector_vendor: fetch the Vector release tarball live via checksum-pinned download, reconsider its separate repo #435 records the same decision for vector_vendor.
  • The satellite repo's aws_sdk_vendor/CMakeLists.txt now also exposes an AWS_SDK_BUILD_ONLY CMake cache variable (default s3, all this workspace needs) so a downstream project can build other AWS service clients without forking the package — forwarded to aws-sdk-cpp's own -DBUILD_ONLY via a $<SEMICOLON>-escaped generator expression (verified: a plain quoted value silently mis-splits a multi-service list before it reaches ExternalProject_Add). It also gets its own CI — a Podman Containerfile build plus a prek-driven format check — mirroring this repo's own conventions.
  • docs/adr/0012-aws-sdk-vendor-flattened-source.md documents the buildfarm network-access finding and both decisions (live fetch, own repo).
  • tools/e2e/Containerfile, .github/workflows/ci.yaml, and tools/e2e/scripts/verify_network_isolation.sh comments updated: the network-isolation-check job is now documented as a permanent, by-design failure for aws_sdk_vendor (not a TODO pending aws_sdk_vendor: flattened source, no network fetch #425), and no longer COPYs a local aws_sdk_vendor/ directory since it now arrives via toolchain-base's existing vcs import step (alongside vector_vendor). .pre-commit-config.yaml and REUSE.toml needed no changes.
  • The github.com/Minipada/aws_sdk_vendor repo (created for aws_sdk_vendor: flattened source, no network fetch #425/PR feat(aws_sdk_vendor): flattened source, no network fetch #433's now-superseded flattened-source design) is un-archived and its content replaced with the thin wrapper above — not deleted, since it's needed again.

Verification (acceptance criteria)

  • colcon build --packages-select aws_sdk_vendor from scratch, fetched from the satellite repo, succeeds — ~13min, clean except one benign upstream LEGACY_BUILD CMake warning.
  • Full workspace colcon build + colcon test (tools/e2e/Containerfile's workspace stage): 17 packages build, 634 tests pass, 0 failures, dc_bridge builds and links in ~15s.
  • ldd against the built dc_bridge binary (after sourcing install/setup.bash) resolves every AWS SDK/CRT .so — no "not found" entries.
  • tools/e2e/scripts/verify_network_isolation.sh still fails as designed: aws_sdk_vendor's build fails under --network=none (the download step errors out) while vector_vendor's succeeds — confirming this is a permanent, attributable seam, not a regression.
  • The AWS_SDK_BUILD_ONLY escaping was verified in isolation (a standalone ExternalProject_Add harness) for both the default single-value case and a multi-service (s3;dynamodb) override, confirming the inner configure step receives the correct semicolon-separated list either way.
  • github.com/Minipada/aws_sdk_vendor's own new CI (Podman build) verified locally against its Containerfile.
  • prek run --all-files --skip build-doc passes clean in this repo, including reuse and hadolint; reuse lint passes clean in the satellite repo too.

Closes #434

🤖 Generated with Claude Code

https://claude.ai/code/session_0169YY4yK42mPfah3DbsXwmt

…ct flattened source

ROS buildfarm binarydeb jobs actually run with `docker run --net=host`
(ros_buildfarm's own job-generation source), and zmqpp_vendor proves the
live-git-clone-at-build-time idiom already works there. That invalidates the
no-network-access assumption #425/PR #433's flattened-source-plus-own-repo
design rested on, so this reverts to a live fetch instead — now via
ament_cmake_vendor_package's ament_vendor() macro (VCS_TYPE git) rather than
a hand-rolled ExternalProject_Add, matching zmqpp_vendor/tinyxml_vendor/
yaml_cpp_vendor. GLOBAL_HOOK is required since dc_bridge only ever
find_package(AWSSDK)s, never find_package(aws_sdk_vendor) first.

Verified end to end: a from-scratch `colcon build --packages-select
aws_sdk_vendor` succeeds (13min), `colcon build`/`colcon test` for the full
workspace pass (634 tests, 0 failures), and `ldd` against the built
dc_bridge binary resolves every AWS SDK/CRT library. The network-isolation
check still fails under --network=none, now permanently by design
(docs/adr/0012), not as a TODO pending #425.

Also archives the github.com/Minipada/aws_sdk_vendor repo #425/PR #433
created — nothing in this workspace pulls from it.

Closes #434

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0169YY4yK42mPfah3DbsXwmt
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
@codecov

codecov Bot commented Aug 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.93%. Comparing base (71be58b) to head (02f72d1).

Additional details and impacted files
@@           Coverage Diff           @@
##            jazzy     #437   +/-   ##
=======================================
  Coverage   70.93%   70.93%           
=======================================
  Files         119      119           
  Lines        7401     7401           
=======================================
  Hits         5249     5249           
  Misses       2152     2152           
Flag Coverage Δ
cpp-jazzy 70.93% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The previous commit archived github.com/Minipada/aws_sdk_vendor and dropped
aws_sdk_vendor into this repo as a local package. Per feedback, that's the
wrong call: this repo's vendor packages live in their own repos as a rule
(matching vector_vendor, ADR-0002's amendment), independent of whether a
given package's own content is large enough to justify the split on git-bloat
grounds alone — a thin ament_vendor() recipe carries no such bloat, but
splitting it anyway keeps every vendor package's layout and bump/release
workflow consistent.

- Un-archives github.com/Minipada/aws_sdk_vendor, replaces its flattened
  vendored source with the same live-fetch ament_vendor() recipe this repo
  briefly held locally (now also supporting an AWS_SDK_BUILD_ONLY override
  for downstream consumers needing services other than s3), and gives it its
  own CI (Podman build + prek format check).
- Removes the local aws_sdk_vendor/ package from this repo; adds it to
  ros2_data_collection.repos (vcs import), the same path vector_vendor uses.
- tools/e2e/Containerfile no longer COPYs a local aws_sdk_vendor/ directory —
  it arrives via toolchain-base's existing vcs import step alongside
  vector_vendor.
- docs/adr/0012 rewritten to record the "own repo" decision and its
  rationale; doc/src/dc/setup.md updated for the new .repos entry.

Re-verified end to end after the change: colcon build --packages-select
aws_sdk_vendor succeeds from the new repo (13min), the full workspace build
+ colcon test pass (634 tests, 0 failures), and ldd against the built
dc_bridge binary resolves every AWS SDK/CRT library.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0169YY4yK42mPfah3DbsXwmt
Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
@Minipada
Minipada merged commit 9d76442 into jazzy Aug 30, 2026
12 checks passed
@github-actions
github-actions Bot deleted the feature/434-aws-sdk-vendor-fetch-aws-sdk-cpp-live-fr branch August 30, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant