vector_vendor: no network fetch, split into its own repo

.github/workflows/ci.yaml

@@ -18,9 +18,9 @@
 #   kpi-views (standalone: SQL + a throwaway Postgres, no workspace image needed)
 #   raw-volume (standalone: stdlib Python fixtures over the shared volume accounting)
 #   network-isolation-check (standalone: builds tools/e2e/Containerfile's
-#     vendor-network-check stage — #423. Disabled (if: false) until #424/#425 replace
-#     aws_sdk_vendor's/vector_vendor's build-time network fetches with vendored/prebuilt
-#     sources — until then this job's build is *expected* to fail.)
+#     vendor-network-check stage — #423. vector_vendor's build-time fetch is already
+#     replaced with a checked-in binary (#424); disabled (if: false) until #425 does
+#     the same for aws_sdk_vendor's — until then this job's build is *expected* to fail.)
 #
 # e2e calls the same tools/e2e/scripts/run.sh a developer runs locally, driving the
 # harness with plain podman (no compose) so nothing extra is installed on the runner;
@@ -339,11 +339,10 @@ jobs:
   # continue-on-error: true — the job's own check-run conclusion reports "failure"
   # regardless of continue-on-error (only the *workflow's* aggregate conclusion and
   # `needs:`-blocking are affected by it), so every PR still showed a red ❌ for a
-  # failure that's expected today, not actionable. aws_sdk_vendor still git-clones
-  # aws-sdk-cpp and vector_vendor still downloads a prebuilt binary, both at
-  # colcon-build time, until #424/#425 replace those fetches with vendored/prebuilt
-  # sources — flip `if: false` to `if: true` (and drop this comment) once they land
-  # and this build can actually pass.
+  # failure that's expected today, not actionable. vector_vendor now installs a
+  # checked-in binary (#424), but aws_sdk_vendor still git-clones aws-sdk-cpp at
+  # colcon-build time — flip `if: false` to `if: true` (and drop this comment) once
+  # #425 replaces that fetch too and this build can actually pass.
   network-isolation-check:
     runs-on: ubuntu-latest
     if: false
@@ -353,7 +352,7 @@ jobs:
       - name: Verify podman
         run: podman --version
 
-      - name: Attempt the network-isolated vendor package build (expected to fail until #424/#425 land)
+      - name: Attempt the network-isolated vendor package build (expected to fail until #425 lands)
         run: ./tools/e2e/scripts/verify_network_isolation.sh
 
   # The KPI definitions are SQL, not workspace code: this needs a database and nothing else.

doc/src/dc/migration.md

@@ -583,9 +583,10 @@ dc_bridge:
   root to run it). `fluent_bit_plugins` and `dc_destinations` no longer exist.
 - **No Go toolchain.** The `out_minio.so` / `out_files_metrics.so` Go plugins are gone
   along with `plugin_path`.
-- **Vector is vendored.** `vector_vendor` downloads a pinned, checksummed Vector binary
-  at build time. Point `-Dvector_path=/usr/bin/vector` (or `VECTOR_PATH`) at a
-  system-installed binary for air-gapped builds.
+- **Vector is vendored.** `vector_vendor` installs a pinned, checksummed Vector binary
+  checked directly into the package — no network access at build time. Point
+  `-Dvector_path=/usr/bin/vector` (or `VECTOR_PATH`) at a system-installed binary
+  instead if preferred.
 - **Startup is ordered.** The Bridge starts before the collection nodes and a readiness
   gate blocks activation until the Shipper is accepting connections, so no Record is
   produced before the pipeline can accept it. See

doc/src/dc/setup.md

@@ -12,7 +12,7 @@ and has a considerably longer setup; if you are coming from it, read the
 ## Requirements
 
 - ROS 2 Jazzy (`ros-jazzy-ros-base` or larger), on Ubuntu 24.04 or a Debian equivalent
-- `colcon`, `rosdep`, `git`, a C++17 compiler
+- `colcon`, `rosdep`, `git`, `vcstool` (`python3-vcstool`), a C++17 compiler
 - x86-64 or aarch64 — the architectures `vector_vendor` has a pinned Vector binary for
 
 ## Build
@@ -22,9 +22,11 @@ and has a considerably longer setup; if you are coming from it, read the
 mkdir -p ~/ws/src && cd ~/ws/src
 git clone https://github.com/minipada/ros2_data_collection.git
 
-# 2. Register DC's local rosdep rules (two header-only C++ libraries upstream
-#    rosdistro has no key for), then resolve dependencies
+# 2. Pull in vector_vendor (its own repo — see ADR-0002's amendment), register DC's
+#    local rosdep rules (two header-only C++ libraries upstream rosdistro has no key
+#    for), then resolve dependencies
 cd ~/ws
+vcs import src < src/ros2_data_collection/ros2_data_collection.repos
 echo "yaml file://$PWD/src/ros2_data_collection/rosdep/dc.yaml" \
   | sudo tee /etc/ros/rosdep/sources.list.d/10-dc.list
 rosdep update
@@ -35,12 +37,12 @@ source /opt/ros/jazzy/setup.bash
 colcon build
 ```
 
-That is the whole install. `colcon build` also runs `vector_vendor`, which downloads a
-pinned, checksummed [Vector](https://vector.dev/) binary — the external **Shipper** the
-Bridge supervises at runtime (ADR-0002).
+That is the whole install. `colcon build` also runs `vector_vendor`, which installs a
+pinned, checksummed [Vector](https://vector.dev/) binary checked into the package — the
+external **Shipper** the Bridge supervises at runtime (ADR-0002).
 
 ```admonish tip title="Air-gapped or distro-packaged Vector"
-Point the build at a Vector binary you already have instead of downloading one:
+Point the build at a Vector binary you already have instead of the checked-in one:
 
     colcon build --cmake-args -Dvector_path=/usr/bin/vector
 

docs/adr/0002-vector-as-default-shipper.md

@@ -4,8 +4,79 @@ Both upstream Fluent Bit and Vector can sit behind the Bridge's Forward-protocol
 
 ## Consequences
 
-- CI publishes a "vector-slim" prebuilt binary (feature-flagged build, ~30–40MB) for constrained deployments; users never compile Vector themselves.
-- A `vector_vendor` ament package downloads the official static binary at build time, pinned to an exact version by checksum (arch-detected); a `vector_path` parameter allows using a system-installed Vector instead. Apt-repo install and Docker remain documented alternatives.
+- A `vector_vendor` ament package vendors the official static binary, pinned to an exact version by checksum (arch-detected); a `vector_path` parameter allows using a system-installed Vector instead. Apt-repo install and Docker remain documented alternatives.
+
+## Amendment: checked-in binary instead of a build-time download (#424)
+
+`vector_vendor` originally fetched its pinned binary over the network at build time via
+`file(DOWNLOAD ...)`, which violates the ROS buildfarm's no-network-access policy for
+binarydeb jobs and defeats #423's network-isolated build check. The "vector-slim"
+feature-flagged build this ADR originally proposed CI would publish was never actually
+built — no such CI job exists in this repo — so it is not available to vendor either.
+
+Fixed by checking the official `vectordotdev/vector` release tarballs (x86_64 and
+aarch64, same bytes `file(DOWNLOAD ...)` used to fetch, same pinned SHA256 checksums)
+directly into `vector_vendor/prebuilt/`; `CMakeLists.txt` now extracts and installs from
+the local file, performing no network I/O. The *tarball*, not the extracted binary, is
+what's committed: the extracted `vector` binary is 120–142MB (stripped/unstripped)
+depending on target, over GitHub's 100MB hard per-file limit, while the release
+tarball's upstream gzip compression brings each architecture's file to ~50-54MB — under
+the limit.
+
+**Not tracked via Git LFS**, per #421's original implementation decision ("no
+release-time injection step, no `git-lfs`: committed like any other tracked file").
+LFS was evaluated and briefly adopted mid-review: the specific concern that motivated
+#421's original call — that LFS content wouldn't survive `bloom`'s release-tarball
+export step — turned out to be unfounded. `bloom`'s `export_upstream` delegates to
+`vcstools.GitClient.checkout()` (a plain `git clone` + `git checkout <tag>`, which does
+run LFS's smudge filter), and the export machinery itself
+(`vcstools.git_archive_all.GitArchiver`) reads files off the working-tree filesystem
+rather than through `git archive`'s blob-store plumbing (the actual, narrower reason
+plain `git archive` breaks LFS) — confirmed empirically against a throwaway LFS repo and
+the real `vcstools`/`bloom` code, not just read.
+
+LFS was reverted anyway for a more basic reason: GitHub's free LFS tier is 1GB storage
+and 1GB bandwidth per month, and every future Vector version bump adds both tarballs
+(~106MB) as new, non-deduplicated LFS objects — roughly 9 version bumps before storage
+alone exhausts the free tier, independent of and sooner than the plain-blob approach's
+own cost (git history growing by the same ~106MB per bump, but against no comparable
+quota). A plain committed blob has no such ceiling; the tradeoff is that every future
+Vector bump grows this repo's ordinary git history by ~106MB, permanently, since git
+does not deduplicate binary blobs across versions. Revisit if/when that accumulation
+becomes the more pressing cost — the LFS path is proven to work, should it be needed.
+
+## Amendment: split into its own repo, pulled in via `.repos`
+
+The ~106MB-per-bump cost the previous amendment accepted as `vector_vendor`'s tradeoff
+doesn't have to be `ros2_data_collection`'s cost to carry — nothing about the rest of
+this repo's history needs to grow every time Vector ships a release. `vector_vendor`
+now lives at [github.com/Minipada/vector_vendor](https://github.com/Minipada/vector_vendor)
+(default branch `jazzy`, matching this repo's own active line), containing exactly what
+used to sit at `ros2_data_collection/vector_vendor/`: the same `CMakeLists.txt`
+(checked-in tarball, checksum-pinned, no network at build time — the previous
+amendment's decision is unchanged, just relocated) and the same two prebuilt tarballs.
+`package.xml`'s `<version>` there tracks the vendored Vector version directly (`0.57.0`
+at the time of the split), not an independent counter.
+
+Pulled into this workspace via `ros2_data_collection.repos` (vcstool) rather than a
+`COPY`/git submodule: `vcs import` pins to a **tag** (`v<VECTOR_VERSION>`), not a
+branch, so a given `ros2_data_collection` commit always resolves the same
+`vector_vendor` content — a floating branch would make the build's vector_vendor
+content silently drift out from under an unrelated `ros2_data_collection` change.
+`tools/e2e/Containerfile`'s `toolchain-base` stage runs the import (network required,
+same as its rosdep install right below it); every downstream stage's actual `colcon
+build` — including #423's `--network=none` check — builds against the already-fetched
+result, unaffected by where the source physically came from. Bumping Vector now touches
+two repos: land the new binaries + tag in `vector_vendor`, then bump the pinned
+`version:` in `ros2_data_collection.repos` to match.
+
+This is a development-workspace convenience only (`vcs import`/`colcon build` from
+source); it does not by itself make `vector_vendor` installable via `apt` — that
+requires its own independent `bloom-release` into `rosdistro`, not yet done, at which
+point `ros2_data_collection` packages that need it would instead declare a normal
+`<depend>vector_vendor</depend>` and let rosdep resolve the released `.deb`, same as any
+other ROS package dependency. Until then, `.repos` is how CI and local dev get a
+buildable workspace.
 
 ## Why the forward protocol is the Bridge→Shipper wire format
 

ros2_data_collection.repos

@@ -0,0 +1,13 @@
+# SPDX-FileCopyrightText: 2022-2026 David Bensoussan
+# SPDX-License-Identifier: MPL-2.0
+
+# External source repos this workspace needs alongside ros2_data_collection itself,
+# for `vcs import` (see doc/src/dc/setup.md). vector_vendor was split out of this repo
+# (see docs/adr/0002-vector-as-default-shipper.md's amendment) so its binary content's
+# growth doesn't land in ros2_data_collection's own git history. Pinned to a tag, not a
+# branch, so a workspace built from a given ros2_data_collection commit is reproducible.
+repositories:
+  vector_vendor:
+    type: git
+    url: https://github.com/Minipada/vector_vendor.git
+    version: v0.57.0

tools/e2e/Containerfile

@@ -75,12 +75,26 @@ RUN echo "apt cache bust: ${APT_CACHEBUST}" && \
     apt-get -q update && \
     apt-get -q -y dist-upgrade && \
     apt-get -q install --no-install-recommends -y \
-    git python3-pip ccache lcov && \
+    git python3-pip python3-vcstool ccache lcov && \
     pip install --break-system-packages --no-cache-dir fastcov mcap && \
     rm -rf /var/lib/apt/lists/*
 
 WORKDIR /root/ws
 
+# vector_vendor lives in its own repo (docs/adr/0002-vector-as-default-shipper.md's
+# amendment), so it has to be fetched into the workspace source tree the same way a
+# developer's own `vcs import` would, before anything downstream can `colcon build` it.
+# A separate COPY (not folded into the apt RUN above) keys this layer's own cache on
+# ros2_data_collection.repos's content: it re-fetches when that file changes (a version
+# bump) or when APT_CACHEBUST invalidates the layer above (daily) — not on an ordinary
+# source edit elsewhere in the repo. `vcs import` needs network; every later `colcon
+# build`/`colcon test` in downstream stages runs against the already-materialized
+# result, so #423's network-isolated check (which branches from this same stage) still
+# isolates only the actual build step, not this fetch.
+COPY ros2_data_collection.repos .
+RUN mkdir -p src/ros2_data_collection && \
+    vcs import --shallow src/ros2_data_collection < ros2_data_collection.repos
+
 FROM toolchain-base AS toolchain
 
 # aws_sdk_vendor built in its own layer, isolated from the rest of the toolchain stage
@@ -122,29 +136,30 @@ RUN apt-get -q update && \
     rosdep install -y --from-paths src --ignore-src --rosdistro jazzy --as-root=apt:false && \
     rm -rf /var/lib/apt/lists/*
 
-# --- vendor-network-check: proves the aws_sdk_vendor / vector_vendor build-time network
-# fetches (ExternalProject_Add's GIT_REPOSITORY, file(DOWNLOAD ...)) are isolated to the
-# actual `colcon build` step, not smeared together with rosdep/apt's own (allowed)
-# network use for installing system packages (#423). Branches from `toolchain-base` —
-# *before* the `toolchain` stage above already built aws_sdk_vendor — so this is a real,
-# from-scratch build attempt each time, not a cache hit against work the `toolchain`
-# stage already did.
+# --- vendor-network-check: proves aws_sdk_vendor's build-time network fetch
+# (ExternalProject_Add's GIT_REPOSITORY) is isolated to the actual `colcon build` step,
+# not smeared together with rosdep/apt's own (allowed) network use for installing
+# system packages (#423). Branches from `toolchain-base` — *before* the `toolchain`
+# stage above already built aws_sdk_vendor — so this is a real, from-scratch build
+# attempt each time, not a cache hit against work the `toolchain` stage already did.
+# vector_vendor no longer needs checking here: it's fetched (with network) by
+# `toolchain-base`'s own `vcs import` step above, and its build has been network-free
+# since #424 — this stage now exists purely for aws_sdk_vendor, until #425 lands.
 #
 # `RUN --network=none` is a per-instruction Buildah/Podman flag (independent of, and
 # verified to take precedence over, build.sh's own top-level `--network host`): only the
 # colcon build below loses network access, while the rosdep install right above it keeps
-# the same network access every other stage's rosdep install gets. Today (before #424 /
-# #425 replace the git clone and the binary download with vendored/prebuilt sources)
-# that colcon build is expected to fail here — that's the point: this stage exists to
-# prove the isolation seam itself works, with a clear, attributable network error, before
-# it's ever asked to pass. Not part of the production image graph — nothing downstream
-# depends on it. Built via tools/e2e/scripts/verify_network_isolation.sh.
+# the same network access every other stage's rosdep install gets. Today (before #425
+# replaces the git clone with a vendored source) that colcon build is expected to fail
+# here — that's the point: this stage exists to prove the isolation seam itself works,
+# with a clear, attributable network error, before it's ever asked to pass. Not part of
+# the production image graph — nothing downstream depends on it. Built via
+# tools/e2e/scripts/verify_network_isolation.sh.
 FROM toolchain-base AS vendor-network-check
 SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 
 COPY rosdep src/ros2_data_collection/rosdep
 COPY aws_sdk_vendor src/ros2_data_collection/aws_sdk_vendor
-COPY vector_vendor src/ros2_data_collection/vector_vendor
 RUN echo "yaml file:///root/ws/src/ros2_data_collection/rosdep/dc.yaml" \
       > /etc/ros/rosdep/sources.list.d/10-dc.list && \
     apt-get -q update && \

tools/e2e/README.md

@@ -245,7 +245,7 @@ can't establish:
 ./tools/e2e/scripts/run_load_driver_shipper_test.sh
 ```
 
-Pulls the exact Vector version `vector_vendor/CMakeLists.txt` pins, runs it as a bare
+Pulls the exact Vector version `ros2_data_collection.repos` pins vector_vendor to, runs it as a bare
 `fluent` source (global acknowledgements on, exactly as `render.cpp` configures it for
 the real Bridge) plus a `file` sink, points the driver at it, and asserts every frame was
 accepted and acknowledged and that the driver's ledger names the exact same records

tools/e2e/scripts/run_limits_drain_rate.sh

@@ -123,9 +123,9 @@ cleanup() {
 trap cleanup EXIT
 
 # The exact Vector version dc_bridge is built and tested against, same as
-# run_limits_two_tier.sh / run_load_driver_shipper_test.sh -- read out of the file that
-# pins it, not duplicated as a second source of truth.
-VECTOR_VERSION="$(grep -oP 'set\(VECTOR_VERSION "\K[^"]+' "$REPO_ROOT/vector_vendor/CMakeLists.txt")"
+# run_limits_two_tier.sh / run_load_driver_shipper_test.sh -- read out of the .repos pin
+# that fetches vector_vendor, not duplicated as a second source of truth.
+VECTOR_VERSION="$(grep -oP 'version: v\K[0-9.]+' "$REPO_ROOT/ros2_data_collection.repos")"
 VECTOR_IMAGE="docker.io/timberio/vector:${VECTOR_VERSION}-debian"
 
 remove_stack

tools/e2e/scripts/run_limits_shipper_fanin.sh

@@ -162,7 +162,9 @@ else
   DC_IMAGE="dc-e2e:latest"
 fi
 
-VECTOR_VERSION="$(grep -oP 'set\(VECTOR_VERSION "\K[^"]+' "$REPO_ROOT/vector_vendor/CMakeLists.txt")"
+# Read out of the .repos pin that fetches vector_vendor (its own repo as of #424's
+# follow-up split), not duplicated here as a second source of truth.
+VECTOR_VERSION="$(grep -oP 'version: v\K[0-9.]+' "$REPO_ROOT/ros2_data_collection.repos")"
 VECTOR_IMAGE="docker.io/timberio/vector:${VECTOR_VERSION}-debian"
 
 remove_all

tools/e2e/scripts/run_limits_two_tier.sh

@@ -199,9 +199,9 @@ else
 fi
 
 # The exact Vector version dc_bridge is built and tested against, for the standalone
-# aggregating Shipper — read out of the file that pins it, not duplicated as a second
-# source of truth (same as run_load_driver_shipper_test.sh).
-VECTOR_VERSION="$(grep -oP 'set\(VECTOR_VERSION "\K[^"]+' "$REPO_ROOT/vector_vendor/CMakeLists.txt")"
+# aggregating Shipper — read out of the .repos pin that fetches vector_vendor (same as
+# run_load_driver_shipper_test.sh), not duplicated as a second source of truth.
+VECTOR_VERSION="$(grep -oP 'version: v\K[0-9.]+' "$REPO_ROOT/ros2_data_collection.repos")"
 VECTOR_IMAGE="docker.io/timberio/vector:${VECTOR_VERSION}-debian"
 
 # Clean any leftovers from a previous (possibly DC_E2E_KEEP=true) run.