feat(e2e): two-container split topology under Compose (dc-ros + vector) - #458
Conversation
Scenario 2 of #440's split-deployment epic: dc-ros and vector as separate Compose-managed containers on one shared network, instead of one process tree. vector runs unmodified from the upstream image (docker.io/timberio/ vector:0.57.0-debian, matching vector_vendor's pinned version), waits for dc-ros's atomically-rendered config to appear on a shared volume, then runs with --watch-config so a later render reloads without a restart. dc-ros runs in unmanaged-shipper mode (#444) with no depends_on on vector anywhere in compose.split.yaml — dc_bringup's existing readiness gate is what lets it recover on its own if the Shipper starts late. The Shipper's buffer and the Bridge's upload state stay on separate volumes, matching #441. Discovered while wiring this up: dc_bridge's Forwarder and readiness prober parse vector_forward_host with inet_pton() (a literal IPv4 parse, not a resolver call), so the vector service needs a fixed compose-network IP rather than its DNS name — worked around at the deployment level (a pinned subnet + static ipv4_address), no dc_bridge change needed. tools/e2e/scripts/run_split.sh folds the scenario into the zero-loss E2E harness: starts dc-ros well before vector to prove no orchestrator-level ordering is required, then runs the same steady-state/outage/restart/ drain sequence as run.sh, reusing verify_zero_loss.py's Postgres/ledger/ upload-intent-queue checks. Its own params file carries no passthrough/ MCAP/raw config (out of scope for this container-boundary proof), so verify_zero_loss.py's --passthrough-file/--mcap-summary-file/--raw-file flags become optional — every existing caller keeps passing all three unchanged. Not wired into ci.yaml, same as every other narrow E2E scenario. Closes #445 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GAFgrfo5YzvvnUW4i6eTVu Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## jazzy #458 +/- ##
==========================================
+ Coverage 71.00% 71.03% +0.03%
==========================================
Files 120 121 +1
Lines 7462 7471 +9
==========================================
+ Hits 5298 5306 +8
- Misses 2164 2165 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
forwarder.cpp and readiness.cpp parsed the Shipper's host with inet_pton() — a literal IPv4 parse, not a resolver call — so a hostname there (e.g. a Compose service name in the split-deployment topology, #445) failed startup immediately with "invalid host address". Add dc_bridge_core's resolve_ipv4() (net_resolve.hpp/.cpp), backed by getaddrinfo(), and use it in both places; a literal IP still resolves with no network round trip, so every existing IP-based deployment is unaffected. Update compose.split.yaml/e2e_split_params.yaml to point vector_forward_host at the plain "vector" service name now that it resolves correctly, dropping the fixed-subnet/static-IP workaround from the previous commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GAFgrfo5YzvvnUW4i6eTVu Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
|
Follow-up commit: fixed the root cause instead of working around it. `dc_bridge`'s Forwarder and readiness prober now resolve `vector_forward_host` via `getaddrinfo()` (new `dc_bridge/src/net_resolve.cpp`) instead of the old literal-IPv4-only `inet_pton()` parse. `compose.split.yaml`/`e2e_split_params.yaml` now use the plain `vector` service name — the fixed-subnet/static-IP workaround from the first commit is gone. Added unit coverage: |
Shorten the compose.split.yaml/e2e_split_params.yaml/run_split.sh headers and a few C++ comments down to one or two lines each, per CLAUDE.md's comment guidance. No behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GAFgrfo5YzvvnUW4i6eTVu Signed-off-by: David Bensoussan <d.bensoussan@proton.me>
Summary
Scenario 2 of #440's split-deployment epic:
dc-rosandvectorrun as separate Compose-managed containers on one shared network instead of one process tree.tools/e2e/compose.split.yaml— a standalone, runnable Compose deployment:vectorruns unmodified from the upstreamtimberio/vector:0.57.0-debianimage (matchingvector_vendor's pinned version), waits fordc-ros's atomically-rendered config on a shared volume, then runs with--watch-configso a later render reloads without a restart.dc-rosruns in unmanaged-shipper mode (Bridge: unmanaged-shipper mode and atomic config write #444). Nodepends_onanywhere between the two —dc_bringup's existing readiness gate is what letsdc-rosrecover on its own if the Shipper starts late. The Shipper's buffer and the Bridge's upload state stay on separate volumes (Split shipper.data_dir into separate Shipper and Uploader directories #441).dc_bridge's Forwarder/readiness prober parsevector_forward_hostwithinet_pton()(a literal IPv4 parse, not a resolver call), sovectorneeds a fixed compose-network IP rather than its DNS name. Worked around at the deployment level (pinned subnet + staticipv4_address) — nodc_bridgecode change.tools/e2e/scripts/run_split.shfolds the topology into the zero-loss E2E harness: startsdc-roswell beforevectorto prove no orchestrator-level ordering is required, then runs the same steady-state/outage/restart/drain sequence asrun.sh, reusingverify_zero_loss.py's Postgres/ledger/upload-intent-queue checks.verify_zero_loss.py's--passthrough-file/--mcap-summary-file/--raw-fileflags become optional (wererequired=True) since this scenario's own params file carries no passthrough/MCAP/raw config — out of scope for a container-boundary proof. Every existing caller keeps passing all three unchanged, sorun.sh/run_degraded.share unaffected.Not wired into
ci.yaml, matching every other narrow E2E scenario (retention/incident/degraded/limits axes).Closes #445
Test plan
prek run --all-files --skip build-docpasses (ruff, shellcheck, clang-format, REUSE, YAML/JSON/TOML syntax, etc.)bash -n+shellcheckclean onrun_split.shcompose.split.yaml/e2e_split_params.yamlparse as valid YAMLpodman compose:postgres/rustfscome up and are reachable via compose service-name DNS;vectorgets its pinned static IP; thevectorservice genuinely blocks until the shared config file appears, then starts and picks up a config-file change via--watch-configwith no restart (confirmed in the container's own logs)docker.io/timberio/vector:0.57.0-debianis a real, pullable tag matchingvector_vendor's pinned versionrun_split.shexecution against a builtdc-e2eimage (a fullcolcon buildof the workspace was outside this session's time budget; the harness reusesrun.sh's own build path unchanged, and the parts specific to this issue — compose mechanics, the wait/watch entrypoint, DNS vs. static-IP addressing — were validated directly against real containers as above)